Help with remote backup and SureBackup

[mboddicker]

Here is our situation, we have two sites, where site A is our main office, and site B is our DR location at a third party hosting facility.

At site A, we are running two ESX 4 servers which connect to an iSCSI SAN to store the virtual machine files. At this site we have a Veeam server running on its own stand-alone server. All of our production servers run from these two ESX servers at this site.

At site B, our DR site, we are running a single ESX server which is attached to a large disk array for storage. At this site, we run a Veeam server as a virtual server as well as two other production servers, one is a domain controller and the other is our Internet web server.

Between the two sites, we have a 2xT1 WAN link for connectivity.

Currently, we are replicating all of our production servers from site A to site B once a day. The nightly replication process currently takes about 11 hours to complete. The replication is initiated from the Veeam server located at site A. In addition, we also replicate the web server at site B to site A using the Veeam backup server at site B.

After reading some posts on this forum, we have discovered a couple of potential problems with the way are currently doing things. One concern I have, is that if a disaster occurs during the 11 hours of replication, we have the potential to have some corrupted virtual machines. In addition, since we initiate the backup from site A, if we lose that site, then we cannot restore any previous restore points at site B, only the main VMDK file which may be corrupted. Based on posts on this forum, it appears if we wanted to avoid that, the replication of machines from site A, should actually be initiated from the Veeam server on site B if I understand correctly but that still only resolved the restore point problem.

With the release of Veeam 5, we have thought about trying something different. Ideally, we want to be able to backup servers from site A, and test them at site B using SureBackup. To test this process, we created a test server on site A and backed it up using the Veeam server at site A to a data volume located at site B. The backup itself works fine. After the initial seeding, all other backups don’t appear to take any longer or use any more bandwidth then using replication that we could see. So it appears that only the delta changes on the server are being backed up remotely on subsequent backups. This process leaves us with a good base image at the remote site along with multiple restore points in the event of a disaster which is ideal.

Where it goes all wrong is when we try to setup a SureBackup job from Veeam running on site A, but the Test Lab and data files are located on site B. It appears that the Veeam server is pulling the backup files across the WAN link and serving it up as NFS share to the ESX server running on site B. So when the ESX server at site B attempts to start a virtual server in the Virtual Lab so the data is being transmitted from storage on site B, to the Veeam server on site A, then back to the ESX server located on site B.

We tried setting up the SureBackup job from the Veeam server on site B, but it has to be linked to the initial backup job which was from another server. We also tried importing the job into the server but you still cannot run a SureBackup job from the imported job.

Is there another way to make this work or is there something that we are missing? The ability to backup our production servers to a remote site, and to test and verify that those remote backups are good would be the “holy grail” for us. As it appears now, this product is only useful for backing up and testing all within the same local network. If this is the case, can the product be expanded to support remote backup and testing?

[Gostev]

Hi Matt, thanks for detailed email. You should be able to import backup into a separate Site B Veeam Backup install, and test them there. Did you consider this? Thanks!

[mboddicker]

Yes, we tried this but when setting up a SureBackup job on the Veeam server at site B, there is no way to select the imported backup job. I will do some futher testing of this tomorrow, as we are currently testing a different way to do this.

Right now we are testing the backup of data from site A with the Veeam server on site B. The Veeam server has an iSCSI connection to the SAN volumes located on site A. So far this is working good. We were able to back up our test machine on site A, to our backup data volume on site B, and we were able to run a SureBackup job on site B when it completed. This may be the best way to do this but we have questions regarding bandwidth usage in this scenario. During an incremental backup, does the Veeam server read about the same amount of data from the source volume as it writes to the destination volume? If it reads in way more data then it writes to the destination, then having the backup server doing backups from the remote site would not be an efficient use of the bandwidth between the sites. I assume that this is the case due to the compression and de-duplication that the Veeam server is probably performing.

[Gostev]

Yes, please test again as this does work. When you are adding VM to the application group, there is a way to add VM from backup (Add VM > From Backup).

With ESX 4.0 and later, Veeam Backup only reads changed block from source during the incremental run.

[mboddicker]

Ok, I have some testing results. We backed up a server on site A using a Veeam server on the same site, but the destination was a storage volume on site B.

From the Veeam server on site B, we imported this backup set. One of the issues we discovered is that when we initially imported the backup, it had our test machine and it contained 4 restore points. After the import, we kicked off another backup from site A to add a fifth restore point, but the server at site B still only shows 4. There doesn't appear to be any options to refresh the imported backup, so you have to delete and re-import the backup job to get the latest restore point. Is there a way to automate via a script to delete and add an imported backup job?

After we imported the backup set, we created a new SureBackup job but here is where we are having a hard time. The only way to test the imported backups is to add all of the VM's to an application group. That would mean that for our environment, we would need to startup and test 14 backed up VM's at the same time, rather then starting up an application group with just a domain controller first, then starting and testing the rest of the 13 VM's individually via a linked backup job. Unfortunately, you cannot link a SureBackup job to an imported backup. Is there a way to put all 14 VM's in a single application group, start them one at a time and then shutdown those that are not needed by the other VM's for testing? Can a test script be written shut a machine down after the rest of the tests on that machine have been completed?

[Gostev]

Yep, I know what you are saying - I have been pushing devs for "re-imports" as well. Got to see actual request/confirmation. It may indeed be possible to script everything, if you are willing to research our PowerShell SDK capabilities.

Good point on no possibility of linking imported backups, I have not thought of this issue before, but this will definitely need to be addressed along with re-import. Definitely need ability to link imported backup file as a whole to SureBackup job and perform automatic "re-import" selected location before SureBackup job starts.

[mwpreston]

[merged]

Just wondering, has anyone had any success running a surebackup job on another instance of a veeam server? Here's what I'm trying to do...

Nightly Backups are being copied offsite to another veeam server
I mount those backups to the offsite veeam server
What I want to do is run the sure backup jobs on my offsite server, rather than putting the load on my production hosts. Backups seem to mount fine, however when trying to run a surebackup job I get an error directly after it attempts to publish the vm stating Server Error: End of File.

I'm all up to date regarding veeam versions (5.0.2)

Just wanted to know if anyone had any ideas or if anyone has been successful with something like this...???

Thanks!

[mwpreston]

Has there been any progress on this? or does anyone know of anything I can do to get around it? I can't seem to use any of my imported backups in surebackup jobs or even the instant vm recovery jobs. I always just get server error: end of file when publishing the VM. Just thought I would check back in as I'm just getting around to looking at this again....
Thanks

[Gostev]

Looks like SureBackup has just found an issue with your backup files. This error usually indicates that there is an issue with your backup files, sounds like they can be corrupted. I bet you will get the same error if you attempt to perform full VM restore from these imported backups. This workaround with imported backups should work for sure.

[mwpreston]

Hmmm, I can restore from the backup files at my main location. Maybe there is something that gets lost during my transfer of the files to the second veeam server. I'm using rsync so maybe it isn't actually copying...I will try a native copy of a smaller backup file and see if that works...Thanks gostev. Great preso @ vmworld too by the way

[Gostev]

Exactly, sounds like something is not working with the transfer as it should. Great demonstration of why you always want to test and verify recoverability of your backups at location where you want them to be recoverable, not before you ship them there.

[mwpreston]

Hmmm, looks like Ask Gostev was right again. The backup that I just copied without rsync worked fine....guess I will have to revisit my nightly replication of the backups. Thanks!

[NightBird]

How to Surebackup remote site.

At the Main site Veeam Backup Server, at the remote site Veeam Backup Repository and Veeam Backup Proxy on the same physical server, 1 VMware vsphere 5 cluster (2 hosts) on the remote site.

remote Veeam Backup Proxy/Repository do remote vSphere VM Backup controlled by the Veeam Backup Server at the main site.

How to run SureBackup in this scenario ?

Thanks
Boris

[Gostev]

Set it up normally as you would in the production site, and schedule a PowerShell script to trigger refresh of the backup repository for auto importing of newly copied over backup files into the B&R configuration. This way, SureBackup jobs will always test the latest backup. Thanks!

[NightBird]

Import the backup on the main site ?

It's not possible to surebackup at the remote site directly ? The remote site have ESX server and backup/repo server

[Gostev]

No, import the backup in the remote site. I assumed your are syncing/delivering backups there somehow?

If you are just backing up to backup repository located in the remote site, then not even sure what is the question.
You just set up the virtual lab on the remote site ESX server, and use SureBackup normally.

[NightBird]

Yes but the ping test and so on are launch on the Veeam Backup Management Server at the main site. And this server could not reach the masquerade network at the remote site.
Why the ping test (script test) are not launched by the Veeam proxy ?

[Gostev]

Remember that you can always setup another backup server at DR site, and use that for SureBackup.

Why the ping test (script test) are not launched by the Veeam proxy ?

Because backup proxies do not participate in the SureBackup job anyhow.

[NightBird]

Could I install the Veeam backup server on the actual Veeam proxy/repo server ?

[Gostev]

Yes, no reason why you could not do that.