Veeam 5 - Offsite replication of backup

[danscott]

Hi Everyone,

I've got a scenario that I'd like to run past you all to see if what I'd like to do with Backup and Replication 5 is possible-

- I have a client who has 2x ESXi 4.1 hosts and vcenter server running Essentials Plus.
- They have a Netgear ReadyNas 2100 8TB that is used as a backup destination (Primary storage is a HP MSA SAS SAN)
- Currently backup is just held on-site with a mixture of Windows Server Backup and VMware Data Recovery taking the backup.

The client would like to add protection against fire/theft in their building by having the backups replicated offsite, then if the worst were to happen it would just be a case of getting a server to their office with ESXi on it and reimport the replicated VM's from local storage / loan NAS etc. First point of restore would always be from the local office for speed of data transfer.

We have a data center where we could host say another Readynas 2100 for them to use as storage for replicated backups and put another ESXi host in the datacenter to act as a target for replication using the second NAS as an NFS/iSCSI datastore.

Internet links are a bit of a problem here - it would be great if everyone had 100mb leased lines but sadly they only have 8mbps up and downstream bandwidth (Bonded SDSL) Datacenter end is no problem - 100mb circuit.

So would this be possible-
- Create LAN-LAN VPN between the two sites
- In Veeam B&R5 add the clients local vCenter and remote ESXi host as available servers.
- Create a replication job for a VM using a USB disk as a way of getting the initial replication done without having to send it over the slow internet connection.
- Take USB disk to data center, import replicated VMDK's into new ESXi host.
- Allow replication job to continue as normal.

I guess the key to this working is that after the above steps, does Veeam only send incrementals to the remote host?

We've had a go at attempting this by using NAS r-sync replication but with standard backups the files generated are too big and it can't back up in time. Main problem with standard backup is sooner or later the original full backup file has incrementals rolled up into it and as a result the big file changes requiring replication which is too slow.

I thought that by doing the replication with Veeam this might work as it would only be incrementals that would be sent over the internet.

Any help would be greatly appreciated - I really don't miss tapes one bit but it would be great to have a strategy that can allow Veeam backups to be held offsite that doesn't require really quick leased lines etc at both ends as with clients using ESX Essentials these lines are usually too expensive.

Thanks!
Dan

[marty9876]

I would not attempt to try w/ Veeam v5, wait for v6 which should be out this quarter hopefully.

It's possible yes but likely a study in frustration. It all depends on the delta changes along with Veeam's attitude of the day. Core issues in v5 are really poor bandwidth usage and really unreliable results.

From my experience v5 is hit or miss on using all available bandwidth per job and regardless of how you set things up it's either worse or worse off options regarding bandwidth usage & requirements. Reports of random massive increases in replication jobs have been posted and I've seen it myself.

Reliability is a big miss with v5 and WAN replication. If a job starts, fails for any reason, your replica is shot at the DR site until a new jobs is completed.

I've given up until v6 which should resolve these issues I hope.

[Vitaliy S.]

I guess the key to this working is that after the above steps, does Veeam only send incrementals to the remote host?

This would certainly work, however I would support Marty in asking you to wait for v6 to come out, as our replication engine has been improved dramatically.

[danscott]

Thanks Marty and Vitaliy,

From my research so far (Read Googling around for stuff!) I have found that a lot of people seem to be using WAN optimizers/accelerators such as Netex HyperIP example http://read.virtualizeplanet.com/?p=460 has anyone found these to be effective at implementing this type of solution? Problem is there's no mention on their website of cost so not sure how many 0's we're talking at the end of the price tag?

Thanks for the tip on version 6 - do you have any approximate ship dates on this yet? and can you upgrade version 5 (I have some other clients already using V5)

From my searches this seems to be a common problem for people moving away from BackupExec>Tape drives when they implement virtualised platforms for customers rather than small file/mail/database server setups so if Veeam can get it right with V6 then it will be a big plus. If you've got a nice VPLS network between multiple offices and loads of bandwidth it's not a problem but for the SME market it's a bit of a struggle at the moment to get their backups out of the building.

[chrisdearden]

I thought rsync was supposed to be a more block level replication ? unless the readynas uses a pretty old version of rsync ?

nope - it seems to be block level...

http://www.readynas.com/forum/viewtopic ... 31&t=23356

[danscott]

We've tried using the Readynas Replicate feature (Paid license add-on for Readynas business units) http://www.readynas.com/?cat=93

So far it's not been great - might have a look at the rsync options.

[Vitaliy S.]

has anyone found these to be effective at implementing this type of solution

There are many real-life examples reported by our existing customers, please check it out: WAN accelleration tools

Thanks for the tip on version 6 - do you have any approximate ship dates on this yet?

Very soon.

and can you upgrade version 5 (I have some other clients already using V5)

Yes, all customers with active maintenance agreement in force are eligible to v6 free of change. Thanks!

[mcwill]

We successfully use QNAP NAS boxes for the above using simple rsync, one point to bear in mind if you do go down this route is that default processing for Veeam is to rename the backup after each successful run. This breaks the change tracking utilised by rsync on simple NAS boxes as they normally only permit replication of a folder contents to a remote site.

There is a registry setting that disables the above on Veeam 5 which means the filenames are left unchanged and results in much faster replication.

[st0623]

From my research so far (Read Googling around for stuff!) I have found that a lot of people seem to be using WAN optimizers/accelerators such as Netex HyperIP example http://read.virtualizeplanet.com/?p=460 has anyone found these to be effective at implementing this type of solution? Problem is there's no mention on their website of cost so not sure how many 0's we're talking at the end of the price tag?

I'm with the HyperIP team at NetEx. If you need pricing, just reach out to me and we'll get you some budget numbers.

[danscott]

We successfully use QNAP NAS boxes for the above using simple rsync, one point to bear in mind if you do go down this route is that default processing for Veeam is to rename the backup after each successful run. This breaks the change tracking utilised by rsync on simple NAS boxes as they normally only permit replication of a folder contents to a remote site.

There is a registry setting that disables the above on Veeam 5 which means the filenames are left unchanged and results in much faster replication.

Thanks mcwill, so if say there's 1 backup image file and it's 10GB then has 1GB appended as a result of that night's incremental, the r-sync job will recognise that the file has only been appended to and send the 1GB of changes over the WAN to the other NAS?

You don't happen to have that reg key to hand as well by any chance?

Thanks,
Dan

[danscott]

I'm with the HyperIP team at NetEx. If you need pricing, just reach out to me and we'll get you some budget numbers.

Thanks Steve, email om it's way to you now!

[mcwill]

Thanks mcwill, so if say there's 1 backup image file and it's 10GB then has 1GB appended as a result of that night's incremental, the r-sync job will recognise that the file has only been appended to and send the 1GB of changes over the WAN to the other NAS?

Dan,

Yes that's correct, each night we transfer approx 8GB of data to update 199GB of backups from 17 VMs.

You don't happen to have that reg key to hand as well by any chance?

I do, but I don't know what the protocol is wrt sharing info from Support, perhaps Vitaliy could indicate if this is permited?

Regards,
Iain

[tsightler]

The information for reverting to "old style" naming is available in the Veeam support KB.

http://www.veeam.com/kb_articles.html/R ... 1/registry

[Vitaliy S.]

Thanks, Tom! Dan, also be aware that there is another workaround that would work with rsync natively: V5 VBK file names

[danscott]

OK update from my experiments so far.

I've rigged up a test setup and replicated a basic Windows XP VM (Smallest install I could think of in the moment) from one site to another.

I've setup Veeam in a 'pull' replication ie it is installed at the 'Offsite' end rather than the live server end.

I've made the reg changes to stop the files being renamed as per the links posted.

So now I'll put in some dummy data to form delta increments and see how things go - one question - the feature of using temporary storage to perform the initial replication, if using the 'pull' method of installing Veeam at the DR site, is this feature now redundant as obviously veeam isn't local to the VM's so can't output the initial backup to a device local to the VM itself being backed up?

[foggy]

Yes, replica seeding does not apply to the pull scenario.

[Vitaliy S.]

However with version 6 Veeam Backup server placement (production or DR site) no longer matters, so notions such as push/pull replication, or ESX/ESXi target differences are no more.

[danscott]

Long shot this one Vitaliy - does a Veeam 5 purchase have any active maintenance as standard? We have a number of clients with B&R 5 that is less than 1 year old - would they have any upgrade eligability? (As I said - long shot I know!)

Thanks,
Dan

[danscott]

Yes, replica seeding does not apply to the pull scenario.

Thanks for confirming that Foggy - I thought as much which is a shame as in scenarios where there's a massive VM to replicate a push replication is the only way (At least with V5 from what Vitaliy says below.)

[foggy]

Long shot this one Vitaliy - does a Veeam 5 purchase have any active maintenance as standard? We have a number of clients with B&R 5 that is less than 1 year old - would they have any upgrade eligability? (As I said - long shot I know!)

All new licenses include 1 year of maintenance and upgrade will be free for customers with active maintenance.

[danscott]

Great news - will look forward to testing out V6 then!

[danscott]

Hi - me again!

OK well my test replication is working well and the deltas I'm simulating are going across - not too quickly but they are going. So on the 'DR' server I have a replicated VM and at the moment about 14 restore points.

Question is say we lost the main server/site so we have on the 'DR' ESXi host the replicated VM. As the network may have been lost in the 'Disaster' how would I go about getting a copy of the VM that we could then take and put on the replacement host that would go in?

Veeam obviously gives me the option to fail over to a replication point but what would be great is just to grab the VMDK/VMX etc and be able to put the, onto the replacement host and add to the inventory.

What sort of process do you guys think would be the best course of action in this situation (This is obviously still a lab test - noones server has gone belly-up )

Thanks again!
Dan

[foggy]

Generally, we do not recommend to power on replica VMs manually using vSphere tools as in this case you will lost the ability to rollback to any previous restore point. For example, if it turns to be some issue with the latest state after powering on the VM manually, you will no longer be able to failover to an earlier state. That's why we recommend using Veeam B&R UI to perform failover.

[danscott]

Generally, we do not recommend to power on replica VMs manually using vSphere tools as in this case you will lost the ability to rollback to any previous restore point. For example, if it turns to be some issue with the latest state after powering on the VM manually, you will no longer be able to failover to an earlier state. That's why we recommend using Veeam B&R UI to perform failover.

Thanks Foggy - if the primary site is completely gone - like on fire on something and all the VM's are replicated to a DR esx server using Veeam B&R what would the recommended method to get them back onto a replcement primary server?

[foggy]

The best way is to failover to replicated VMs and use them until your primary server is back online. Once it is up again, you can vMotion VMs back to the primary site. Also, our upcoming v6 will allow to perform automatic failback to the original VMs transferring only differences since the time of disaster.

[danscott]

Only problem I can see is that the vmotion if the DR site is accross a WAN link could take forever! Ideally it would be great to be able to get the VMDK's and take them to the new server manually and import them. If they're in a datacenter accross a slow wan link then DR recovery could be tricky.

[foggy]

As I said, with v6 you will be able to transfer only differences with the Failback option. This was implemented right for the WAN scenario.

[danscott]

OK been doing test 2 this afternoon (I'm getting bored of backups now )

So this test is local VM's running standard Veeam Backup jobs to a NAS share (Readynas 2100) then on a remote Readynas with a VPN over the WAN running an rsync job to backup the share where Veeam outputs the job to.

I've made the reg key change to disable renaming of backup files.

About the job setting, current settings are-
- Incrementals turned on
- Synthetic full's enabled on a Saturday
- 14 day retention and 14 day deleted VM retention

So this means that I will end up with 2x full backups (VBK) files and VIB files in the middle. Ideally I'd only like to have 1x VNK as they are obviously very large and take forever to send offsite using rsync.

What would be my best option to only have 1xVBK for efficient rsyncing? would the option for 'Transform previous full backup chains into rollbacks' be a good idea or would it be better to only have a 7 day retention period which I think would stop the second vbk being created.

All help greatly appreciated!

[Gostev]

You need to be using reversed incremental backup mode for the registry key to work.

[danscott]

I knew that honest

OK job recreated and running now as a reversed incremental - is this the best job setup to use for backing up to a share and then rsyncing offsite? Looking to keep file sizes down as much as possible so there's the smallest amount of data to send over the WAN.