Which method to use - backup, replication, or both?

[jyarborough]

We are in the evaluation stages of Veeam B&R (again) and so far 6.1 looks a lot more promising than previous versions. Maybe it is because vSphere 5 is much more stable, or maybe they have just really made improvements, but it is noticably more stable and runs a lot smoother. Or maybe we just have a better infrastructure now, who knows. We'll give the credit to Veeam though

Our layout is something like 80+ VM's going from a 100mbit link at the primary site to a 50mbit link at the DR site. We would be looking at throttling the throughput to about 30mbit during weekdays and letting it run at the full 50mbit afterhours and on weekends. Based on that, I think I am going to be cutting it pretty close, maybe even not getting it to fit at all.

Anyway, I've been playing around with replicas because I really like the "failover" capabilities built in and I like that the machine is essentially ready to be powered on in place. I've also looked at backup jobs because I really like the optimizations such as deduplication within the job and the extras like SureBackup and virtual labs. So basically I am torn. Do I go with replication because it provides essentially a warm site that is ready to be powered on? Or do I go with backups to achieve more flexibility? Or do people have success using a combination?

• Some things I don't like about straight replicas:
• If a disk size changes, it forces a full re-replication because the actual disk structure changed. Backups seem to just handle this. It seems like we do this often enough that taking fresh seed loads will get a little old if we use replicas.
• If we have several "similar" machines being backed up (4 nearly identical RDS servers for example), replication sees no benefit from the fact that the data on each is nearly identical. Backups take that into account and only transfer the unique changes (if I recall correctly).
• Replicas do not seem to be able to utilize the newer technologies like SureBackup, etc for ensuring that the data is truly valid and consistent.

• Some things I don't like about straight backups:
• Can't send a ready to be powered on VM directly to VMFS at the remote site. We can use vPower NFS (at least I think I understand that correctly) to provide near power on ready VM's, but it's just not the same as having the DR host's inventory populated with the VM's and ready to go from VMFS datastores.
• Can't have complex retention policies. It looks like the retention policies are still essentially X number of days. I would really like to see something like X daily, X weekly, X monthly, X yearly, where it rolls up and keeps those historically. That is for another time though...

Is there a way to use backups to generate replicas? For example, I would really like to be able to use the backup method to achieve those optimizations and to be able to use SureBackup and all the other features you get with that. However, I would really like to stream the backups both locally (for quick recovery) and offsite (for longer term storage and DR), then use the offsite backups to create my replica VM's that ready for failover. Does that make sense? Is it possible? Maybe I'm just not far enough into my demo for this to make sense yet? Maybe I'm thinking about it all wrong?

I really want to get some feedback to discuss what others are using in this type of case. I see a lot of forum posts about people in the same boat as me, they are doing their initial designs, but not many that come back and say this is how we did it and it worked or don't do it this way because... Instead, the existing discussions really seem to blur together the backups with the replicas.

Any help, thoughts, beer, etc would be appreciated!

Thanks!

John

[tsightler]

Hi John,

First just want to address a few things that might not be technically 100% accurate:

• If a disk size changes, it forces a full re-replication because the actual disk structure changed. Backups seem to just handle this. It seems like we do this often enough that taking fresh seed loads will get a little old if we use replicas.

[/color]
Changing the disk size doesn't require a full re-replicaiton, however, all existing restore points on the target replica are deleted so if you have a replica that normally has 7 restore points, and you change the disk size on the source, the very next run of the replica job will basically delete all but the most recent restore point, the disk size will be changed, and finally the new data replicated. From that point replication will continue on. I believe that this is due to a VMware limitation where disk sizes cannot be changed on a VM that has active snapshots (you can add new disks however).

• Can't send a ready to be powered on VM directly to VMFS at the remote site. We can use vPower NFS (at least I think I understand that correctly) to provide near power on ready VM's, but it's just not the same as having the DR host's inventory populated with the VM's and ready to go from VMFS datastores.

[/color]
That's accurate, but I always like to note that another important aspect to consider is that vPower NFS is not going to let you power on 80+VMs at a time. While there is no "official" limit, realistically vPower NFS performance will likely top out around 5 active VMs. vPower NFS is like a "spare tire" (perhaps one of those little doughnut spares). It can absolutely save you bacon in a bad situation, but you shouldn't build a DR strategy with a tight RTO around it.

• Can't have complex retention policies. It looks like the retention policies are still essentially X number of days. I would really like to see something like X daily, X weekly, X monthly, X yearly, where it rolls up and keeps those historically. That is for another time though...

[/color]
You can do this with manual scripts. Veeam backups are just files so the script doesn't really have to do anything but make copies of full backups and then age them out. Pretty simple. I agree it would be nice to have such retention in the GUI, but for now scripts can handle most requirements.

With regards to using backups for replicas, this is only possible for creating the initial replica. You can take a backup of a bunch of VMs, ship them to the remote site, and then use them to "seed" the replca copies, but from that point they will need to replicate independently.

In most cases to meet all requirements your going to need both backup and replication, although it's possible that the two technologies can complement each other nicely. Typically replication is target when you have systems that need to be up to meet tight RTO's and you have the budget for the hardware and bandwidth. Many customers choose to replicate only a subset of their systems, typically systems that must be up to meet business continuance objectives (i.e. keep making money during a disaster).

On the other hand backups are all about being able to recover data, both from accidents/disasters, but also from simple errors and mistakes. Corrupt Excel spreadsheets, accidentally deleted mail, or VMs that no longer boot or were damaged by an upgrade gone awry. Backups are compressed and deduped but still easy to extract data. They're just files so you can replicate them offsite or via tape.

Most customers I know backup VMs locally, and replicate a subset of the same VMs offsite. They may then additional ship and/or replicate the local backups offsite as well using things as simple as rsync, software to push the files to tape, or dedupe appliance with built-in replication. The first thing you want to do is define your goals, then figure out the right combination of hardware and software to meet those goals.

[jyarborough]

Wow, thanks for the quick detailed response and the clarification. I think I understand your point about the disk size changes but I need to test it further I guess to make it sink in. I definitely consider vPower a spare tire, which is why I would really like to see backup jobs have an additional step that could spawn a replica from the backup. As for the policies and scripts, I have read about those, but I'm not enough of a scripter to make it work for a production environment. Part of me loves when things like this can be scripted, but the other part of me hates the answer "you can script that". I can also write code in assembly language, but I like GUI's

Anyway, I definitely see the benefits of both backups and replicas. Our "goal" is to have all of our VM's replicated offsite at least nightly. I realize that requires a lot so our realistic outcome is probably going to be more like replicating the critical "data heavy" servers offsite nightly and the "access" servers that do not have a lot of unique data every few days or even once a week. That way RTO for critical functionality is keep low and we can always engineer ways to access the data.

I guess in the end I am trying to compare this with AppAssure. I hate their product for a number of reasons, but they did do a few things well. They had retention policies down, and they had a very easy way to backup to disk, then replicate that backup data offsite and then once offsite, create a virtual standby server as a VM. It was sort of cludgy but it did work fairly well.

Thanks again for the reply!

[averylarry]

Changing the size of a disk causes problems every time for my replications going offsite (10 Mb WAN link). I had one that hit a hard timeout limit of 7 days. I always bring the remote server physically back to the main location to re-sync. I try to avoid doing this. If this is a common issue, you can always go with thin disks . . .

My feelings -- replications and backups just do different things. I use both. At one client we do this:

2 SANs.
1 remote site (VPN over internet -- around 6Mb link).

Every night:

1) Backup all the VMs. Keep 120 backups for a decently long history. Great compression/deduplication. Best fit for "traditional" backups and "version history". You can keep a long way back if you want.
2) Replicate each SANs VMs to the other SAN. We do this in the event that we lose a SAN, we can just power up the replica and continue on without having to hassle with the backup/NFS and then migrating or restoring. Especially important since we do NOT have storage vMotion.
3) Replicate everything to the remote site. Purely for disaster recovery.

One thing of note -- having lots of snapshots **can** cause problems and **will** cause slowdowns. I'm going off of memory, but if you search these forums you should find some "practical" limitations on how many replica snapshots you can keep. (Someone correct me if that's not true anymore.)

Summary -- replicas are good/great for WAN applications (disaster recovery) and very fast and easy recovery. Replicas are not as good for individual file restores and generally poor for long-term history and disk space usage. Backups are good/great for compression/deduplication, day-to-day restores, long term history, and backup verification (using surebackup or manual instant restore or file level restore). Backups are not as good at disaster recovery or full VM restores (instant restore is awesome, but you still have to either do a restore or you have to somehow storage vMotion to production storage).


That's just off the top of my head.

[jyarborough]

Awesome, thanks for the feedback. I'm curious how many VM's (or more specifically, how much data) your client is pushing over that 6mbps link each night? We have approximately 8TB's of used space across approximately 80 VM's so that is the biggest reason I worry about this. First from a bandwidth perspective. Second from a processing perspective. I have just setup some throttling to emulate being over our WAN link and doing further testing on our replica jobs while the DR equipment is still physically at our main site. Hopefully these tests prove that we can in fact get everything across our current connections on a nightly basis.

[tsightler]

Changing the size of a disk causes problems every time for my replications going offsite (10 Mb WAN link). I had one that hit a hard timeout limit of 7 days. I always bring the remote server physically back to the main location to re-sync. I try to avoid doing this. If this is a common issue, you can always go with thin disks . . .

If you are still seeing these issues when increasing in disk sizes with 6.1 then you should open a support case. 6.1 properly handles increases in disk space with only the above side effect (all previous restore points are deleted). It even still leverages CBT when the disk size is increased so only changes are sent.

Even with 6.0 it was never required to resync locally as you could always just manually consolidate the snapshots on the replica and perform a replica mapping. This would require a new digests calculation, but assuming properly configured proxies only changed data would traverse the wire.

One thing of note -- having lots of snapshots **can** cause problems and **will** cause slowdowns. I'm going off of memory, but if you search these forums you should find some "practical" limitations on how many replica snapshots you can keep. (Someone correct me if that's not true anymore.)

With Veeam 6.x and VMware replication you are limited to 28 restore points per replica. I would generally suggest keeping the snapshot tree somewhat short as the biggest impact will be if you have to failover for a disaster. There will be a performance impact and if you have to run a long time you will likely need to commit the snapshot tree. Having a long snapshot tree can take a very long time to commit.

[averylarry]

If you are still seeing issues with increasing in disk sizes with 6.1 then you should open a support case. 6.1 properly handles increases in disk space with only the above side effect (all previous restore points are deleted). It even still leverages CBT when the disk size is increased so only changes are sent.

Even with 6.0 it was never required to resync locally as you could always just manually consolidate the snapshots on the replica and perform a replica mapping. This would require a new digests calculation, but assuming properly configured proxies only changed data would traverse the wire.

I haven't tried it with 6.1 yet. But I will in the next week or so. Thanks. (I may not have tried it in v.6.0 either.)

[averylarry]

My largest client has about 5Tb of data on 14 VMs using 5 separate replication jobs. Some of my larger VMs replicate more than once/day.

All the initial replication jobs start between 6pm and 11 pm and they generally all finish before midnight.

My Exchange server (1.3 Tb, ~200 fairly heavy users) doesn't replicated during the work day, but I run it at 6pm (to get the day's changes), 2am (after the nightly maintenance run), and 6am (as close to the start of the day as I dare to minimize the 6pm run). The 6pm run last night ran for 3 hours 15 minutes. It transferred 7.7 Gb across the WAN at 4.6X compression for a total of 35.4Gb of changes.

My main file server runs similar to the Exchange server schedule. It's 1.9Tb. Last night's run went for 1 hour and 35 minutes. It transferred 5.6Gb of data at only 1.7X compression. Not many changes -- only 9.6Gb of changes.

Most of the rest of my servers are in 1 replication job that runs 1 time/day. Last night's run went for 3 hours and 20 minutes. It transferred 5.3 Gb at 31.2X compression (wow) for a total of 165.6Gb of changes. And apparently that's pretty typical for that job (I never looked before). About a week ago this job transferred 21.6Gb at 18.1X compressiong for a total of 392.5Gb of changes in 9 hours.

[averylarry]

Note -- I use "Optimal" compression level and "WAN target" storage optimization.

[Guido]

[merged]

I'm new to veeam B&R and trying to find documentation about mixing replication and backup. What are the rescrictions when using incremental backups in combination with replication.
We are implementing version 6.1 on a Vsphere5 environment.

Thanks

[dellock6]

No restrictions at all, you can mix both as desired. Only remember a single VM can only be processed by a single job at once, so schedule backups and replicas involving the same VM accordingly. It they overlap, the second job will wait for the first one to finish.

Luca.

[Guido]

I also want to put a image backup on tape with HP DataProtector. I have 3 options. Backup the production VM with DP, backup the replica with dp or backup the vbk file from the veeam backup repository. Wich one to choose?

[Vitaliy S.]

The 3rd option is the most preferable one, as you won't need to stress your production storage with another snapshot taken by DP and you will not increase your backup window by backing up the same VM twice.