Replication scenario: touch VM once, replicate off-site

[jpiscaer]

Guys,

I have two independent sites (different datacenters, different vCenter / SAN / VMs), both using VMware vSphere 5.
On both sites, I have a Veeam installation (proxy, local repo, Enterprise Manager).
I backup all VMs on either site to the local backup repository with 14 restore points (jobs run dialy).

Currently, I also have added the remote backup server as a 2nd repository to each console and run a clone of each job to this remote repository with 1 single restore point (jobs runs daily). This works fine with v6.1. Heads-up: it doesn't work (AT ALL) with v6.0. It corrupted the databases on both Veeam servers.

Anyway, using this method I can replicate VMs using Veeam's awesome compression and deduplication and using their reverse incremental backup method.
I use this method for the following reasons:
1. Only incremental data travels the 100mbit inter-site link.
2. The replicated data is as small as possible ('best' compression, deduplicated and optimized for 'WAN Target'). This is an explicit requirement stated by the customer to maximize retention of the local backup. Their primary storage (EqualLogic SAN members) is not able to store all the replicated VMs (data); they simply do not have enough space available on the SAN. This means I cannot use the 'replication' job type in Veeam as this would 1. store VMs on production SAN storage and 2. be uncompressed and not deduplicated. I NEED to use the compressed and duplicated Veeam file formats AND store those off-site replicated VMs on the remote backup server.
3. I want only a single restore point to be present on the remote repository; it's for disaster recovery purposes only.

The major down-side of this setup is that those jobs 'touch' the source VMs (i.e. the VMs in the vSphere-environment) twice. Although this setup works brilliantly, I really want to reduce the number of snapshots per VM per run. I now have to cope with two snapshots per VM per night, which reduces the flexibility and possible growth of the backup solution.

I think there's no way to:
1. Touch the source VMs only once (I want 'source -> proxy -> network -> repository' flow to happen only once)
2. Have only incremental data travel the 100mbit inter-site link
3. Store the replicated data in Veeam's compressed and deduplicated file formats (i.e. NOT stored in native VMDK format)
4. Store the replicated data on the remote Veeam backup repository (i.e. NOT stored on production SAN)
5. Only store a single (most recent) restore point.

In my view, Veeam v6.1 is missing a couple of technically not-so-different but hugely important features to make this scenario viable:
1. Backup to two repositories from a single job simultaniously
2. or "Re-play" the recalculation done by the repository (to create a reverse incremental) to an additional (remote) repository.

My questions for you guys: Can I actually accomplish what I want using currently available solutions?

[Vitaliy S.]

Hi Joep,

Yes, there is a solution that would help you to overcome your current challenge (not having an ability to backup to multiple repositories at the same time). Please check out these threads for more info: Best backup setup for mirroring backup repository and V5 VBK file names

And thanks for the feature requests, much appreciated!

[jpiscaer]

Hi Vitaliy,

I've read those threads, but I think there's no direct solution for my scenario:
1. With (forward) incremental, I will need to store full + all incrementals until new full on remote site (and replicate a full regularly)
2. With reverse incremental, most (windows based) file sync tools have difficulty determining which data is already at the remote site; this means that I need to transfer way more data than just incremental data.

Do you know any Windows-based file sync tools that'll 'integrate" (replicate only changes) with reverse incremental backups in such a way that I only need to keep one reverse incremental at the remote site?

[Vitaliy S.]

Hi Joep,

Yes, you're correct, the suggested methods are just workarounds, but at least they would allow you to achieve what you want for now.

Cwrsync might be a way to go if you want to run rsync on a Windows machine.

Hope this helps!

[jpiscaer]

I'm testdriving Syncrify. My worry is the file renaming on the VBK files. on v6.1, can I change this behavious to have static filenames for the VBK?

[foggy]

Yes, the link provided by Vitaliy in his first post should work.

[jpiscaer]

Synrify needs to read every file (local and remote) to determine changed blocks, doesn't save this state to disk. Ergo: every sync run takes ages. That's not acceptable in my environment.

I guess I need Veeam to be able to write to two repositories from the same job, or at least be able to do the recalculation from previous reverse incremental once but write it twice...

[Vitaliy S.]

Writing backup files to multiple repository servers is on our road map, thanks for the feedback.

[ortoscale]

Hi there,

I've setup new repository with twinstrata trial and create a new job to backup vm there as a second job. Then I've got this infamous move all backup files to the new backup repository first. Does that mean that I cannot use two repositories for backing up VMs? I want to backup all of them locally and in the cloud for DR purposes. Thanks for feedback.

[Vitaliy S.]

Backup job can only use one backup repository at a time, however with a post backup job script you can offload your backup files to the offsite location. Thanks!

[ortoscale]

Backup job can only use one backup repository at a time, however with a post backup job script you can offload your backup files to the offsite location. Thanks!

Hi Vitaly, that's awesome, but what if I want different politics for local and cloud, which is most likely in most scenarios ?

[foggy]

I've setup new repository with twinstrata trial and create a new job to backup vm there as a second job. Then I've got this infamous move all backup files to the new backup repository first. Does that mean that I cannot use two repositories for backing up VMs? I want to backup all of them locally and in the cloud for DR purposes. Thanks for feedback.

If you use two separate jobs, then you are ok with two different repositories. What is the exact procedure that leads you to the "move all backup files..." error?

[dellock6]

Some considerations: Veeam uses incremental jobs, not differentials, so you cannot replicate a full backup and only some daily files, you need the complete chain. So, based on maths you have to do about bandwidth, backup files size, deduplication and others, you can for example use forward incremental locally (avoid reverse if replicating offsite since the full backup is a new file everyday and you would need to replicate it everyday...), and then replicate only the weekly full offsite.
Maybe a deduplicated storage on DR can help, now you can also use Windows 2012 instead of usual hardware appliances.

Luca.

[ortoscale]

If you use two separate jobs, then you are ok with two different repositories. What is the exact procedure that leads you to the "move all backup files..." error?

When you create secondary job with secondary repository and try to select VM that is present in primary job the mesage occurs : move all backup files to new backup repository first. Correct me on this one if possible.

[foggy]

I cannot reproduce this behavior in my lab. I can successfully create a new job, add the VM that is already backed up in another job, and specify another backup repository. I would suggest to open a case and ask our technical support staff to review your setup via webex, that seems to be the most efficient way to tell what's wrong here.

[ortoscale]

I cannot reproduce this behavior in my lab. I can successfully create a new job, add the VM that is already backed up in another job, and specify another backup repository. I would suggest to open a case and ask our technical support staff to review your setup via webex, that seems to be the most efficient way to tell what's wrong here.

Hi foggy. You are absolutely right. I cannot reproduce the same problem again. I really have no idea why this message shows up, because I was getting it when creating a new job and using new repository.Now few days later message doesn't show up. I would like to reproduce it but I can't. It's not shown in logs either.