Host-based backup of VMware vSphere VMs.
Post Reply
jpiscaer
Enthusiast
Posts: 42
Liked: 9 times
Joined: Jun 16, 2009 11:36 am
Full Name: Joep Piscaer

Replication scenario: touch VM once, replicate off-site

Post by jpiscaer »

Guys,

I have two independent sites (different datacenters, different vCenter / SAN / VMs), both using VMware vSphere 5.
On both sites, I have a Veeam installation (proxy, local repo, Enterprise Manager).
I backup all VMs on either site to the local backup repository with 14 restore points (jobs run dialy).

Currently, I also have added the remote backup server as a 2nd repository to each console and run a clone of each job to this remote repository with 1 single restore point (jobs runs daily). This works fine with v6.1. Heads-up: it doesn't work (AT ALL) with v6.0. It corrupted the databases on both Veeam servers.

Anyway, using this method I can replicate VMs using Veeam's awesome compression and deduplication and using their reverse incremental backup method.
I use this method for the following reasons:
1. Only incremental data travels the 100mbit inter-site link.
2. The replicated data is as small as possible ('best' compression, deduplicated and optimized for 'WAN Target'). This is an explicit requirement stated by the customer to maximize retention of the local backup. Their primary storage (EqualLogic SAN members) is not able to store all the replicated VMs (data); they simply do not have enough space available on the SAN. This means I cannot use the 'replication' job type in Veeam as this would 1. store VMs on production SAN storage and 2. be uncompressed and not deduplicated. I NEED to use the compressed and duplicated Veeam file formats AND store those off-site replicated VMs on the remote backup server.
3. I want only a single restore point to be present on the remote repository; it's for disaster recovery purposes only.

The major down-side of this setup is that those jobs 'touch' the source VMs (i.e. the VMs in the vSphere-environment) twice. Although this setup works brilliantly, I really want to reduce the number of snapshots per VM per run. I now have to cope with two snapshots per VM per night, which reduces the flexibility and possible growth of the backup solution.

I think there's no way to:
1. Touch the source VMs only once (I want 'source -> proxy -> network -> repository' flow to happen only once)
2. Have only incremental data travel the 100mbit inter-site link
3. Store the replicated data in Veeam's compressed and deduplicated file formats (i.e. NOT stored in native VMDK format)
4. Store the replicated data on the remote Veeam backup repository (i.e. NOT stored on production SAN)
5. Only store a single (most recent) restore point.

In my view, Veeam v6.1 is missing a couple of technically not-so-different but hugely important features to make this scenario viable:
1. Backup to two repositories from a single job simultaniously
2. or "Re-play" the recalculation done by the repository (to create a reverse incremental) to an additional (remote) repository.

My questions for you guys: Can I actually accomplish what I want using currently available solutions?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by Vitaliy S. »

Hi Joep,

Yes, there is a solution that would help you to overcome your current challenge (not having an ability to backup to multiple repositories at the same time). Please check out these threads for more info: Best backup setup for mirroring backup repository and V5 VBK file names

And thanks for the feature requests, much appreciated!
jpiscaer
Enthusiast
Posts: 42
Liked: 9 times
Joined: Jun 16, 2009 11:36 am
Full Name: Joep Piscaer

Re: Replication scenario: touch VM once, replicate off-site

Post by jpiscaer »

Hi Vitaliy,

I've read those threads, but I think there's no direct solution for my scenario:
1. With (forward) incremental, I will need to store full + all incrementals until new full on remote site (and replicate a full regularly)
2. With reverse incremental, most (windows based) file sync tools have difficulty determining which data is already at the remote site; this means that I need to transfer way more data than just incremental data.

Do you know any Windows-based file sync tools that'll 'integrate" (replicate only changes) with reverse incremental backups in such a way that I only need to keep one reverse incremental at the remote site?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by Vitaliy S. »

Hi Joep,

Yes, you're correct, the suggested methods are just workarounds, but at least they would allow you to achieve what you want for now.

Cwrsync might be a way to go if you want to run rsync on a Windows machine.

Hope this helps!
jpiscaer
Enthusiast
Posts: 42
Liked: 9 times
Joined: Jun 16, 2009 11:36 am
Full Name: Joep Piscaer

Re: Replication scenario: touch VM once, replicate off-site

Post by jpiscaer »

I'm testdriving Syncrify. My worry is the file renaming on the VBK files. on v6.1, can I change this behavious to have static filenames for the VBK?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by foggy »

Yes, the link provided by Vitaliy in his first post should work.
jpiscaer
Enthusiast
Posts: 42
Liked: 9 times
Joined: Jun 16, 2009 11:36 am
Full Name: Joep Piscaer

Re: Replication scenario: touch VM once, replicate off-site

Post by jpiscaer »

Synrify needs to read every file (local and remote) to determine changed blocks, doesn't save this state to disk. Ergo: every sync run takes ages. That's not acceptable in my environment.

I guess I need Veeam to be able to write to two repositories from the same job, or at least be able to do the recalculation from previous reverse incremental once but write it twice...
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by Vitaliy S. »

Writing backup files to multiple repository servers is on our road map, thanks for the feedback.
ortoscale
Service Provider
Posts: 246
Liked: 20 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by ortoscale »

Hi there,

I've setup new repository with twinstrata trial and create a new job to backup vm there as a second job. Then I've got this infamous move all backup files to the new backup repository first. Does that mean that I cannot use two repositories for backing up VMs? I want to backup all of them locally and in the cloud for DR purposes. Thanks for feedback.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by Vitaliy S. »

Backup job can only use one backup repository at a time, however with a post backup job script you can offload your backup files to the offsite location. Thanks!
ortoscale
Service Provider
Posts: 246
Liked: 20 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by ortoscale »

Vitaliy S. wrote:Backup job can only use one backup repository at a time, however with a post backup job script you can offload your backup files to the offsite location. Thanks!
Hi Vitaly, that's awesome, but what if I want different politics for local and cloud, which is most likely in most scenarios ?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by foggy »

bc2011 wrote:I've setup new repository with twinstrata trial and create a new job to backup vm there as a second job. Then I've got this infamous move all backup files to the new backup repository first. Does that mean that I cannot use two repositories for backing up VMs? I want to backup all of them locally and in the cloud for DR purposes. Thanks for feedback.
If you use two separate jobs, then you are ok with two different repositories. What is the exact procedure that leads you to the "move all backup files..." error?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by dellock6 »

Some considerations: Veeam uses incremental jobs, not differentials, so you cannot replicate a full backup and only some daily files, you need the complete chain. So, based on maths you have to do about bandwidth, backup files size, deduplication and others, you can for example use forward incremental locally (avoid reverse if replicating offsite since the full backup is a new file everyday and you would need to replicate it everyday...), and then replicate only the weekly full offsite.
Maybe a deduplicated storage on DR can help, now you can also use Windows 2012 instead of usual hardware appliances.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
ortoscale
Service Provider
Posts: 246
Liked: 20 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by ortoscale »

foggy wrote: If you use two separate jobs, then you are ok with two different repositories. What is the exact procedure that leads you to the "move all backup files..." error?
When you create secondary job with secondary repository and try to select VM that is present in primary job the mesage occurs : move all backup files to new backup repository first. Correct me on this one if possible.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by foggy »

I cannot reproduce this behavior in my lab. I can successfully create a new job, add the VM that is already backed up in another job, and specify another backup repository. I would suggest to open a case and ask our technical support staff to review your setup via webex, that seems to be the most efficient way to tell what's wrong here.
ortoscale
Service Provider
Posts: 246
Liked: 20 times
Joined: Aug 02, 2011 9:30 pm
Full Name: Matjaž Antloga
Location: Celje, Slovenia
Contact:

Re: Replication scenario: touch VM once, replicate off-site

Post by ortoscale »

foggy wrote:I cannot reproduce this behavior in my lab. I can successfully create a new job, add the VM that is already backed up in another job, and specify another backup repository. I would suggest to open a case and ask our technical support staff to review your setup via webex, that seems to be the most efficient way to tell what's wrong here.
Hi foggy. You are absolutely right. I cannot reproduce the same problem again. I really have no idea why this message shows up, because I was getting it when creating a new job and using new repository.Now few days later message doesn't show up. I would like to reproduce it but I can't. It's not shown in logs either.
Post Reply

Who is online

Users browsing this forum: dbeerts and 77 guests