Offsite Incremental to Synthetic Full Utility

[jfowler]

If I wanted to just copy over the first .vbk and then all the .vib Incremental files to a cheap NAS at a remote site every night, is there a way I could takes those .vib files and build a synthetic full outside of the backup job? Hopefully something I could easily script? Looking for a way to avoid copying the synthetic full that is created every week locally to the remote site.

Also, Is there a way to specify a location for each of the backup types? Like so:

D:\Backups\Job1.vbk
E:\Incremental\Job12010-11-17T220058.vib
F:\Archives\Job12010-11-11T220058.vrb

[Gostev]

Hi Jeremy, no it is not possible to build new full outside backup job or specify individual locations for VBK, VIB and VRB files. Please clarify why do you need to create synthetic full offsite?

[jfowler]

Hi Jeremy, no it is not possible to build new full outside backup job or specify individual locations for VBK, VIB and VRB files.

How do I submit a feature request?

Please clarify why do you need to create synthetic full offsite?

So I don't have hundreds of incrementals to restore from when disaster occurs or during DR testing. Here is what I want to do:

1. Do Full backup and copy the .vbk file to cheap 4TB NAS (Iomega StorCenter Pro ix4-100) Talking about 2TB of data here...
2. Ship NAS to remote site
3. Incremental backup job locally every weekday night
4. Rsync Incremental files to offsite NAS
5. Synthetic Fulls built on Fridays locally
6. Do the same remotely from a server at the remote site (WITHOUT having to rsync the new .vbk file to save on bandwidth 2+TB of data)
7. Convert incremental to reversal and store as archives on large capacity, lower cost drives (locally only, offsite incrementals are deleted)
8. Sleep soundly knowing my DR plan works

[Gostev]

Sounds good. Will see what we can do in the next releases to facilitate this.

[tsightler]

This is a great feature request, and one I have previously indicated would be an excellent idea. We currently get similar functionality by basically, running two Veeam jobs, one to local storage, and a second to remote Linux target. That way, synthetic fulls are performed locally and remotely, without significant WAN impact. It would be really nice to, instead of running two jobs, actually rsync the local incremental to the remote site and have the remote site be able to create the synthetic fulls by running some standalone utility.

Of course, a nice, WAN efficient, asynchronous replication built into Veeam would pretty much solve this issue on it's own.

[Mindflux]

This is a great feature request, and one I have previously indicated would be an excellent idea. We currently get similar functionality by basically, running two Veeam jobs, one to local storage, and a second to remote Linux target. That way, synthetic fulls are performed locally and remotely, without significant WAN impact. It would be really nice to, instead of running two jobs, actually rsync the local incremental to the remote site and have the remote site be able to create the synthetic fulls by running some standalone utility.

Of course, a nice, WAN efficient, asynchronous replication built into Veeam would pretty much solve this issue on it's own.

There was a post by someone who said all their remote backups were taking forever even though both the source/destination were remote. Support confirmed that the data had to travel TWICE over wan (once each way) to back up with veeam without a local install

How is yours different or am I misunderstanding you?

[Gostev]

It was me who confirmed, not support that topic was about backing up VM located in site B, by backup server located in site A, to a CIFS share located in site B. Tom is talking about backing up VM located in site A, by backup server located in site A, to a Linux (agent enabled) target located in site B.

[tsightler]

There was a post by someone who said all their remote backups were taking forever even though both the source/destination were remote. Support confirmed that the data had to travel TWICE over wan (once each way) to back up with veeam without a local install

How is yours different or am I misunderstanding you?

We currently install a Veeam server at each site so that each of our facilities has a "local" Veeam server. That "local" install pushes the backup to a remote linux target at a DR location, which is the most WAN efficient manner currently available. If we needed to restore one of these "remote" backups at the DR location we simply import the backup into the Veeam install local to the DR site and perform the restore.

Basically, as long as either the source, or the target is local to the Veeam server then the data only traverses the WAN once. In the case you are talking about, both the ESX server, and the backup target (a CIFS share) were at the remote site, while the Veeam server was at the "main" site, thus the traffic had to traverse the WAN twice.

[gglynn]

Sounds good. Will see what we can do in the next releases to facilitate this.

I'm bumping this topic to see if any further consideration has been given to the development of this utility. This would be extremely useful to me and several of my customers.

[virtualwatts]

Second!! We also could use that utility. That is our Number 2 Enhancement Request - right behind a better UI.
And there I was thinking everyone else has amazing fast WAN speeds and can move 30GB-40GB nightly with ease

Right now we are attempting to run incremental with synth full on local.
We sneakernet the VBK then push the VIB via WAN to DR.
Our next step will be to push the Veeam server and restore it to the DR, where we will attempt a synth full remote from a restored Veeam VM.

We spent quite a bit of time on the phone with Veeam tech going over the details and the response was "well, that should work" and then "technically that is not a supported solution". Oddly, it was the exact use case we presented to the sales guys.

Why? Because Veeam replication is syncronous and our WAN capacity cannot handle the large volumes and long run time.
Our WAN cannot handle the VBK push each night. Backing up from DR would be impractical as well for same reason.
Our ultimate desire is to push VIB files to a DR and have the Veeam in the DR import and synth full.

We are awaiting additional storage in our EMC array on DR before testing this out.

Just FYI - BackupExec 2010 has introduced this functionality in their VMware backup solution, so for no other reason perhaps Veeam should do it to stay ahead of the competition. Another reason of course is that they can sell production licenses for both source and target.

Rick

[Gostev]

Our WAN cannot handle the VBK push each night. Backing up from DR would be impractical as well for same reason.
Our ultimate desire is to push VIB files to a DR and have the Veeam in the DR import and synth full.

Hi Rick, so why you cannot use what tsightler had shared above for now?

[gglynn]

Hi Rick, so why you cannot use what tsightler had shared above for now?

I can't speak for Rick, but the reason I can't use tsightler's suggestion is that I need the backup to exist both locally (because it's the only backup software we use, and we might want to use U-AIR to recover something from a backup quickly) and at the DR site, and I don't have time to run the backup twice every time.

[Vitaliy S.]

George, have you considered using rsync in this case? We recommend using rsync to sync backup files offsite. You can search this forum for offsite and rsync for more information on this.

[gglynn]

George, have you considered using rsync in this case? We recommend using rsync to sync backup files offsite. You can search this forum for offsite and rsync for more information on this.

We're using rsync now, but it's not ideal. The problem is that rsync is most effective when you're doing reverse incremental backups, because that's the only way you end up with a file with changed blocks so you don't have to replicate the entire full backup over the wire again, which is key, because we don't have a super-fast link to our offsite. This is also why we'd want to keep a local copy of the backup. However, in our opinion, reverse incremental isn't the safest backup mode, because we'd prefer to not touch our VBK if we don't have to.

Synthetic fulls are great, because if the process to create one fails in the middle, you still have the original full VBK and incremental VIB backup files intact and untouched. If we had a standalone utility capable of creating a synthetic full from a full and a series of incrementals, then we could do one full backup when we bring a new VM online, sneakernet it over to the offsite, and then just replicate the smaller nightly (or more often) incrementals to the offsite over the wire, creating synthetic fulls at the main site and the offsite once a week or so, independently, without the need to ship a massive amount of data across the wire. This is ideal because the data we need to do this is already there at the offsite, it's just not in its most useful form without the requested utility.

Personally, I have a Veeam B&R server installed at both the main and offsite locations, so I could even get away with a new job type inside B&R: a standalone "Create Synthetic Full" job. However, I still think it would be easier to just have a standalone utility for most people.

[foxj77]

We are in the same setup as jfowler.

I'd love the feature too!

[thomashodek]

Add me to that list, I'd really like to see that feature...

[arsprod]

Just to keep this conversation going, this is exactly what I need as well. I can't imagine this simple scenerio, using cheap NAS server offsite for DR isn't a priority feature. Rsync is complicated and basically does what Veeam should be able to do... imho!

[vipthomps]

Agreed this would make lots of sense and would solve an issue i'm looking at now as well.

[dasherrill]

This is the # 1 feature request for us and everyone else I have talked with....

[stewks]

Ping! Same issue here on new install. I was quite happy with first veeam backup done locally then incremental by WAN until it hit the first synthetic date and it is taking as long or longer then doing a full backup! This makes Veeam almost a non starter for what we were hoping for, backup to a remote DR site... I'm familiar with linux, but not much detail on using it as WAN host, currently using QNAP NAS at the DR site, which runs on linux but remote agent is apparently not getting installed to do the synthetic fulls remotely.

[virtualwatts]

I can't speak for Rick, but the reason I can't use tsightler's suggestion is that I need the backup to exist both locally (because it's the only backup software we use, and we might want to use U-AIR to recover something from a backup quickly) and at the DR site, and I don't have time to run the backup twice every time.

Actually you did a fantastic job speaking for me. That is exactly the problem.

Veeam replication is syncronous and we don't have the wan for it, backup needs to exists in two separate places and we don't have time to run two separate backups. And running a backup from one site for another would take too long with slow network wan.

[stevil]

We have the same problem as above. We're trying to backup across the wan incrementally, which works OK. It takes about 1hr 30mins to do the backup each night. When it comes to doing a new full, or synthetic full, we're talking 44hrs+. We're not sure how long it actually takes as it's soooo long that the backup stopped (maybe due to a network glitch).
In it's current state it renders WAN backups useless OR am I missing a trick somewhere?

I know we can use rsync to 'fix' this, but this relies on us having enough storage to do the backups locally so that we can rsync them across the WAN, which we don't have in all our cases.

I second the suggestion for an offsite 'rollup' utility or similar. Without it, or something similar, we'll have to rethink our deployment of VB&R across our small sites (of which we have hundreds!)

[stevil]

reading another thread backup using WAN storrage and synthetic full question it seems as though the answer is to use Linux as a target at the remote end. The agent does all the hard work converting the synthetic backups. I've yet to try this, but it seems to be the answer, assuming you have linux skills. the thread also mentions that this will be available on Windows at some point (!)

[Gostev]

In v6.

[tfleener]

Until version 6, does it make sense to use Pull instead of push and take the bandwidth hit on the first full backup ? After that only change blocks are pulled and then the transform works locally.

Would that be a best practice... if you have the bandwidth ? We have 100MB pipe.
We need a windows CIFS backup as we only have BackupExec window agents to get it to tape.

[Gostev]

Yes, pull would be more efficient in the long run with v5.

Alternatively, you could also backup to Linux as suggested above, and share out the folder with your backups from Linux server via samba. Then any Windows server can read it.

[ccrichard]

Why not just use 2 seperate Veeam backup jobs? One to copy/backup it locally and another to copy it to the remote site? The remote site backup can be a synthetic full so all it will ever send is the changed blocks.

[Gostev]

That's exactly what was being discussed on the previous page of this topic

[tfleener]

Main reason for not using two jobs is disk space requirements, we are still a little tight at both ends. Somehow, you never have enough.

[bc07]

Yes, pull would be more efficient in the long run with v5.

But with that all the data (transfered over WAN) is uncompressed, right? But it should do dedpulication, right?

Enrico