Comprehensive data protection for all workloads
jfowler
Novice
Posts: 7
Liked: 1 time
Joined: Dec 17, 2009 5:00 pm
Full Name: Jeremy Fowler
Contact:

Offsite Incremental to Synthetic Full Utility

Post by jfowler » 1 person likes this post

If I wanted to just copy over the first .vbk and then all the .vib Incremental files to a cheap NAS at a remote site every night, is there a way I could takes those .vib files and build a synthetic full outside of the backup job? Hopefully something I could easily script? Looking for a way to avoid copying the synthetic full that is created every week locally to the remote site.

Also, Is there a way to specify a location for each of the backup types? Like so:

D:\Backups\Job1.vbk
E:\Incremental\Job12010-11-17T220058.vib
F:\Archives\Job12010-11-11T220058.vrb
Gostev
Chief Product Officer
Posts: 31775
Liked: 7274 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by Gostev »

Hi Jeremy, no it is not possible to build new full outside backup job or specify individual locations for VBK, VIB and VRB files. Please clarify why do you need to create synthetic full offsite?
jfowler
Novice
Posts: 7
Liked: 1 time
Joined: Dec 17, 2009 5:00 pm
Full Name: Jeremy Fowler
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by jfowler »

Gostev wrote:Hi Jeremy, no it is not possible to build new full outside backup job or specify individual locations for VBK, VIB and VRB files.
How do I submit a feature request? :)
Gostev wrote:Please clarify why do you need to create synthetic full offsite?
So I don't have hundreds of incrementals to restore from when disaster occurs or during DR testing. Here is what I want to do:

1. Do Full backup and copy the .vbk file to cheap 4TB NAS (Iomega StorCenter Pro ix4-100) Talking about 2TB of data here...
2. Ship NAS to remote site
3. Incremental backup job locally every weekday night
4. Rsync Incremental files to offsite NAS
5. Synthetic Fulls built on Fridays locally
6. Do the same remotely from a server at the remote site (WITHOUT having to rsync the new .vbk file to save on bandwidth 2+TB of data)
7. Convert incremental to reversal and store as archives on large capacity, lower cost drives (locally only, offsite incrementals are deleted)
8. Sleep soundly knowing my DR plan works
Gostev
Chief Product Officer
Posts: 31775
Liked: 7274 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by Gostev »

Sounds good. Will see what we can do in the next releases to facilitate this.
tsightler
VP, Product Management
Posts: 6034
Liked: 2859 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by tsightler »

This is a great feature request, and one I have previously indicated would be an excellent idea. We currently get similar functionality by basically, running two Veeam jobs, one to local storage, and a second to remote Linux target. That way, synthetic fulls are performed locally and remotely, without significant WAN impact. It would be really nice to, instead of running two jobs, actually rsync the local incremental to the remote site and have the remote site be able to create the synthetic fulls by running some standalone utility.

Of course, a nice, WAN efficient, asynchronous replication built into Veeam would pretty much solve this issue on it's own. :mrgreen:
Mindflux
Enthusiast
Posts: 32
Liked: never
Joined: Nov 10, 2010 7:52 pm
Full Name: RyanW
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by Mindflux »

tsightler wrote:This is a great feature request, and one I have previously indicated would be an excellent idea. We currently get similar functionality by basically, running two Veeam jobs, one to local storage, and a second to remote Linux target. That way, synthetic fulls are performed locally and remotely, without significant WAN impact. It would be really nice to, instead of running two jobs, actually rsync the local incremental to the remote site and have the remote site be able to create the synthetic fulls by running some standalone utility.

Of course, a nice, WAN efficient, asynchronous replication built into Veeam would pretty much solve this issue on it's own. :mrgreen:
There was a post by someone who said all their remote backups were taking forever even though both the source/destination were remote. Support confirmed that the data had to travel TWICE over wan (once each way) to back up with veeam without a local install

How is yours different or am I misunderstanding you?
Gostev
Chief Product Officer
Posts: 31775
Liked: 7274 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by Gostev »

It was me who confirmed, not support :) that topic was about backing up VM located in site B, by backup server located in site A, to a CIFS share located in site B. Tom is talking about backing up VM located in site A, by backup server located in site A, to a Linux (agent enabled) target located in site B.
tsightler
VP, Product Management
Posts: 6034
Liked: 2859 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by tsightler »

Mindflux wrote: There was a post by someone who said all their remote backups were taking forever even though both the source/destination were remote. Support confirmed that the data had to travel TWICE over wan (once each way) to back up with veeam without a local install

How is yours different or am I misunderstanding you?
We currently install a Veeam server at each site so that each of our facilities has a "local" Veeam server. That "local" install pushes the backup to a remote linux target at a DR location, which is the most WAN efficient manner currently available. If we needed to restore one of these "remote" backups at the DR location we simply import the backup into the Veeam install local to the DR site and perform the restore.

Basically, as long as either the source, or the target is local to the Veeam server then the data only traverses the WAN once. In the case you are talking about, both the ESX server, and the backup target (a CIFS share) were at the remote site, while the Veeam server was at the "main" site, thus the traffic had to traverse the WAN twice.
gglynn
Novice
Posts: 5
Liked: never
Joined: Aug 12, 2010 4:54 pm
Full Name: George Glynn
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by gglynn »

Gostev wrote:Sounds good. Will see what we can do in the next releases to facilitate this.
I'm bumping this topic to see if any further consideration has been given to the development of this utility. This would be extremely useful to me and several of my customers.
virtualwatts
Enthusiast
Posts: 50
Liked: never
Joined: Nov 18, 2010 2:41 pm
Full Name: Rick Watts

Re: Offsite Incremental to Synthetic Full Utility

Post by virtualwatts »

Second!! We also could use that utility. That is our Number 2 Enhancement Request - right behind a better UI.
And there I was thinking everyone else has amazing fast WAN speeds and can move 30GB-40GB nightly with ease

Right now we are attempting to run incremental with synth full on local.
We sneakernet the VBK then push the VIB via WAN to DR.
Our next step will be to push the Veeam server and restore it to the DR, where we will attempt a synth full remote from a restored Veeam VM.

We spent quite a bit of time on the phone with Veeam tech going over the details and the response was "well, that should work" and then "technically that is not a supported solution". Oddly, it was the exact use case we presented to the sales guys.

Why? Because Veeam replication is syncronous and our WAN capacity cannot handle the large volumes and long run time.
Our WAN cannot handle the VBK push each night. Backing up from DR would be impractical as well for same reason.
Our ultimate desire is to push VIB files to a DR and have the Veeam in the DR import and synth full.

We are awaiting additional storage in our EMC array on DR before testing this out.

Just FYI - BackupExec 2010 has introduced this functionality in their VMware backup solution, so for no other reason perhaps Veeam should do it to stay ahead of the competition. Another reason of course is that they can sell production licenses for both source and target.

Rick
Gostev
Chief Product Officer
Posts: 31775
Liked: 7274 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by Gostev »

virtualwatts wrote:Our WAN cannot handle the VBK push each night. Backing up from DR would be impractical as well for same reason.
Our ultimate desire is to push VIB files to a DR and have the Veeam in the DR import and synth full.
Hi Rick, so why you cannot use what tsightler had shared above for now?
gglynn
Novice
Posts: 5
Liked: never
Joined: Aug 12, 2010 4:54 pm
Full Name: George Glynn
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by gglynn »

Gostev wrote:Hi Rick, so why you cannot use what tsightler had shared above for now?
I can't speak for Rick, but the reason I can't use tsightler's suggestion is that I need the backup to exist both locally (because it's the only backup software we use, and we might want to use U-AIR to recover something from a backup quickly) and at the DR site, and I don't have time to run the backup twice every time.
Vitaliy S.
VP, Product Management
Posts: 27359
Liked: 2788 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by Vitaliy S. »

George, have you considered using rsync in this case? We recommend using rsync to sync backup files offsite. You can search this forum for offsite and rsync for more information on this.
gglynn
Novice
Posts: 5
Liked: never
Joined: Aug 12, 2010 4:54 pm
Full Name: George Glynn
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by gglynn »

Vitaliy S. wrote:George, have you considered using rsync in this case? We recommend using rsync to sync backup files offsite. You can search this forum for offsite and rsync for more information on this.
We're using rsync now, but it's not ideal. The problem is that rsync is most effective when you're doing reverse incremental backups, because that's the only way you end up with a file with changed blocks so you don't have to replicate the entire full backup over the wire again, which is key, because we don't have a super-fast link to our offsite. This is also why we'd want to keep a local copy of the backup. However, in our opinion, reverse incremental isn't the safest backup mode, because we'd prefer to not touch our VBK if we don't have to.

Synthetic fulls are great, because if the process to create one fails in the middle, you still have the original full VBK and incremental VIB backup files intact and untouched. If we had a standalone utility capable of creating a synthetic full from a full and a series of incrementals, then we could do one full backup when we bring a new VM online, sneakernet it over to the offsite, and then just replicate the smaller nightly (or more often) incrementals to the offsite over the wire, creating synthetic fulls at the main site and the offsite once a week or so, independently, without the need to ship a massive amount of data across the wire. This is ideal because the data we need to do this is already there at the offsite, it's just not in its most useful form without the requested utility.

Personally, I have a Veeam B&R server installed at both the main and offsite locations, so I could even get away with a new job type inside B&R: a standalone "Create Synthetic Full" job. However, I still think it would be easier to just have a standalone utility for most people.
foxj77
Novice
Posts: 4
Liked: never
Joined: Jan 06, 2011 2:33 pm
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by foxj77 »

We are in the same setup as jfowler.

I'd love the feature too!
thomashodek
Lurker
Posts: 1
Liked: never
Joined: Feb 15, 2011 12:35 pm
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by thomashodek »

Add me to that list, I'd really like to see that feature...
arsprod
Enthusiast
Posts: 86
Liked: never
Joined: Jan 21, 2011 6:09 pm
Full Name: Aaron Spiegel
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by arsprod »

Just to keep this conversation going, this is exactly what I need as well. I can't imagine this simple scenerio, using cheap NAS server offsite for DR isn't a priority feature. Rsync is complicated and basically does what Veeam should be able to do... imho!
vipthomps
Service Provider
Posts: 60
Liked: 6 times
Joined: Dec 06, 2010 7:51 pm
Full Name: Eric Thompson
Location: Boston, MA
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by vipthomps »

Agreed this would make lots of sense and would solve an issue i'm looking at now as well.
dasherrill
Novice
Posts: 8
Liked: never
Joined: Mar 29, 2010 4:26 pm
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by dasherrill »

This is the # 1 feature request for us and everyone else I have talked with....
stewks
Lurker
Posts: 2
Liked: never
Joined: Mar 03, 2011 9:30 pm
Full Name: Scott Tewksbury

Re: Offsite Incremental to Synthetic Full Utility

Post by stewks »

Ping! Same issue here on new install. I was quite happy with first veeam backup done locally then incremental by WAN until it hit the first synthetic date and it is taking as long or longer then doing a full backup! This makes Veeam almost a non starter for what we were hoping for, backup to a remote DR site... I'm familiar with linux, but not much detail on using it as WAN host, currently using QNAP NAS at the DR site, which runs on linux but remote agent is apparently not getting installed to do the synthetic fulls remotely.
virtualwatts
Enthusiast
Posts: 50
Liked: never
Joined: Nov 18, 2010 2:41 pm
Full Name: Rick Watts

Re: Offsite Incremental to Synthetic Full Utility

Post by virtualwatts »

gglynn wrote: I can't speak for Rick, but the reason I can't use tsightler's suggestion is that I need the backup to exist both locally (because it's the only backup software we use, and we might want to use U-AIR to recover something from a backup quickly) and at the DR site, and I don't have time to run the backup twice every time.
Actually you did a fantastic job speaking for me. That is exactly the problem.

Veeam replication is syncronous and we don't have the wan for it, backup needs to exists in two separate places and we don't have time to run two separate backups. And running a backup from one site for another would take too long with slow network wan.
stevil
Expert
Posts: 114
Liked: 3 times
Joined: Sep 02, 2010 2:23 pm
Full Name: Steve B
Location: Manchester, UK
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by stevil »

We have the same problem as above. We're trying to backup across the wan incrementally, which works OK. It takes about 1hr 30mins to do the backup each night. When it comes to doing a new full, or synthetic full, we're talking 44hrs+. We're not sure how long it actually takes as it's soooo long that the backup stopped (maybe due to a network glitch).
In it's current state it renders WAN backups useless :( OR am I missing a trick somewhere?

I know we can use rsync to 'fix' this, but this relies on us having enough storage to do the backups locally so that we can rsync them across the WAN, which we don't have in all our cases.

I second the suggestion for an offsite 'rollup' utility or similar. Without it, or something similar, we'll have to rethink our deployment of VB&R across our small sites (of which we have hundreds!)
stevil
Expert
Posts: 114
Liked: 3 times
Joined: Sep 02, 2010 2:23 pm
Full Name: Steve B
Location: Manchester, UK
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by stevil »

reading another thread backup using WAN storrage and synthetic full question it seems as though the answer is to use Linux as a target at the remote end. The agent does all the hard work converting the synthetic backups. I've yet to try this, but it seems to be the answer, assuming you have linux skills. the thread also mentions that this will be available on Windows at some point (!)
Gostev
Chief Product Officer
Posts: 31775
Liked: 7274 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by Gostev »

In v6.
tfleener
Influencer
Posts: 21
Liked: never
Joined: Jun 08, 2010 2:59 pm
Full Name: Tom Fleener
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by tfleener »

Until version 6, does it make sense to use Pull instead of push and take the bandwidth hit on the first full backup ? After that only change blocks are pulled and then the transform works locally.

Would that be a best practice... if you have the bandwidth ? We have 100MB pipe.
We need a windows CIFS backup as we only have BackupExec window agents to get it to tape.
Gostev
Chief Product Officer
Posts: 31775
Liked: 7274 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by Gostev »

Yes, pull would be more efficient in the long run with v5.

Alternatively, you could also backup to Linux as suggested above, and share out the folder with your backups from Linux server via samba. Then any Windows server can read it.
ccrichard
Enthusiast
Posts: 26
Liked: never
Joined: Sep 04, 2010 11:06 pm
Full Name: Richard Yamauchi
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by ccrichard »

Why not just use 2 seperate Veeam backup jobs? One to copy/backup it locally and another to copy it to the remote site? The remote site backup can be a synthetic full so all it will ever send is the changed blocks.
Gostev
Chief Product Officer
Posts: 31775
Liked: 7274 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by Gostev »

That's exactly what was being discussed on the previous page of this topic ;)
tfleener
Influencer
Posts: 21
Liked: never
Joined: Jun 08, 2010 2:59 pm
Full Name: Tom Fleener
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by tfleener »

Main reason for not using two jobs is disk space requirements, we are still a little tight at both ends. Somehow, you never have enough.
bc07
Enthusiast
Posts: 85
Liked: never
Joined: Mar 03, 2011 4:48 pm
Full Name: Enrico
Contact:

Re: Offsite Incremental to Synthetic Full Utility

Post by bc07 »

Gostev wrote:Yes, pull would be more efficient in the long run with v5.
But with that all the data (transfered over WAN) is uncompressed, right? But it should do dedpulication, right?

Enrico
Post Reply

Who is online

Users browsing this forum: mdippold, Paul.Loewenkamp, Semrush [Bot] and 75 guests