Offsite Incremental to Synthetic Full Utility

Availability for the Always-On Enterprise

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby Vitaliy S. » Fri Apr 29, 2011 11:59 am

Yes, with backup server located on the target site you'll get no network traffic compression. And we won't be transmitting the same blocks compared to the previous backup runs.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby pvz » Sat May 28, 2011 11:51 am

A utility such as this would be incredibly useful in an environment I'm currently setting up.

I'd envision something like this:

1. Veeam is set up backing up in incremental mode, backing up on-site VMs onto on-site disk storage.
2. After the backup job is complete, the latest incremental file is copied off-site over the network.
3. When the file is copied, a program is run, both on the local and remote sites, which changes the direction of the full+incremental to a reverse incremental+synthetic full.
4. Old reverse incrementals are pruned according to retention requirements.
5. Md5sum are run on all files, checking whether there is a mismatch between remote and local. The administrator is notified if that is the case.

I'll create a simple chart to show the principle.

Legend:
F = Synthetic full backup
R = Reverse incremental
I = Incremental
_ = Missing file


Code: Select all
Weekday:                                                  1234567
Local site, before backup:                                RRRRRF_
Remote site, before backup:                               RRRRRF_
Local site, after backup:                                 RRRRRFI
Remote site, after incremental is copied:                 RRRRRFI
Local site, after consolidation job is run:               RRRRRRF
Remote site, after consolidation job is run:              RRRRRRF
Local, after old reverse incrementals have been pruned:   _RRRRRF
Remote, after old reverse incrementals have been pruned:  _RRRRRF


So basically, the backup job would be creating forward incrementals based on the latest full backup (which will actually be a synthetic full, except when the system is just being started). The incremental would then be copied to the remote site, and a utility on both sites would massage the backup chain creating a reverse-incremental chain including the new full backup.

This to me would combine the best of both worlds regarding incremental and reverse incremental backups, and all of this could be done without having to touch the VMs we're backing up more than once per cycle.
pvz
Novice
 
Posts: 4
Liked: never
Joined: Sat May 28, 2011 10:12 am
Full Name: Per von Zweigbergk

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby marine » Mon May 30, 2011 3:10 pm

Here! Here! I am lucky to have a 1Gb link to my DR site but it is a complete waste of bandwidth to push a huge VBK file through the pipe every weekend when the local synthetic full happens. There REALLY needs to be some way to sneakernet the original VBK to the DR site and then simply feed the DR site with the incremental VIB files each night. And have some utility on the DR site that can create the synthetic fulls.

This is a major issue that should be very easy to implement.
marine
Novice
 
Posts: 8
Liked: never
Joined: Fri Apr 08, 2011 3:19 pm

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby pvz » Tue May 31, 2011 11:36 am

After brainstorming with a colleague, I think we have found a way to leverage filesystems with snapshots to workaround this issue. Please note that this hasn't been tested yet, and it's still in the idea stage with us.

We'd put our Veeam backup system into reverse-incremental mode.

We'd make a post-backup script that made the following things happen:

- A filesystem level snapshot is taken on the remote file server.
- As required, old snapshots are deleted to free up space on the remote file server.
- Rsync is used to synchronize the most recent VBK file with the current VBK file on the remote file server, overwriting the current VBK file. But that's OK, the less-recent VBK files are still present as filesystem snapshots, ready to be taken out as needed! And since the snapshots are block-level, we even end up with some cheap deduplication.

Does anyone see any potential problems with this approach? We'd probably end up building a cheap FreeBSD+ZFS based storage server to do this, since ZFS has the features we need (snapshots) and also aggressively combats bit rot on the offsite backup disks.
pvz
Novice
 
Posts: 4
Liked: never
Joined: Sat May 28, 2011 10:12 am
Full Name: Per von Zweigbergk

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby Joe_L » Tue May 31, 2011 3:54 pm

Hear Hear to PVZ's post. The one slight difference I would want is step 2 goes to tape instead of offsite DR.

I guess whether people use offsite DR or tape the principal is the same: (1) the generation of online/onsite synthetic plus reverse incrementals for speedy restore & efficient storage, (2) and (relatively) low-bandwidth forward incrementals to secondary backup whether it's tape or offsite DR.
Joe_L
Influencer
 
Posts: 17
Liked: 2 times
Joined: Tue May 31, 2011 3:21 pm
Full Name: Joe Lenartowicz

Veeam in an MSP scenario

Veeam Logoby jimmymc » Mon Oct 10, 2011 3:38 pm

[merged]

Hi

This follows on slightly from a few existing posts but doesn't seem to fix exactly into any one of them:

The most simple and bandwidth friendly way for Veeam backing up to an MSP, would seem to be a full, followed by incremental forever.. but this is trouble in the long run as not only would you eventually run out of space; but just as important - the more VIBs you have, the more chance you have of one of them becoming corrupt, thus ruining the chain.

I had a some thoughts about how it would work from an MSP perspective:

- Local Veeam runs a full backup
- Resulting VBK can be sent to remote site via USB drive and copied into a 'customer folder'
- Local Veeam runs an incremental
- Resulting VIB can be sent to remote site and dropped into the 'customer folder' via some method over the Internet - FTP, SFTP, FTPS or whatever (but to be scaleable, would need to be over a 'standard internet protocol' as opposed to copying to a fileshare which means VPN (for example)- and this isn't that scaleable and as self-managing considering interoperable issues with different FW/VPN vendors)
- A 'standalone' Veeam service at remote site, creates a synthetic full out of the original VBK and any subsequent VIBs in the same folder (i.e. every new VIB that turns up in that folder, is injected into the VBK that also resides in said folder)
- Locally, you could still run synthetic fulls on a regular basis, but just as long as the VIB before the local synthetic full starts, is copied to the remote site (or a 'transfer/drop folder')

Seems to me, there are no fundemental changes in the technology required.. Veeam already have the ability to create synthetic fulls, so maybe there can be a 'standalone' service to accomplish this on the remote site?

The copying of the VIBs doesn't really need to be anything Veeam-coded either (although it would be cleaner in the long run); as there are plenty of products out there to manage things like this - a favourite of mine is Super Flexible File Sync.

Don't know what everyone else's thoughts are, and I may be way off base but I thought I'd inject my two-penneth..

Cheers, James
jimmymc
Service Provider
 
Posts: 29
Liked: 3 times
Joined: Thu Dec 09, 2010 3:06 pm

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby jp34 » Mon Oct 17, 2011 9:55 am

Hi,
I have not found a reliable answer to this question... building a full backup from previous full backup and increments will be possible in v6 ?
Please could you confirm ?
Thanks.
jp34
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Tue Dec 22, 2009 8:31 am
Full Name: JP

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby foggy » Mon Oct 17, 2011 11:07 am

New version will allow to implement forever-incremental backup with synthetic fulls assembled locally on the backup target. Thanks.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby jp34 » Mon Jan 09, 2012 10:15 am

And how to do synthetic fulls assembled REMOTELY on another PC ?
In other word, I have copied a vbk file and a vib file to a remote PC, I'd like to rebuild a new vbk including vib data. How can I do ?
Cheers, JP.
jp34
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Tue Dec 22, 2009 8:31 am
Full Name: JP

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby Vitaliy S. » Mon Jan 09, 2012 9:20 pm

JP, basically this is achieved with the help of remote backup repositories. All you need to do is to deploy a backup repository, move backup files to this repository, rescan it to locate backup files and then map these files for existing backup jobs. This would allow you to keep synthetic full traffic locally.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Veeam in an MSP scenario

Veeam Logoby jp34 » Mon Jan 09, 2012 10:12 pm

Ok I think I've not been clear... Jimmy's explanation is better (previous post in this page) :

jimmymc wrote:[merged]
- Local Veeam runs a full backup
- Resulting VBK can be sent to remote site via USB drive and copied into a 'customer folder'
- Local Veeam runs an incremental
- Resulting VIB can be sent to remote site and dropped into the 'customer folder' via some method over the Internet - FTP, SFTP, FTPS or whatever (but to be scaleable, would need to be over a 'standard internet protocol' as opposed to copying to a fileshare which means VPN (for example)- and this isn't that scaleable and as self-managing considering interoperable issues with different FW/VPN vendors)
- A 'standalone' Veeam service at remote site, creates a synthetic full out of the original VBK and any subsequent VIBs in the same folder (i.e. every new VIB that turns up in that folder, is injected into the VBK that also resides in said folder)
- Locally, you could still run synthetic fulls on a regular basis, but just as long as the VIB before the local synthetic full starts, is copied to the remote site (or a 'transfer/drop folder')


It's exactly what I need, how to do that ?
jp34
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Tue Dec 22, 2009 8:31 am
Full Name: JP

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby jp34 » Tue Jan 10, 2012 7:31 am

Basically a simple command line utility could do it at remote DR site:
mergeutility "basevbkfile" "vibfile" "syntheticvbkfile"
basevbkfile: path of the vbk file generated by veeam (old synthetic or full backup)
vibfile: path of the vib file generated by veeam
syntheticvbkfile: path of the resulting vbk file (new vbk including vib data)

In this way, we would have to send 1 vbk file once to DR site (via USB drive) and only 1 vib file periodically... (no more big vbk transfert consuming bandwidth and time)
jp34
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Tue Dec 22, 2009 8:31 am
Full Name: JP

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby Vitaliy S. » Tue Jan 10, 2012 8:37 am

There is no standalone tool to create synthetic backup on demand, but the procedure I outlined above should help you to achieve what you want.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby jp34 » Tue Jan 10, 2012 9:34 am

Ok, I read many times your procedure... and I think I've understood now :)
I'm going to look at this.
Thanks.
jp34
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Tue Dec 22, 2009 8:31 am
Full Name: JP

Re: Offsite Incremental to Synthetic Full Utility

Veeam Logoby jp34 » Tue Jan 10, 2012 4:16 pm

What I've done...

On production site:
- Veeam has been installed
- A incremental backup job has been created and launched -> 1 vbk file created
- The backup job is launched on more time -> 1 vbi file created

On DR site:
- Another Veeam has been installed
- A DR repository has been created
- vbm, vbk and vib files have been copied from production repository to DR repository
- DR repository has been rescanned -> backup with 1 restore point imported

Now, how to generate a synthetic full on DR site ?

Thanks in advance.
jp34
Enthusiast
 
Posts: 47
Liked: 1 time
Joined: Tue Dec 22, 2009 8:31 am
Full Name: JP

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot] and 36 guests