Offsite Incremental to Synthetic Full Utility

[Vitaliy S.]

Yes, with backup server located on the target site you'll get no network traffic compression. And we won't be transmitting the same blocks compared to the previous backup runs.

[pvz]

A utility such as this would be incredibly useful in an environment I'm currently setting up.

I'd envision something like this:

1. Veeam is set up backing up in incremental mode, backing up on-site VMs onto on-site disk storage.
2. After the backup job is complete, the latest incremental file is copied off-site over the network.
3. When the file is copied, a program is run, both on the local and remote sites, which changes the direction of the full+incremental to a reverse incremental+synthetic full.
4. Old reverse incrementals are pruned according to retention requirements.
5. Md5sum are run on all files, checking whether there is a mismatch between remote and local. The administrator is notified if that is the case.

I'll create a simple chart to show the principle.

Legend:
F = Synthetic full backup
R = Reverse incremental
I = Incremental
_ = Missing file

Code: Select all

Weekday:                                                  1234567
Local site, before backup:                                RRRRRF_
Remote site, before backup:                               RRRRRF_
Local site, after backup:                                 RRRRRFI
Remote site, after incremental is copied:                 RRRRRFI
Local site, after consolidation job is run:               RRRRRRF
Remote site, after consolidation job is run:              RRRRRRF
Local, after old reverse incrementals have been pruned:   _RRRRRF
Remote, after old reverse incrementals have been pruned:  _RRRRRF

So basically, the backup job would be creating forward incrementals based on the latest full backup (which will actually be a synthetic full, except when the system is just being started). The incremental would then be copied to the remote site, and a utility on both sites would massage the backup chain creating a reverse-incremental chain including the new full backup.

This to me would combine the best of both worlds regarding incremental and reverse incremental backups, and all of this could be done without having to touch the VMs we're backing up more than once per cycle.

[marine]

Here! Here! I am lucky to have a 1Gb link to my DR site but it is a complete waste of bandwidth to push a huge VBK file through the pipe every weekend when the local synthetic full happens. There REALLY needs to be some way to sneakernet the original VBK to the DR site and then simply feed the DR site with the incremental VIB files each night. And have some utility on the DR site that can create the synthetic fulls.

This is a major issue that should be very easy to implement.

[pvz]

After brainstorming with a colleague, I think we have found a way to leverage filesystems with snapshots to workaround this issue. Please note that this hasn't been tested yet, and it's still in the idea stage with us.

We'd put our Veeam backup system into reverse-incremental mode.

We'd make a post-backup script that made the following things happen:

- A filesystem level snapshot is taken on the remote file server.
- As required, old snapshots are deleted to free up space on the remote file server.
- Rsync is used to synchronize the most recent VBK file with the current VBK file on the remote file server, overwriting the current VBK file. But that's OK, the less-recent VBK files are still present as filesystem snapshots, ready to be taken out as needed! And since the snapshots are block-level, we even end up with some cheap deduplication.

Does anyone see any potential problems with this approach? We'd probably end up building a cheap FreeBSD+ZFS based storage server to do this, since ZFS has the features we need (snapshots) and also aggressively combats bit rot on the offsite backup disks.

[Joe_L]

Hear Hear to PVZ's post. The one slight difference I would want is step 2 goes to tape instead of offsite DR.

I guess whether people use offsite DR or tape the principal is the same: (1) the generation of online/onsite synthetic plus reverse incrementals for speedy restore & efficient storage, (2) and (relatively) low-bandwidth forward incrementals to secondary backup whether it's tape or offsite DR.

[jimmymc]

[merged]

Hi

This follows on slightly from a few existing posts but doesn't seem to fix exactly into any one of them:

The most simple and bandwidth friendly way for Veeam backing up to an MSP, would seem to be a full, followed by incremental forever.. but this is trouble in the long run as not only would you eventually run out of space; but just as important - the more VIBs you have, the more chance you have of one of them becoming corrupt, thus ruining the chain.

I had a some thoughts about how it would work from an MSP perspective:

- Local Veeam runs a full backup
- Resulting VBK can be sent to remote site via USB drive and copied into a 'customer folder'
- Local Veeam runs an incremental
- Resulting VIB can be sent to remote site and dropped into the 'customer folder' via some method over the Internet - FTP, SFTP, FTPS or whatever (but to be scaleable, would need to be over a 'standard internet protocol' as opposed to copying to a fileshare which means VPN (for example)- and this isn't that scaleable and as self-managing considering interoperable issues with different FW/VPN vendors)
- A 'standalone' Veeam service at remote site, creates a synthetic full out of the original VBK and any subsequent VIBs in the same folder (i.e. every new VIB that turns up in that folder, is injected into the VBK that also resides in said folder)
- Locally, you could still run synthetic fulls on a regular basis, but just as long as the VIB before the local synthetic full starts, is copied to the remote site (or a 'transfer/drop folder')

Seems to me, there are no fundemental changes in the technology required.. Veeam already have the ability to create synthetic fulls, so maybe there can be a 'standalone' service to accomplish this on the remote site?

The copying of the VIBs doesn't really need to be anything Veeam-coded either (although it would be cleaner in the long run); as there are plenty of products out there to manage things like this - a favourite of mine is Super Flexible File Sync.

Don't know what everyone else's thoughts are, and I may be way off base but I thought I'd inject my two-penneth..

Cheers, James

[jp34]

Hi,
I have not found a reliable answer to this question... building a full backup from previous full backup and increments will be possible in v6 ?
Please could you confirm ?
Thanks.

[foggy]

New version will allow to implement forever-incremental backup with synthetic fulls assembled locally on the backup target. Thanks.

[jp34]

And how to do synthetic fulls assembled REMOTELY on another PC ?
In other word, I have copied a vbk file and a vib file to a remote PC, I'd like to rebuild a new vbk including vib data. How can I do ?
Cheers, JP.

[Vitaliy S.]

JP, basically this is achieved with the help of remote backup repositories. All you need to do is to deploy a backup repository, move backup files to this repository, rescan it to locate backup files and then map these files for existing backup jobs. This would allow you to keep synthetic full traffic locally.

[jp34]

Ok I think I've not been clear... Jimmy's explanation is better (previous post in this page) :

[merged]
- Local Veeam runs a full backup
- Resulting VBK can be sent to remote site via USB drive and copied into a 'customer folder'
- Local Veeam runs an incremental
- Resulting VIB can be sent to remote site and dropped into the 'customer folder' via some method over the Internet - FTP, SFTP, FTPS or whatever (but to be scaleable, would need to be over a 'standard internet protocol' as opposed to copying to a fileshare which means VPN (for example)- and this isn't that scaleable and as self-managing considering interoperable issues with different FW/VPN vendors)
- A 'standalone' Veeam service at remote site, creates a synthetic full out of the original VBK and any subsequent VIBs in the same folder (i.e. every new VIB that turns up in that folder, is injected into the VBK that also resides in said folder)
- Locally, you could still run synthetic fulls on a regular basis, but just as long as the VIB before the local synthetic full starts, is copied to the remote site (or a 'transfer/drop folder')

It's exactly what I need, how to do that ?

[jp34]

Basically a simple command line utility could do it at remote DR site:
mergeutility "basevbkfile" "vibfile" "syntheticvbkfile"
basevbkfile: path of the vbk file generated by veeam (old synthetic or full backup)
vibfile: path of the vib file generated by veeam
syntheticvbkfile: path of the resulting vbk file (new vbk including vib data)

In this way, we would have to send 1 vbk file once to DR site (via USB drive) and only 1 vib file periodically... (no more big vbk transfert consuming bandwidth and time)

[Vitaliy S.]

There is no standalone tool to create synthetic backup on demand, but the procedure I outlined above should help you to achieve what you want.

[jp34]

Ok, I read many times your procedure... and I think I've understood now
I'm going to look at this.
Thanks.

[jp34]

What I've done...

On production site:
- Veeam has been installed
- A incremental backup job has been created and launched -> 1 vbk file created
- The backup job is launched on more time -> 1 vbi file created

On DR site:
- Another Veeam has been installed
- A DR repository has been created
- vbm, vbk and vib files have been copied from production repository to DR repository
- DR repository has been rescanned -> backup with 1 restore point imported

Now, how to generate a synthetic full on DR site ?

Thanks in advance.

[Vitaliy S.]

Now you need to re-use existing backup files with a backup job. Once you do it, just schedule synthetic full with transform on a desired day.

[foggy]

You do not need a second VBR server in the DR site, create the new backup repository within your production VBR instead. Then, specify the new repository as a target for your job and map it to the copied backup files - starting from the next run all synthetic backup activity will occur on the repository in the remote site.

[jp34]

Now you need to re-use existing backup files with a backup job. Once you do it, just schedule synthetic full with transform on a desired day.

I see I can schedule a synthetic full on a day, but can I choose when (hh:mm) on this day ? can I launch it manually ?

You do not need a second VBR server in the DR site, create the new backup repository within your production VBR instead. Then, specify the new repository as a target for your job and map it to the copied backup files - starting from the next run all synthetic backup activity will occur on the repository in the remote site.

I want to keep the production job running locally (full synthetic on sunday and incremental each week day) AND I'd like to do a synthetic full every day on DR site (after the daily copy of the vib file from production to DR site). Is it possible ?

[Vitaliy S.]

I see I can schedule a synthetic full on a day, but can I choose when (hh:mm) on this day ? can I launch it manually ?

Synthetic backup cannot be launched manually, as it is triggered right after your incremental job pass.

I want to keep the production job running locally (full synthetic on sunday and incremental each week day) AND I'd like to do a synthetic full every day on DR site (after the daily copy of the vib file from production to DR site). Is it possible ?

For that you need to create two similar backup jobs and point them to different backup repositories. In this case, you will have two synthetic backups on the main and on DR site.

[jp34]

In production VBR, I can create 2 jobs, one pointing to a local repository, the other one to the DR repository. Ok.
But I'm afraid that the job to the DR repository consums a lot of bandwidth:
- it would have to compare local changes (occured in the VM) to the remote last full synthetic (read/receive all bytes of the remote vbk). Right ?
- it would have to read the entire remote full synthetic and remote vib (read/receive all bytes of the remote vbk + vbi) to generate the new full synthetic (write/send all bytes of the new remote vbk). Right ?

[foggy]

This job will only transfer the changes over link between sites. The changes are calculated with the help of Change Block Tracking or (in case it is not available) using the metadata stored along with the backup (.VBM file, which is an order of magnitude smaller than the actual .VBK). So no need to "read/receive all bytes of the remote vbk". All synthetic full activity will occur locally in the DR, so no need to "read/receive all bytes of the remote vbk + vbi" and "write/send all bytes of the new remote vbk".

[jp34]

Changes has been transfered to the remote repository (vib through network) and then full sythetic generation on DR site doesn't consume any bandwidth.
It's perfect.

To sum up:

On production site:
- Veeam has been installed
- A incremental backup job with full synthetic has been created and launched -> 1 vbk file created
- The backup job is launched one more time -> 1 vbi file created

On DR site:
- A DR repository has been created (Veeam Transport service installed on remote computer)
- vbm, vbk and vib files have been copied from production repository to DR repository
- DR repository has been rescanned -> backup imported
- A new job has been created: same source VM, DR repository, incremental with full synthetic, mapped to DR repository

Result:
A great backup solution with minimum bandwidth consumption and local and remote full (synthetic) backups.

Thank you all.

[npoIT]

A while ago Gustav posted a response which said "In V6" - did that ever happen and if so, exactly what were you referring to?

Having a utility (ideally linux-based) which can create a synthetic full on a remote site using an "additional" incremental file synced from the primary would greatly simplify creating a DR copy offsite.

[foggy]

reading another thread backup using WAN storrage and synthetic full question it seems as though the answer is to use Linux as a target at the remote end. The agent does all the hard work converting the synthetic backups. I've yet to try this, but it seems to be the answer, assuming you have linux skills. the thread also mentions that this will be available on Windows at some point (!)

"In v6" was referred to the last statement in this quote: VBR6 now allows to implement forever-incremental backup with synthetic fulls assembled locally on the Windows backup target.

[fvdkroon]

Changes has been transfered to the remote repository (vib through network) and then full sythetic generation on DR site doesn't consume any bandwidth.
It's perfect.

To sum up:

On production site:
- Veeam has been installed
- A incremental backup job with full synthetic has been created and launched -> 1 vbk file created
- The backup job is launched one more time -> 1 vbi file created

On DR site:
- A DR repository has been created (Veeam Transport service installed on remote computer)
- vbm, vbk and vib files have been copied from production repository to DR repository
- DR repository has been rescanned -> backup imported
- A new job has been created: same source VM, DR repository, incremental with full synthetic, mapped to DR repository

Result:
A great backup solution with minimum bandwidth consumption and local and remote full (synthetic) backups.

Thank you all.

I've been reading this post with great interresest because we have the same "problem". The above solution sounds great, but does this mean that 2 mainly identical jobs run each night, so each VM is being back-uped twice. Once for the local site and once for the remote? The backup window increases with 100% if this is true.

Regards,

Freek

[Vitaliy S.]

If there are two destination targets/backup repositories, then yes, you will have to run two backup jobs. However there is another way to sync your backup files with an offsite location. Please search these forums for rsync usage examples. Thanks!

[hobbit]

Hello,

is it somehow possible to create a new vbk from an existing vbk and several vib´s in the backup repository? I know I can do this in a backup job by selecting the synthetic full on day X option, but can I do somehow do this without running an actual backup?

Regards

[veremin]

As far as I’m concerned, there are no ways to manually create synthetic full backup. The only available option is to choose particular days in the advanced settings to create synthetic full backup.

Furthermore, so far there have been no well-known PowerShell scripts for doing it either.

Hope this helps.
Thanks.

[Ben Milligan]

Vladimir is correct, there is no current way to do this outside of running the aforementioned full.

[zoltank]

Is there any way to run a synthetic full aside from running a backup job?

For our backup Veeam server I only run backups over the weekend, and I want it to create synthetic fulls during the week without running a backup job. Is there anyway to do this?