Veeam, rsync and incremental mode : would this work

[MB-NS]

Hello,

I'm currently investigating how to use rsync with the incremental mode without having to replicate full vbk over the WAN.
As I understand it, by default incremental mode makes the synthetic full VBK with a different name each time.
It makes it impossible for rsync to recognise the new vbk as just the old one but with the week's vib injected into it ==> the full vbk is replicated.
Reverse incremental is not a nice option for me because I need tape backups.

I have devised the following way to circumvent that, and I'm interested in the community opinion on whether it is a viable way to do it or not.
It involves using hard links in a separate folder, in order to always present the same name for the latest vbk file. the links for VIB would use the same name.

Basically, if the real folder (Veeam target) is like this :

1.VBK
2.VIB
3.VIB
4.VIB
5.VIB
6.VIB
7.VIB

The rsynced folder would look like this :

sticky.VBK -> link to 1.VBK
2.VIB -> link to 2.VIB
3.VIB -> link to 3.VIB
4.VIB -> link to 4.VIB
5.VIB -> link to 5.VIB
6.VIB -> link to 6.VIB
7.VIB -> link to 7.VIB

On day 8, a new VBK is created (synthetic full), the real folder is like this :

1.VBK
2.VIB
3.VIB
4.VIB
5.VIB
6.VIB
7.VIB
8.VBK

the rsynced folder would then look like this :

sticky.VBK -> link to 8.VBK
2.VIB -> link to 2.VIB
3.VIB -> link to 3.VIB
4.VIB -> link to 4.VIB
5.VIB -> link to 5.VIB
6.VIB -> link to 6.VIB
7.VIB -> link to 7.VIB

This way rsync will only copy the modified blocks on the VBK file, which are supposed to be only the injected blocks by the synthetic full.
The rsynced folder can be easily maintained using a post-script, it is not destructive as it uses hard links and do not move files from the Veeam folder.
It also do not consume additional space as it is not a copy.
If I want to import the rsynced backups, I just need to rename the sticky.VBK file to its original file (or is it even required ?).

Will this work well, or am I missing something ? The only thing I'm not sure of is the real amount of data that will be replicated when the sticky.VBK file switchs from a synthetic full to another.

[Gostev]

Yep, this should work! Very creative solution

[jingxi02]

[merged]

We have been trying to use Veeam to do local backup and remote replication but run into a limitation of the software. Here is what we are trying to do.

1. Run Veeam Backup job to backup all VMs into a Windows Storage server.
2. Use 3rd party file replication software such as EMC Replistor to replication the directory that contains the Veeam backup files to a DR site over VPN on a T1.

The idea is to have a local backup data in case we need to fast recover files or VMs from the local backup storage. We also would like to transfer all backup file to a offsite location in case of complete disaster in the production site. The issue that prevents us to use this strategy is the Veeam software keeps change the name of the .vbk file. Here is the detail of the file structure and the problem.

Files in backup directory in Day 1 - 7:
Backup-job2011-08-01Txxxxx.vbk 790GB
Backup-job2011-08-02Txxxxx.vbi 128MB
Backup-job2011-08-03Txxxxx.vbi 110MB
Backup-job2011-08-04Txxxxx.vbi 126MB
Backup-job2011-08-05Txxxxx.vbi 140MB
Backup-job2011-08-06Txxxxx.vbi 159MB
Backup-job2011-08-07Txxxxx.vbi 168MB

When we use 3rd party software to replicate these backup files, we can just copy the .vbk file in the USB dirve and do a offline transfer because of the size of the .vbk file. For the .vbi files, we can just left the replication software to replicate it over the VPN since they are pretty small. However, in day 8 when the backup runs the Synthetic full consolidation, it creates a new .VBK file with the new name.

Files in backup directory in Day 8:

Backup-job2011-08-01Txxxxx.vbr 128MB
Backup-job2011-08-02Txxxxx.vbr 110MB
Backup-job2011-08-03Txxxxx.vbr 126MB
Backup-job2011-08-04Txxxxx.vbr 140MB
Backup-job2011-08-05Txxxxx.vbr 159MB
Backup-job2011-08-06Txxxxx.vbr 168MB
Backup-job2011-08-07Txxxxx.vbr 168MB
Backup-job2011-08-08Txxxxx.vbk 870GB
Backup-job2011-08-08Txxxxx.vbi 123MB

As you can see a new .vbk file has created. Since the replication software is monitor the file changes in the directory, it will attempt to replication this large file over a T1 which is impossible to complete in a month.

I've identified and confirmed this issue with the rep when I was in VMworld at Vegas. I'm hoping Veeam can adjust the backup file structure to the way that is capable for low bandwidth offsite replication.

The basic idea is to keep the name of the .vbk file alway the same no matter which backup mode you use and no matter what retention policy on the backup job.

The other idea is most of the 3rd party file replication software can replicate only the content change inside the file instead of the whole file. If Veeam backup injects the incremental into the .vbk file, it most like going to be OK with the replication software because the software can just replicate the changes in the file. However, if Veeam injects the incremental into the .vbk file and change the name of the file, the replication software will see this file is a complete new file and attempt to replicate the whole thing.


It Veeam 5, there is no way you can run the Veeam native Replcaition job over a T1. That is why we were trying to use 3rd party replication software to do this. It seems like Veeam 6 is going to have a new way to do replication, we may be able to run Veeam native replication job over the WAN, but hopefull it doesn't run into this file structure limtation like Veeam 5.

If possible, we really would like to beta test the new Veeam 6 to see if that solves this issue or not.

[Gostev]

The above solution should help.

[chum]

I have the same issue in trying to rsync (deltacopy) Veeam Backup files offsite to a remote windows shared folder.

The solution presented above whereby using hard links sounds interesting. How do you implement that in Windows?

[Vitaliy S.]

Here is some more interesting examples on how our existing customers are leveraging hard links > search.php?st=posts&keywords=hard+links&fid%5B%5D=2

And this article should help you to figure out how to configure hard links: Hard Links vs. Soft Links

[Gostev]

Great article, but I guess it is only useful for Windows API developers because I don't understand much there
But here is good article for regular people like me > http://en.wikipedia.org/wiki/NTFS_symbolic_link

[jingxi02]

I don't think this solution is going to work because the name of the .vbk file keeps changing no metter the job is using incremental or revserse incremental. In reverse incremantal job, the .vbk file get changed every time the job runs. In incremental job, the .vbk file get changed every time the Synthetic full is run. Once the name of the file changed, the hard link or symbolic may be broken.
I suggest Veeam should change the backup file naming scheme to address this issue instead have the end user to work around it. If we just work around it by whatever method and there is a need for us to get support during the restoration process, Veeam support will not support it because the modiffcation we made is not offical supported solution.
In the programing prospective, I don't think it's too much work to change the file naming scheme in the backup job. Just make the name of the .vkb to <name of the job.vbk>.

[Gostev]

I don't think this solution is going to work because the name of the .vbk file keeps changing

Read the first post carefully

[jftuga]

You might consider the --fuzzy rsync option:

--fuzzy
This option tells rsync that it should look for a basis file for any destination file that is missing. The current algorithm looks in the same directory as the destination file for either a file that has an identical size and modified-time, or a similarly-named file. If found, rsync uses the fuzzy basis file to try to speed up the transfer.

Note that the use of the --delete option might get rid of any potential fuzzy-match files, so either use --delete-after or specify some filename exclusions if you need to prevent this.

[Gostev]

Actually, fuzzy is not needed. With RSYNC, you can sync any file with any file. So, while source file name will always be different, target file will always be the same. This is the whole idea behind the approach that OP has posted.

[MB-NS]

I have the same issue in trying to rsync (deltacopy) Veeam Backup files offsite to a remote windows shared folder.

The solution presented above whereby using hard links sounds interesting. How do you implement that in Windows?

Hello,

although I did not write it yet, a script which use "mklink" along with some name-parsing logic (to identify the latest VBK) should do the trick.

[Shogan]

Hey guys,

I have a situation where there are two QNAP NAS devices at two different locations. (A + B).

Veeam Backup is run from location A to backup various VMs in various jobs, sending the jobs to location B over the network (1GBit) to a QNAP NAS at location B. Location B's QNAP NAS, then runs an rsync job after these backups are complete to replication the backed up files, back to Location A on Location A's QNAP NAS. (The original VMs sit on a separate SAN device at Location A too).

I would love to be able to use the OP's method of hard links, however I am unsure as to how to achieve this with two QNAP NAS devices. Both NAS devices are TS-859U-RP+ models.

Does anyone have any advice or clues as to how to go about achieving this? Currently the Reverse Incremental VBK files are duplicated each day because the name changes, wasting both disk space and bandwidth over this dedicated 1GBit network (as well as taking too long!)

[foggy]

Please consider this topic for the hardlinks solution for reverse incremental mode. Also, this one might be useful in this case.

BTW, why do you backup remotely first and then rsync back to the site A instead of backing up locally and then syncing offsite? This would take much less bandwidth utilizing the link only once for each backup. Do you have any considerable reasons for such strategy?

[Shogan]

Please consider this topic for the hardlinks solution for reverse incremental mode. Also, this one might be useful in this case.

BTW, why do you backup remotely first and then rsync back to the site A instead of backing up locally and then syncing offsite? This would take much less bandwidth utilizing the link only once for each backup. Do you have any considerable reasons for such strategy?

Hi foggy,

Thanks for those links, even through searching the forum on rsync I hadn't come across that. I am still quite undecided as to how to handle the situation. I will have to do some testing and see if I can get hard links working on the share(s) on the QNAP NAS I am backing up to from the Veeam server.

Regarding the way I am backing up - well its not decided as a final solution yet. Currently backing up all production to a SAN on the other side of the network, which is enterprise level, and therefore happy the backed up data is safe. However going forward I would like to backup to the cheaper and more abundant storage on the QNAP NAS. However, this isn't very safe if the hardware failed, so therefore have a 2nd NAS that I want the data replicated to. At the moment, I am thinking it is best to have a backup of the VMs offsite ASAP - therefore backing up to remote NAS, then rsync'ing back to the extra NAS at the same location as the originally backed up VMs. I know it is a bit of a waste, and will re-evaluate once I have some other solutions the direction the backups work - you're right though, backing up locally, then rsync'ing across to remote site would save 1 step over the process compared to the method in my last post!

Going back to those links you gave me, it seems that 1 or 2 guys with QNAP NAS devices ended up looking for the hotfix/reg patch to keep filenames the same... I hope I don't need to do that. I would ideally just like to be able to rsync my entire VM backup job set to remote location, working off syncing only changes into the full reverse incremental VBK file, and not having to rsync the entire VBK all over again!