Veeam, rsync and incremental mode : would this work

[MB-NS]

Hello,

I'm currently investigating how to use rsync with the incremental mode without having to replicate full vbk over the WAN.
As I understand it, by default incremental mode makes the synthetic full VBK with a different name each time.
It makes it impossible for rsync to recognise the new vbk as just the old one but with the week's vib injected into it ==> the full vbk is replicated.
Reverse incremental is not a nice option for me because I need tape backups.

I have devised the following way to circumvent that, and I'm interested in the community opinion on whether it is a viable way to do it or not.
It involves using hard links in a separate folder, in order to always present the same name for the latest vbk file. the links for VIB would use the same name.

Basically, if the real folder (Veeam target) is like this :

1.VBK
2.VIB
3.VIB
4.VIB
5.VIB
6.VIB
7.VIB

The rsynced folder would look like this :

sticky.VBK -> link to 1.VBK
2.VIB -> link to 2.VIB
3.VIB -> link to 3.VIB
4.VIB -> link to 4.VIB
5.VIB -> link to 5.VIB
6.VIB -> link to 6.VIB
7.VIB -> link to 7.VIB

On day 8, a new VBK is created (synthetic full), the real folder is like this :

1.VBK
2.VIB
3.VIB
4.VIB
5.VIB
6.VIB
7.VIB
8.VBK

the rsynced folder would then look like this :

sticky.VBK -> link to 8.VBK
2.VIB -> link to 2.VIB
3.VIB -> link to 3.VIB
4.VIB -> link to 4.VIB
5.VIB -> link to 5.VIB
6.VIB -> link to 6.VIB
7.VIB -> link to 7.VIB

This way rsync will only copy the modified blocks on the VBK file, which are supposed to be only the injected blocks by the synthetic full.
The rsynced folder can be easily maintained using a post-script, it is not destructive as it uses hard links and do not move files from the Veeam folder.
It also do not consume additional space as it is not a copy.
If I want to import the rsynced backups, I just need to rename the sticky.VBK file to its original file (or is it even required ?).

Will this work well, or am I missing something ? The only thing I'm not sure of is the real amount of data that will be replicated when the sticky.VBK file switchs from a synthetic full to another.

[Gostev]

Yep, this should work! Very creative solution

[jingxi02]

[merged]

We have been trying to use Veeam to do local backup and remote replication but run into a limitation of the software. Here is what we are trying to do.

1. Run Veeam Backup job to backup all VMs into a Windows Storage server.
2. Use 3rd party file replication software such as EMC Replistor to replication the directory that contains the Veeam backup files to a DR site over VPN on a T1.

The idea is to have a local backup data in case we need to fast recover files or VMs from the local backup storage. We also would like to transfer all backup file to a offsite location in case of complete disaster in the production site. The issue that prevents us to use this strategy is the Veeam software keeps change the name of the .vbk file. Here is the detail of the file structure and the problem.

Files in backup directory in Day 1 - 7:
Backup-job2011-08-01Txxxxx.vbk 790GB
Backup-job2011-08-02Txxxxx.vbi 128MB
Backup-job2011-08-03Txxxxx.vbi 110MB
Backup-job2011-08-04Txxxxx.vbi 126MB
Backup-job2011-08-05Txxxxx.vbi 140MB
Backup-job2011-08-06Txxxxx.vbi 159MB
Backup-job2011-08-07Txxxxx.vbi 168MB

When we use 3rd party software to replicate these backup files, we can just copy the .vbk file in the USB dirve and do a offline transfer because of the size of the .vbk file. For the .vbi files, we can just left the replication software to replicate it over the VPN since they are pretty small. However, in day 8 when the backup runs the Synthetic full consolidation, it creates a new .VBK file with the new name.

Files in backup directory in Day 8:

Backup-job2011-08-01Txxxxx.vbr 128MB
Backup-job2011-08-02Txxxxx.vbr 110MB
Backup-job2011-08-03Txxxxx.vbr 126MB
Backup-job2011-08-04Txxxxx.vbr 140MB
Backup-job2011-08-05Txxxxx.vbr 159MB
Backup-job2011-08-06Txxxxx.vbr 168MB
Backup-job2011-08-07Txxxxx.vbr 168MB
Backup-job2011-08-08Txxxxx.vbk 870GB
Backup-job2011-08-08Txxxxx.vbi 123MB

As you can see a new .vbk file has created. Since the replication software is monitor the file changes in the directory, it will attempt to replication this large file over a T1 which is impossible to complete in a month.

I've identified and confirmed this issue with the rep when I was in VMworld at Vegas. I'm hoping Veeam can adjust the backup file structure to the way that is capable for low bandwidth offsite replication.

The basic idea is to keep the name of the .vbk file alway the same no matter which backup mode you use and no matter what retention policy on the backup job.

The other idea is most of the 3rd party file replication software can replicate only the content change inside the file instead of the whole file. If Veeam backup injects the incremental into the .vbk file, it most like going to be OK with the replication software because the software can just replicate the changes in the file. However, if Veeam injects the incremental into the .vbk file and change the name of the file, the replication software will see this file is a complete new file and attempt to replicate the whole thing.


It Veeam 5, there is no way you can run the Veeam native Replcaition job over a T1. That is why we were trying to use 3rd party replication software to do this. It seems like Veeam 6 is going to have a new way to do replication, we may be able to run Veeam native replication job over the WAN, but hopefull it doesn't run into this file structure limtation like Veeam 5.

If possible, we really would like to beta test the new Veeam 6 to see if that solves this issue or not.

[Gostev]

The above solution should help.

[chum]

I have the same issue in trying to rsync (deltacopy) Veeam Backup files offsite to a remote windows shared folder.

The solution presented above whereby using hard links sounds interesting. How do you implement that in Windows?

[Vitaliy S.]

Here is some more interesting examples on how our existing customers are leveraging hard links > http://forums.veeam.com/search.php?st=p ... id%5B%5D=2

And this article should help you to figure out how to configure hard links: Hard Links vs. Soft Links

[Gostev]

Great article, but I guess it is only useful for Windows API developers because I don't understand much there
But here is good article for regular people like me > http://en.wikipedia.org/wiki/NTFS_symbolic_link

[jingxi02]

I don't think this solution is going to work because the name of the .vbk file keeps changing no metter the job is using incremental or revserse incremental. In reverse incremantal job, the .vbk file get changed every time the job runs. In incremental job, the .vbk file get changed every time the Synthetic full is run. Once the name of the file changed, the hard link or symbolic may be broken.
I suggest Veeam should change the backup file naming scheme to address this issue instead have the end user to work around it. If we just work around it by whatever method and there is a need for us to get support during the restoration process, Veeam support will not support it because the modiffcation we made is not offical supported solution.
In the programing prospective, I don't think it's too much work to change the file naming scheme in the backup job. Just make the name of the .vkb to <name of the job.vbk>.

[Gostev]

I don't think this solution is going to work because the name of the .vbk file keeps changing

Read the first post carefully

[jftuga]

You might consider the --fuzzy rsync option:

--fuzzy
This option tells rsync that it should look for a basis file for any destination file that is missing. The current algorithm looks in the same directory as the destination file for either a file that has an identical size and modified-time, or a similarly-named file. If found, rsync uses the fuzzy basis file to try to speed up the transfer.

Note that the use of the --delete option might get rid of any potential fuzzy-match files, so either use --delete-after or specify some filename exclusions if you need to prevent this.

[Gostev]

Actually, fuzzy is not needed. With RSYNC, you can sync any file with any file. So, while source file name will always be different, target file will always be the same. This is the whole idea behind the approach that OP has posted.

[MB-NS]

I have the same issue in trying to rsync (deltacopy) Veeam Backup files offsite to a remote windows shared folder.

The solution presented above whereby using hard links sounds interesting. How do you implement that in Windows?

Hello,

although I did not write it yet, a script which use "mklink" along with some name-parsing logic (to identify the latest VBK) should do the trick.

[Shogan]

Hey guys,

I have a situation where there are two QNAP NAS devices at two different locations. (A + B).

Veeam Backup is run from location A to backup various VMs in various jobs, sending the jobs to location B over the network (1GBit) to a QNAP NAS at location B. Location B's QNAP NAS, then runs an rsync job after these backups are complete to replication the backed up files, back to Location A on Location A's QNAP NAS. (The original VMs sit on a separate SAN device at Location A too).

I would love to be able to use the OP's method of hard links, however I am unsure as to how to achieve this with two QNAP NAS devices. Both NAS devices are TS-859U-RP+ models.

Does anyone have any advice or clues as to how to go about achieving this? Currently the Reverse Incremental VBK files are duplicated each day because the name changes, wasting both disk space and bandwidth over this dedicated 1GBit network (as well as taking too long!)

[foggy]

Please consider this topic for the hardlinks solution for reverse incremental mode. Also, this one might be useful in this case.

BTW, why do you backup remotely first and then rsync back to the site A instead of backing up locally and then syncing offsite? This would take much less bandwidth utilizing the link only once for each backup. Do you have any considerable reasons for such strategy?

[Shogan]

Please consider this topic for the hardlinks solution for reverse incremental mode. Also, this one might be useful in this case.

BTW, why do you backup remotely first and then rsync back to the site A instead of backing up locally and then syncing offsite? This would take much less bandwidth utilizing the link only once for each backup. Do you have any considerable reasons for such strategy?

Hi foggy,

Thanks for those links, even through searching the forum on rsync I hadn't come across that. I am still quite undecided as to how to handle the situation. I will have to do some testing and see if I can get hard links working on the share(s) on the QNAP NAS I am backing up to from the Veeam server.

Regarding the way I am backing up - well its not decided as a final solution yet. Currently backing up all production to a SAN on the other side of the network, which is enterprise level, and therefore happy the backed up data is safe. However going forward I would like to backup to the cheaper and more abundant storage on the QNAP NAS. However, this isn't very safe if the hardware failed, so therefore have a 2nd NAS that I want the data replicated to. At the moment, I am thinking it is best to have a backup of the VMs offsite ASAP - therefore backing up to remote NAS, then rsync'ing back to the extra NAS at the same location as the originally backed up VMs. I know it is a bit of a waste, and will re-evaluate once I have some other solutions the direction the backups work - you're right though, backing up locally, then rsync'ing across to remote site would save 1 step over the process compared to the method in my last post!

Going back to those links you gave me, it seems that 1 or 2 guys with QNAP NAS devices ended up looking for the hotfix/reg patch to keep filenames the same... I hope I don't need to do that. I would ideally just like to be able to rsync my entire VM backup job set to remote location, working off syncing only changes into the full reverse incremental VBK file, and not having to rsync the entire VBK all over again!

[chum]

For those users (like myself) that use DeltaCopy or rsync to send Veeam Backup files off site, here is a script you can use (usual disclaimer applies here - I take no responsibility for any damage to any software or hardware in using this script...blah, blah, blah, etc...). I have tested this and it works fine in my environment (Windows Server 2003).

What this script does is create a hardlink of the latest full backup (.vbk file). This will then allow you to use DeltaCopy to sync this hardlink to a destination file taking advantage of DeltaCopy's ability to only send the changed portion of the file. The reason for this script is because DeltaCopy (rsync) does not recognise the filename change by Veeam and will sync the entire file and not just the changes.

To use this script, save it as a batch file, change the variables (server name, backup file directory, hardlink directory) to suit your environment and call the batch file from within a Veeam Backup and Replication Job as a "Post Job Activity" (located in the Advanced Settings on the Advanced Tab). Then schedule Deltacopy to sync after the Veeam backup job completes.

Feel free to add any comments or improvements to this script.

Code: Select all

@echo off
:: This script creates/replaces the hardlink for the main server backup file,
:: linking it to the most recent backup for the same server.
::
:: e.g. if the servername is set to "castle", it will look for files named
:: "backup of castle-XXXXXXX.vbk" and use the most recent one as the real
:: backup file. It will then link "backup of castle.vbk" to the real backup
:: file.


set servername=SERVERNAME HERE
set backupdir=SOURCE DIRECTORY HERE
set linkdir=HARDLINK DIRECTORY HERE

SETLOCAL ENABLEDELAYEDEXPANSION

set realbackup=none

:: find the latest backup by parsing the output of 'dir'
:: /B enables bare output (just the filename);
:: /OD sorts in date order (oldest first)
for /F "tokens=1* delims=" %%f in ('DIR /B /OD  "%backupdir%\backup of %servername%-*.vbk"') do set realbackup=%%f
if "%realbackup%"=="none" goto nobackupfound

set linkname=%linkdir%\backup of %servername%.vbk

echo Found most recent backup: %realbackup%
echo Creating link from %linkname%

if exist "%linkname%" del "%linkname%"
if errorlevel 1 goto delerror
fsutil hardlink create "%linkname%" "%backupdir%\%realbackup%"
if errorlevel 1 goto createrror

goto out


:delerror
echo %linkname% exists but unable to delete.
EXIT /B 2

:createrror
echo Failed to create link %linkname%.
EXIT /B 3


:nobackupfound
echo.
echo Error: could not find full backup file.
echo Searched %backupdir%
echo for %servername%
echo.
EXIT /B 1


:out

[gkoers]

Hi,

I'm looking into offsite storage of a backup also. Goal is to have a vbk of each VM on an offsite NAS. Plan is to use rsync to sync the latest changes in the local backup vbk file to the offsite vbk file. I do not seem to get the commands for rsync right, because it's processing the whole VM and is then copying the data. The proces now takes over 4 hours while the changed data is less then 2 GBs. Currently the speed of the line is 4 mbit/s, I would expect the transfer to be a lot faster. Am I missing something?

This is the command I'm using:

Code: Select all

rsync -v -rlt --chmod=a=rw,Da+x --stats --progress --log-file=/cygdrive/c/pathto/rsynclog.txt //NAS/Veeam/DC1/*.vbk admin@192.168.5.101:/share/veeam/DC1/DC1Backup2011-11-17T051523.vbk

[Vitaliy S.]

Hi Gerrit, try this command, should work!

[gummby8]

[merged]

So the scheme I am going for here is
VM Host ---> Local Backup ----> Offsite replicated backup

The 2 options I have researched are, 2 separate backup jobs, one local and one to the offsite backup location or replicate the local backup to the offsite storage.

I don't like the 2 backup jobs simply because the backup files are huge. So I need a few bits of help for the offsite replication job.

The problem is that Veeam names its files with timestamps and whatnot, which will not work for my offsite replication software. Is there a way to make sure Veeam uses 1 file name and keeps that name forever

[dellock6]

If you use forward incremental you simply add new backup files without touching the old ones.

Luca.

[gummby8]

Whats with all this hard link and --fuzzy rsync stuff. Just let us set our own name scheme and this would be a frigging problem

[stevedarl1]

Hello,

We currently backup 4Tb using Veeam B&R 6.5 to an onsite NAS using forever Incrementals. This NAS then replicates using NAS RSYNC over the WAN to an identical NAS.

We want to utilise synthetic fulls so to minimise WAN traffic but it creates a different name each time causing issues with the NAS replica. Is there anyway to use the exisitng name of the original active full every time?

[foggy]

Steve, please look through the tips given in this thread, should address your scenario.