off-site backup ideas

[alexr]

I would like to check if following idea would be of value to other VEEAM users and of interest to VEEAM.

We would like to be able to replicate backup files to remote location(s) over WAN. We have no need to replicate to other ERSX servers just have off premises backups.
Of course we could use rsync or similar products to do that but with VEEAM already updating backups synthetically (or doing replication to ESX) it would be much more efficient if VEEAM did that.

Ideally I think VEEAM would have some service running on windows (or unix?) with which VEEAM would communicate to send changes and the service would use them to update synthetic backup and create roll-back file

Alternatively (performance might not be acceptable) veeam job can be given secondary path(s) to mirror backup files. The secondary path(s) could be on removable media or network. Secondary files should be processed after the primary is done to make sure primary backup performance is not affected

finally if VEEAM can do both synthetic and incremental in one go (and i think it would be very useful not only for off site backups but for long term tape storage) we could probably use rsync more effectively since full backup will be updated only say weekly

I would appreciate your comments

alex

[Gostev]

finally if VEEAM can do both synthetic and incremental in one go (and i think it would be very useful not only for off site backups but for long term tape storage) we could probably use rsync more effectively since full backup will be updated only say weekly

This!

Something we are working on for the next release, you are right - this is the most universal approach that would help in many scenarios. Will post more details closer to the actual release, but essentially you will have an regular incrementals without loosing most of the synthetic backup benefits you can enjoy today.

[alexr]

This!

Something we are working on for the next release, you are right - this is the most universal approach that would help in many scenarios. Will post more details closer to the actual release, but essentially you will have an regular incrementals without loosing most of the synthetic backup benefits you can enjoy today.

Yes it is very useful! But does not solve remote site backup issue since it is still not possible to move complete backup over WAN. I wonder if RSYNC will work well with full backup files. Are they reorganized significantly or changes are localized? What I mean is that compressed/encrypted file would change dramatically even if source file had little change. Would it be the case with VEEAM backup files? Would synthetic backup file (daily sync) be more RSYNC friendly than incremental weekly? I still would love option for synthetic backup to remote location since VEEAM know exactly what needs to be changed

[Gostev]

I still would love option for synthetic backup to remote location since VEEAM know exactly what needs to be changed

This is not impossible (we do this today if you select remote ESX/Linux as your backup target). However, RSYNC will be more efficient, because it operates with very small blocks.

[alexr]

This is not impossible (we do this today if you select remote ESX/Linux as your backup target). However, RSYNC will be more efficient, because it operates with very small blocks.

I never tried RSYNC on really large files - say 200G. I thought that since it needs to scan the whole file (I do not think it keeps any sort of block level hash) it would take a long while even for small changes to be synced. Where VEEAM could ship all deltas zipped to a remote VEEAM service which would reconstruct synthetic backup file - nirvana .

Also, since there is no currently an option to backup to two targets (say local directory and remote linux) at once backing up to remote server only is not quite what I would like

[neto]

This is not impossible (we do this today if you select remote ESX/Linux as your backup target). However, RSYNC will be more efficient, because it operates with very small blocks.

Personally, I think it would be less efficient to use rsync as from what I know rsync would have to re-hash the file, reading the entire contents on both the source and the destinations servers and using cpu power to re-hash the block for comparison. If copying to a low power NAS, this would be terribly slow.

Since Veeam already has to compute and determine what has changed on the backup file when its doing its own backup procedure I think it would be much more efficient for Veeam to keep track of those changes and then send it to a secondary location. I'm not sure how feasible this would be...

[Gostev]

We found that actual hashing takes very little resource (almost unnoticeable). With remote backups, the limiting factor is usually bandwidth - the less data you have to transfer, the faster sync goes. Because Veeam currently uses 1024KB blocks to track changes, and RSYNC uses much smaller block to by understanding, so today RSYNC will be more efficient in most cases. But we do constantly listen to feedback, which is why I said that nothing is impossible!

[olofc]

I have trouble with rsync and large files (200gb and above is not uncommon at our site) This because of rsync is cpu intensive. Has anynone been able to successfully use rsync with larger veeam files?

[tsightler]

We use rsync with files that are quite a bit larger than 200GB (800GB, 1.5TB) and it works fine. We make sure to use the "--inplace" option so that the target file is updated in place rather than the normal of writing a new file. That being said, our backups systems are Linux. If you're using rsync on Windows well, that could be a problem as I've seen Windows rsync use a lot of CPU for no real good reason.

[olofc]

thanks tom for your kind answer, it is probably because we use the windows ported rsync then, i will try using linux rsync instead! Thanks!

[Liberty4All]

I have an idea for an enhancement for backup and replication. I will explain what I am doing today so you can see the need.
Currently I am using Veeam 4.1 via VCB to connect to Virtual Center 2.5 update 5.
I am using Best compression and running different backups based on server groups. So normally I have around 5 servers in a backup job. These jobs run weekly and a different job runs each night of the week. It is not uncommon to have a 55GB vbk file and have a 5GB delta file.

I also have a DR location that I am copying out the bkf files to. This is just a windows file server. I am using delta copy (rsync) to copy these files. I am noticing that even though it only sends the file changes it is really sends most of the file (maybe because of the compare?).

Since veeam already knows what blocks are change because it creates a delta file can the functionality be added to send only the changed blocks to modify the file? This would be extremely helpful since the WAN link I (and others) am using is small. Remember I need the file in both locations and would prefer not to use two jobs to complete this. Thoughts?

[Gostev]

Hello Paul, thank you for your feedback. I have merged your topic with the existing discussion of this feature request.

[Liberty4All]

This is a follow up to those using rsync on large files. If you have a 200GB file what are your delta's? If you monitor what your sending over the network are you sending close to your delta files?

For a short time we were using Avamar to replicate over the WAN. This worked well in only sending changes but is far too expensive of a solution.