Backup Merge Taking Ages

[IHeartCats]

Hi,

We have a nearly 15TB VM that we attempt to back up nightly.

It is in a backup job on it's own.

Uses the forever forward incremental - 8 restore points kept on disk.

Incrementals size up at around 600-800gb a night.

Sometimes merges complete within 10hrs but more frequently they are taking more than 24hrs (Causing the backups for the following day or two to be skipped)
Last merge took 52hrs!!

I'd like to try and get to grips with what factors affect the speed of the merges and if it's something I should be faulting with support.

As a frame of reference. Another 12TB VM job backup up from and to the same storage and location with 150gb of churn takes 2.5hrs approx to merge each night?

Is it simply the size of the incremental that causes long merges as it transforms all that data?

Thanks
Helen

[tsightler]

Can you tell us about the storage you are backing up to? What is the hardware, how many drives does it have and how are they configured (RAID6 for example)?

[IHeartCats]

Hi,
Our backup array is a Netapp FAS 8020.

2 aggregates of 42 disks using RAID-DP.

On top of these sit various volumes that are shared by CIFS used as extents for veeam scale out repos.


Please note the figures I gave for the merge on the 12TB backup job (2.5hrs merge) vs the 15TB(10-60hrs merge) one that is very slow, these are on the same storage on the same repos.

Helen

[tsightler]

Please note the figures I gave for the merge on the 12TB backup job (2.5hrs merge) vs the 15TB(10-60hrs merge) one that is very slow, these are on the same storage on the same repos.

Looking at the minimum times, I didn't see much difference, 2.5 hours for 150GB is almost exactly the same performance as 10 hours for 600GB. Indeed if you are seeing 60 hours, that would be quite long. Are you running any compact operations or periodic active fulls? My guess is, with servers that large and that much change, they are likely to be SQL servers and, over time, the backup files can become very fragmented. That's why it's normally recommended to run active fulls at least periodically.

It's really hard to know without diving deeper into the performance of the storage and latency during the merge however. If feels like the 2.5 or 10 hours times might be "normal" for your storage, but the ones that run longer than that would seem likely to have other options. That being said, it's quite common for large VMs with large change rates to perform much better running weekly active fulls vs trying to perform merges.

[csydas]

Helen,

As far as I know, you have to imagine a lot of read and writes going on at once on the repository, and how much has actually changed within. With CIFS, all of the data has to get sent from whatever is set as the Gateway in the Repository settings, so you can maybe check if by chance it's on Automatic and Veeam is choosing different gateways.

Veeam once gave me this kb to simulate the transform speeds: https://www.veeam.com/kb2014

You can basically run the one for transforms and divide the total result by 2 to get effectively how fast you can write to the storage during a transform. Then just math it out over the size of a 15 TB file and you get a fair idea on the merging process, since keep in mind, the full VBK can get fragmented. (I don't know the specifics, but the fragmentation gets pretty high especially on highly dynamic content). If you're producing 600 GB+ increments, you're going to have a heavily fragmented full backup, and hence it's going to take longer than normal (the test will be faster than your real world speeds).

The difference in size is because I assume the workload for merges does not increase linearly but exponentially given the amount of change churn in/out. If your file was perfectly defragged, you'd see faster merges. So just my $.02 from experience with this.

[IHeartCats]

Hi Folks,

You could be onto something here.

The VM is used to host a share that has lots of data written to it by a piece of imaging equipment and then is processed.

This would seems to make sense that all the churn could be causing the backup file to become fragmented. Especially as the merges have been trending towards getting longer as time goes on since the initial active full.

I don't know the exact mechanism for performing the FFincremental processing, but would switching to a reverse incremental backup be subject to the same fragmentation issues? (I'm guessing it would and just the synthesis of the full that would take ages rather than the merge of the oldest point)?

Thanks for your input

H

[csydas]

There's a compact and defrag option in the Storage Advanced settings for the job, but be warned, the requirements are a doozy: https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Unfortunately, you're not going to have much choice I think, but just keep in mind you need free space = the VBK size in order to do it, and it can take awhile.

[slos]

Is it feasible for you to move to a forward only approach? Eg take a full on day of choice (Saturday perhaps) and then only forwards during the other days. A restore point count of eight means would hold on to the old backup chain for a while (doubling your disc space use for a length of time) but you would never have to merge.

You would also have to absorb a weekly full of the 15 TB VM which may or may not make sense.

[IHeartCats]

Thanks for this suggestion. Yes we are now considering this as an option if we can't convince the users of the data on this server to housekeep it down a bit and perhaps use a different server that we don't back up for the highly dynamic content and only write it to the backed up server once it's processed.

[IHeartCats]

There's a compact and defrag option in the Storage Advanced settings for the job, but be warned, the requirements are a doozy: https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Unfortunately, you're not going to have much choice I think, but just keep in mind you need free space = the VBK size in order to do it, and it can take awhile.

Thanks - I was reading about this option Friday. I think we are more likely to err on the side of taking a new active full every few days rather than compacting as they have similar space implications but the active full would remove the need for the merge and the compact so it's looking like the better option (as they both essentially require 2xVM size of space for the duration of the backup).

However if we can't make the active full work then it's good to have another option.

What we really need is the VM to not be so damn big and dynamic.


Thanks all for you responses

H

[IHeartCats]

Sorry I Forgot I had another question which I mentioned earlier.

Are reverse incremental backups subject to the same fragmentation issues as the forever forward incremental ones?

Thanks
H

[nmdange]

One possible suggestion would be to attach your backup volumes to a server running Windows Server 2016 via iSCSI/FC and not CIFS and format the backup repository as ReFS. You could then take advantage of ReFS fast clone which should speed up your merge quite a bit. I have a backup job for a SQL Server Data warehouse VM that generates a lot of change data via daily ETLs. Moving from NTFS to ReFS reduced the merge time from 4+ hours to just a few minutes.

[foggy]

Are reverse incremental backups subject to the same fragmentation issues as the forever forward incremental ones?

Yes, since this method also maintains a single VBK file that gets fragmented with time.

[IHeartCats]

One possible suggestion would be to attach your backup volumes to a server running Windows Server 2016 via iSCSI/FC and not CIFS and format the backup repository as ReFS. You could then take advantage of ReFS fast clone which should speed up your merge quite a bit. I have a backup job for a SQL Server Data warehouse VM that generates a lot of change data via daily ETLs. Moving from NTFS to ReFS reduced the merge time from 4+ hours to just a few minutes.

Thanks.

I watched a vid about the REFS fast clone option - very cool!