Synthetic full backup stuck for long time

[JosueM]

Good day everyone.

Today I just noticed a backup job stuck since past saturday, the job is at 99% and current action is Synthetic Full as the image describes.

I've read we can configure to make full instead of synthetic. Should I left done since is 24% for 33.5 hours or should I cancel and try again with full option instead?

Any help or recommendation is appreaciated. Thanks

[foggy]

Josue, what kind of target storage do you have? Probably it just does not provide enough IOPS to perform synthetic activity at decent speed. You could stop the current job and try active fulls instead, how long does the active full take in your case?

[JosueM]

Hello Foggy, target backup repo is a windows 2008 VM runing on a dell powerdge 2950 with local storage SAS disks at 15K RPM configured as raid 5.

Initial full took almost 10 hours.

[Gostev]

How many spindles do you have in RAID, and what is the RAID stripe size?

Being sequential write workload, initial full performance cannot really be used as a point of reference here. In case of synthetic full, we need to be looking at random IOPS capabilities of your backup storage.

Since this is not VMware-specific question, I am going to move this topic into the main forum after your next reply.

[JosueM]

Hello Gostev,

It has 5 spindles and currently I don't have the RAID strip size info at this moment.

BTW since my first post I did stop the job but still in "stopping" status, now it has more than 40 hours running. Should I wait until stops or restart the backup server?

Thanks.

[Gostev]

OK. According to my calculations, creating synthetic full with 1.7 TB of data should normally complete in about 10 hours with five 15K disks in RAID5, and you should be seeing about 40-50 MB/s processing performance. What sort of performance are you seeing in the Perfmon?

Since in your case the process takes at least 4 times longer than 10 hours, there are clearly some performance issues with your backup storage. This could be RAID controller issue due to outdated firmware or cache settings, non-optimal RAID stripe setting (256KB is recommended), some external load, etc.

I highly recommend that you have our support take a look at the debug logs to see what the job is doing before killing it.

Thanks!

[JosueM]

Hello Gostev,

I just came this morning and the job still in "stopping" status now it has almost 56 hours. I'll take the logs and create a ticket to try to find out what the problem is.

Thanks.

[JosueM]

Well in fact exporting the logs also hangs , it stays at compressing logs status. It seems I must reboot the server anyway.

[JosueM]

I did a reboot of the backup repository and after that the backup job was success. I hope that was the only issue.

Thanks.

[stormlight]

Gostev, how do you do your calculations?

OK. According to my calculations, creating synthetic full with 1.7 TB of data should normally complete in about 10 hours with five 15K disks in RAID5, and you should be seeing about 40-50 MB/s processing performance. What sort of performance are you seeing in the Perfmon?

Since in your case the process takes at least 4 times longer than 10 hours, there are clearly some performance issues with your backup storage. This could be RAID controller issue due to outdated firmware or cache settings, non-optimal RAID stripe setting (256KB is recommended), some external load, etc.

I highly recommend that you have our support take a look at the debug logs to see what the job is doing before killing it.

Thanks!

[Gostev]

Pure math. For non-sequential backup file operations, it's all about IOPS.

Storage device IOPS depend on amount of spindles, hard drive speed, RAID type and RAID stripe size.
And all operations require well known amount of I/O per block (synthetic full is 2 I/Os per block, read+write).

Absolute best case synthetic full performance on 15K SAS RAID5 is ~10 MB/s per spindle, meaning ~50 MB/s max with 5 spindles at best.

[stormlight]

Thanks, I was looking for the math itself or the formula.

For example I am having a disconnect with the logic behind 15K SAS RAID5 is 10 MB/s.

For example i want to know how can I find out for my self what a 10K SAS raid 10 max MB/s is? Or any other combination.

[Gostev]

Code: Select all

Functional IOPS = ((Total Raw IOPS × Write%) / (RAID Penalty))+(Total Raw IOPS × Read%)
Operation speed = Block size × (Functional IOPS / IO per block) 

That is assuming your workload's block size matches your RAID stripe size, otherwise there will be an extra respective penalty.

There is an easier way though: just Google for "IOPS calculator", there are many online ones.
They will have all the knowledge built-in, including typical Raw IOPS for different hard drives.

[adrien.herve]

Hello guys,

Since few days we have started some tests on Veeam B&R v8: we have created one Forward Incremental job with transformation of previous incremental backup chain into rollbacks (synthetic full scheduled on saturday, approx 3TB).
So this job creates a synthetic full backup each saturday but it takes a while to generate it on the repository (~16 hours). It will be faster to create a fresh active full from source!
Do you think this is time typical or is there something we can check to generate this full synthetic faster?

Thanks,

[foggy]

Adrien, what kind of target storage do you have?

[adrien.herve]

NetApp E2760A connected to an HP server via SAS. One disk pool of 40TB (DDP), one volume of 40TB mapped to the backup server/proxy.

[foggy]

Please review some considerations above, probably they justify the synthetic full performance in your case.