ReFS synthetic fulls suddenly very slow

[perjonsson1960]

Folks,

We are running Backup & Replication 9.5 Update 3 and Windows Agent 2.1.
We have two physical Exchange servers with approx. 7 TB data divided fairly evenly between the two servers.
We run backups every evening at 19:00 and create synthetic fulls on Saturday.
The synthetic full backup files are approx. 2.9 TB and 2.6 TB respectively.
The first four Saturdays the synthetic fulls took around 10 minutes to create, but week #5 the synthetic fulls suddenly took just under 4 hours... And this Saturday around 3 hours.
Is there anyone who can understand what has happened? Is there something we can do to rectify this situation?

Sincerely,
Per Jonsson
Sweden

[Gostev]

Do you use ReFS repository?

[perjonsson1960]

Yes, we use ReFS, RAID 5 storage arrays, and dedup in three HP DL380 Gen 10 servers. We were recommended this by our supplier.

[Gostev]

There's a ReFS performance regression in May Windows updates that will be fixed in July Windows updates.

[perjonsson1960]

Thank you very much!

[perjonsson1960]

Folks,

I have made a backup copy job for our two physical Exchange servers, and for the first time it has made a weekly fullbackup according to our GFS archive policy. It turned out well eventually, but it went on for ages... The creation of the GFS weekly fullbackup took 12 hours and 30 minutes. And then the creation of the new synthetic fullbackup for the backup copy chain took an additional 14 hours and 37 minutes, making a total of just over 27 hours... The fullbackup files are approx. 5.6 TB in size, and that may be considered very large, or it may not, I don't really know... But come on! 27 hours?!

We have three identical state-of-the-art, top-of-the-line, cutting-edge HP DL380 G10 servers, each with two Smart Storage Arrays of 90 TB, using RAID 5, and 384 GB of RAM. The best you can get for money, basically... One of the servers is dedicated for archival purposes, so the backup copies are put there.

If all this agony is caused by the mistake that Microsoft made regarding ReFS performance in a Windows Update in May (which is said to be fixed in July), then this must be the biggest mistake that Microsoft has ever made, throughout the history of time!

Sincerely,
Per Jonsson
Sweden

[Mgamerz]

I mean, there was that time they made an operating system that couldn't run more than 49 days...

[opg70]

two Smart Storage Arrays of 90 TB, using RAID 5, and 384 GB of RAM.

Just as a comment: from what I've read I don't believe RAID5 is safe on these sizes, you should really think about going to RAID6 (despite the associated loss of performance) as you may lose your array in case of any problems with the disks. The rebuild times (and performance loss during that time) when a hard drive fails though will be large in either case though due to your array size.

[perjonsson1960]

We were recommended RAID 5 by our supplier because, as you point out, it is faster than RAID 6. We have 12 disks in each array, and one of them is an idle hot spare that kicks in when one of the other 11 fails. We did not get any information about that RAID 5 should be a bad choice for these array sizes...

There is always something, isn't there...

PJ

[opg70]

Did they take into consideration that RAID5 is only recommended by many only for small arrays or ones created with SSD's due to the likelihood of URE's making a rebuild of the array fail? There are many articles about this, starting from more than 10 years ago, for ex: Why RAID 5 Stops working in 2009. Basically the likely error rate of disks makes the likelihood of a bad sector existing high enough that during an array rebuild after a drive failure - which will take days or weeks at the 90TB range - it is likely to occur, and as the array has no more redundancy the whole array may refuse to rebuild and be lost. Also read Hot Spare or a Hot Mess, as hot spares in a RAID5 array can actually cause it to fail rather than save the day.

[perjonsson1960]

Thanks for your input and information!
I have relayed it to my colleagues who also work with these servers.

PJ

[perjonsson1960]

Is it possible to change from RAID 5 to RAID 6 and keep all the data intact?

PJ

[opg70]

In our environment, yes it was possible to go to the Smart Storage Administrator and just change the array level (after deactivating the hot spare, as it will become part of the array) by going to the logical drive and choosing 'Migrate RAID/Strip Size'. It will take a while though - if you have the space on the other array, the faster option would be to move the data to the other disk then delete the array and recreate it - then do the same again for the other array.

[pswarowski]

Check this post to see if it is AV causing the slow down, https://forums.veeam.com/veeam-backup-r ... ml#p219823

[perjonsson1960]

Folks,

After the cumulative Windows Update for July, the synthetic full backups of our two Exchange servers, that took around 4 hours each during the ReFS problem, took 6 and 8 minutes respectively, and the backup copy of the Exchange servers, which took 12 hours each for the GFS full backup file merge, and an additional 12 hours each for the synthetic full restore point creation, took a total of about 22 minutes for each server!

That is what I call an improvement! Life is worth living again!

PJ