Comprehensive data protection for all workloads
Post Reply
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

ReFS synthetic fulls suddenly very slow

Post by perjonsson1960 »

Folks,

We are running Backup & Replication 9.5 Update 3 and Windows Agent 2.1.
We have two physical Exchange servers with approx. 7 TB data divided fairly evenly between the two servers.
We run backups every evening at 19:00 and create synthetic fulls on Saturday.
The synthetic full backup files are approx. 2.9 TB and 2.6 TB respectively.
The first four Saturdays the synthetic fulls took around 10 minutes to create, but week #5 the synthetic fulls suddenly took just under 4 hours... And this Saturday around 3 hours.
Is there anyone who can understand what has happened? Is there something we can do to rectify this situation?

Sincerely,
Per Jonsson
Sweden
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Synthetic fulls suddenly very slow

Post by Gostev »

Do you use ReFS repository?
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Synthetic fulls suddenly very slow

Post by perjonsson1960 »

Yes, we use ReFS, RAID 5 storage arrays, and dedup in three HP DL380 Gen 10 servers. We were recommended this by our supplier.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Synthetic fulls suddenly very slow

Post by Gostev » 1 person likes this post

There's a ReFS performance regression in May Windows updates that will be fixed in July Windows updates.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by perjonsson1960 »

Thank you very much! :)
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by perjonsson1960 » 1 person likes this post

Folks,

I have made a backup copy job for our two physical Exchange servers, and for the first time it has made a weekly fullbackup according to our GFS archive policy. It turned out well eventually, but it went on for ages... The creation of the GFS weekly fullbackup took 12 hours and 30 minutes. And then the creation of the new synthetic fullbackup for the backup copy chain took an additional 14 hours and 37 minutes, making a total of just over 27 hours... The fullbackup files are approx. 5.6 TB in size, and that may be considered very large, or it may not, I don't really know... But come on! 27 hours?!

We have three identical state-of-the-art, top-of-the-line, cutting-edge HP DL380 G10 servers, each with two Smart Storage Arrays of 90 TB, using RAID 5, and 384 GB of RAM. The best you can get for money, basically... One of the servers is dedicated for archival purposes, so the backup copies are put there.

If all this agony is caused by the mistake that Microsoft made regarding ReFS performance in a Windows Update in May (which is said to be fixed in July), then this must be the biggest mistake that Microsoft has ever made, throughout the history of time!

Sincerely,
Per Jonsson
Sweden
Mgamerz
Expert
Posts: 159
Liked: 28 times
Joined: Sep 29, 2017 8:07 pm
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by Mgamerz » 1 person likes this post

I mean, there was that time they made an operating system that couldn't run more than 49 days...
opg70
Influencer
Posts: 24
Liked: 3 times
Joined: Oct 06, 2013 8:48 am
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by opg70 » 1 person likes this post

Surfside67 wrote: two Smart Storage Arrays of 90 TB, using RAID 5, and 384 GB of RAM.
Just as a comment: from what I've read I don't believe RAID5 is safe on these sizes, you should really think about going to RAID6 (despite the associated loss of performance) as you may lose your array in case of any problems with the disks. The rebuild times (and performance loss during that time) when a hard drive fails though will be large in either case though due to your array size.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by perjonsson1960 »

We were recommended RAID 5 by our supplier because, as you point out, it is faster than RAID 6. We have 12 disks in each array, and one of them is an idle hot spare that kicks in when one of the other 11 fails. We did not get any information about that RAID 5 should be a bad choice for these array sizes...

There is always something, isn't there...

PJ
opg70
Influencer
Posts: 24
Liked: 3 times
Joined: Oct 06, 2013 8:48 am
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by opg70 » 1 person likes this post

Did they take into consideration that RAID5 is only recommended by many only for small arrays or ones created with SSD's due to the likelihood of URE's making a rebuild of the array fail? There are many articles about this, starting from more than 10 years ago, for ex: Why RAID 5 Stops working in 2009. Basically the likely error rate of disks makes the likelihood of a bad sector existing high enough that during an array rebuild after a drive failure - which will take days or weeks at the 90TB range - it is likely to occur, and as the array has no more redundancy the whole array may refuse to rebuild and be lost. Also read Hot Spare or a Hot Mess, as hot spares in a RAID5 array can actually cause it to fail rather than save the day.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by perjonsson1960 »

Thanks for your input and information!
I have relayed it to my colleagues who also work with these servers.

PJ
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by perjonsson1960 »

Is it possible to change from RAID 5 to RAID 6 and keep all the data intact?

PJ
opg70
Influencer
Posts: 24
Liked: 3 times
Joined: Oct 06, 2013 8:48 am
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by opg70 »

In our environment, yes it was possible to go to the Smart Storage Administrator and just change the array level (after deactivating the hot spare, as it will become part of the array) by going to the logical drive and choosing 'Migrate RAID/Strip Size'. It will take a while though - if you have the space on the other array, the faster option would be to move the data to the other disk then delete the array and recreate it - then do the same again for the other array.
pswarowski
Service Provider
Posts: 1
Liked: never
Joined: Oct 25, 2017 10:50 pm
Full Name: Peter Swarowski
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by pswarowski »

Check this post to see if it is AV causing the slow down, https://forums.veeam.com/veeam-backup-r ... ml#p219823
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: ReFS synthetic fulls suddenly very slow

Post by perjonsson1960 » 1 person likes this post

Folks,

After the cumulative Windows Update for July, the synthetic full backups of our two Exchange servers, that took around 4 hours each during the ReFS problem, took 6 and 8 minutes respectively, and the backup copy of the Exchange servers, which took 12 hours each for the GFS full backup file merge, and an additional 12 hours each for the synthetic full restore point creation, took a total of about 22 minutes for each server!

That is what I call an improvement! Life is worth living again! :D

PJ
Post Reply

Who is online

Users browsing this forum: No registered users and 229 guests