ReFS synthetic fulls suddenly very slow

Availability for the Always-On Enterprise

ReFS synthetic fulls suddenly very slow

Veeam Logoby Surfside67 » Sun Jun 24, 2018 10:22 am

Folks,

We are running Backup & Replication 9.5 Update 3 and Windows Agent 2.1.
We have two physical Exchange servers with approx. 7 TB data divided fairly evenly between the two servers.
We run backups every evening at 19:00 and create synthetic fulls on Saturday.
The synthetic full backup files are approx. 2.9 TB and 2.6 TB respectively.
The first four Saturdays the synthetic fulls took around 10 minutes to create, but week #5 the synthetic fulls suddenly took just under 4 hours... And this Saturday around 3 hours.
Is there anyone who can understand what has happened? Is there something we can do to rectify this situation?

Sincerely,
Per Jonsson
Sweden
Surfside67
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Jun 06, 2018 5:41 am
Full Name: Per Jonsson

Re: Synthetic fulls suddenly very slow

Veeam Logoby Gostev » Mon Jun 25, 2018 2:13 pm

Do you use ReFS repository?
Gostev
Veeam Software
 
Posts: 22400
Liked: 2676 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Synthetic fulls suddenly very slow

Veeam Logoby Surfside67 » Tue Jun 26, 2018 6:41 am

Yes, we use ReFS, RAID 5 storage arrays, and dedup in three HP DL380 Gen 10 servers. We were recommended this by our supplier.
Surfside67
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Jun 06, 2018 5:41 am
Full Name: Per Jonsson

Re: Synthetic fulls suddenly very slow

Veeam Logoby Gostev » Tue Jun 26, 2018 3:38 pm 1 person likes this post

There's a ReFS performance regression in May Windows updates that will be fixed in July Windows updates.
Gostev
Veeam Software
 
Posts: 22400
Liked: 2676 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: ReFS synthetic fulls suddenly very slow

Veeam Logoby Surfside67 » Tue Jun 26, 2018 3:54 pm

Thank you very much! :)
Surfside67
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Jun 06, 2018 5:41 am
Full Name: Per Jonsson

Re: ReFS synthetic fulls suddenly very slow

Veeam Logoby Surfside67 » Mon Jul 02, 2018 5:47 pm 1 person likes this post

Folks,

I have made a backup copy job for our two physical Exchange servers, and for the first time it has made a weekly fullbackup according to our GFS archive policy. It turned out well eventually, but it went on for ages... The creation of the GFS weekly fullbackup took 12 hours and 30 minutes. And then the creation of the new synthetic fullbackup for the backup copy chain took an additional 14 hours and 37 minutes, making a total of just over 27 hours... The fullbackup files are approx. 5.6 TB in size, and that may be considered very large, or it may not, I don't really know... But come on! 27 hours?!

We have three identical state-of-the-art, top-of-the-line, cutting-edge HP DL380 G10 servers, each with two Smart Storage Arrays of 90 TB, using RAID 5, and 384 GB of RAM. The best you can get for money, basically... One of the servers is dedicated for archival purposes, so the backup copies are put there.

If all this agony is caused by the mistake that Microsoft made regarding ReFS performance in a Windows Update in May (which is said to be fixed in July), then this must be the biggest mistake that Microsoft has ever made, throughout the history of time!

Sincerely,
Per Jonsson
Sweden
Surfside67
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Jun 06, 2018 5:41 am
Full Name: Per Jonsson

Re: ReFS synthetic fulls suddenly very slow

Veeam Logoby Mgamerz » Mon Jul 02, 2018 7:34 pm

I mean, there was that time they made an operating system that couldn't run more than 49 days...
Mgamerz
Enthusiast
 
Posts: 27
Liked: 5 times
Joined: Fri Sep 29, 2017 8:07 pm

Re: ReFS synthetic fulls suddenly very slow

Veeam Logoby opg70 » Tue Jul 03, 2018 2:31 pm 1 person likes this post

Surfside67 wrote: two Smart Storage Arrays of 90 TB, using RAID 5, and 384 GB of RAM.

Just as a comment: from what I've read I don't believe RAID5 is safe on these sizes, you should really think about going to RAID6 (despite the associated loss of performance) as you may lose your array in case of any problems with the disks. The rebuild times (and performance loss during that time) when a hard drive fails though will be large in either case though due to your array size.
opg70
Influencer
 
Posts: 20
Liked: 3 times
Joined: Sun Oct 06, 2013 8:48 am

Re: ReFS synthetic fulls suddenly very slow

Veeam Logoby Surfside67 » Tue Jul 03, 2018 10:01 pm

We were recommended RAID 5 by our supplier because, as you point out, it is faster than RAID 6. We have 12 disks in each array, and one of them is an idle hot spare that kicks in when one of the other 11 fails. We did not get any information about that RAID 5 should be a bad choice for these array sizes...

There is always something, isn't there...

PJ
Surfside67
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Jun 06, 2018 5:41 am
Full Name: Per Jonsson

Re: ReFS synthetic fulls suddenly very slow

Veeam Logoby opg70 » Wed Jul 04, 2018 4:00 pm 1 person likes this post

Did they take into consideration that RAID5 is only recommended by many only for small arrays or ones created with SSD's due to the likelihood of URE's making a rebuild of the array fail? There are many articles about this, starting from more than 10 years ago, for ex: Why RAID 5 Stops working in 2009. Basically the likely error rate of disks makes the likelihood of a bad sector existing high enough that during an array rebuild after a drive failure - which will take days or weeks at the 90TB range - it is likely to occur, and as the array has no more redundancy the whole array may refuse to rebuild and be lost. Also read Hot Spare or a Hot Mess, as hot spares in a RAID5 array can actually cause it to fail rather than save the day.
opg70
Influencer
 
Posts: 20
Liked: 3 times
Joined: Sun Oct 06, 2013 8:48 am

Re: ReFS synthetic fulls suddenly very slow

Veeam Logoby Surfside67 » Thu Jul 05, 2018 3:57 pm

Thanks for your input and information!
I have relayed it to my colleagues who also work with these servers.

PJ
Surfside67
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Jun 06, 2018 5:41 am
Full Name: Per Jonsson

Re: ReFS synthetic fulls suddenly very slow

Veeam Logoby Surfside67 » Fri Jul 06, 2018 7:08 am

Is it possible to change from RAID 5 to RAID 6 and keep all the data intact?

PJ
Surfside67
Influencer
 
Posts: 17
Liked: 1 time
Joined: Wed Jun 06, 2018 5:41 am
Full Name: Per Jonsson

Re: ReFS synthetic fulls suddenly very slow

Veeam Logoby opg70 » Fri Jul 06, 2018 11:53 am

In our environment, yes it was possible to go to the Smart Storage Administrator and just change the array level (after deactivating the hot spare, as it will become part of the array) by going to the logical drive and choosing 'Migrate RAID/Strip Size'. It will take a while though - if you have the space on the other array, the faster option would be to move the data to the other disk then delete the array and recreate it - then do the same again for the other array.
opg70
Influencer
 
Posts: 20
Liked: 3 times
Joined: Sun Oct 06, 2013 8:48 am

Re: ReFS synthetic fulls suddenly very slow

Veeam Logoby pswarowski » Mon Jul 09, 2018 12:20 am

pswarowski
Service Provider
 
Posts: 1
Liked: never
Joined: Wed Oct 25, 2017 10:50 pm
Full Name: Peter Swarowski


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: cfaragi, DaveWatkins, jason.smith079, servicebkm and 46 guests