Slow backup file merge with REFS

Availability for the Always-On Enterprise

Re: Slow backup file merge with REFS

Veeam Logoby Gostev » Mon May 29, 2017 11:33 am

JimmyO wrote:My first daily merge took 1 hour

With fast clone???
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Slow backup file merge with REFS

Veeam Logoby JimmyO » Mon May 29, 2017 11:44 am

Yes! (I have approx. 1TB of increments /job)
JimmyO
Enthusiast
 
Posts: 38
Liked: 2 times
Joined: Sun Apr 27, 2014 8:19 pm

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Mon May 29, 2017 7:08 pm

Is fragmentation really relevant while fast cloning?
And why is active full slow when there are still 70 TB avaible which have never been written to (we see that in the storage)

Edit:
No matter how high the fragmentation is, we have 96 disks and the system is still "fast-cloning" with only 150 IOPS... This is the speed two disks in a RAID 0 could do random
mkretzer
Expert
 
Posts: 342
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby Gostev » Mon May 29, 2017 9:32 pm

Fast cloning does not move the actual data around, it's about updating metadata so IOPS requirements are very low.
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Tue May 30, 2017 8:44 am

Ok yesterday we had to increase the size of a disk of one of our VMs so this morning the entire disk had to be read. The backup normally takes 30 minutes but now this and the other backups running at the same time basically write with less than 1 MB/s and snapshots are getting bigger and bigger. Repo drive disk queue is basically 0 all the time...

@Gostev: If Veeam wants to look at the situation there is now a chance. But i guess only MS can really solve this and i know first level support cannot help us. So we will move back to NTFS very soon....
mkretzer
Expert
 
Posts: 342
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby Gostev » Tue May 30, 2017 12:35 pm

Did you try rebooting the repository server? I've only heard about such issue (slow writes) once in the past half a year since 9.5 release, and that one time it was fixed by server reboot, so this is not necessarily related to ReFS usage.
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Tue May 30, 2017 1:16 pm

@gostev Yes, last week when i opened a Veeam case... Your support told me to copy a file with windows explorer and the copy speed was also close to zero.
It is definately REFS: When we had 128 GB RAM we had the server crash, now since we have 384 GB our free RAM goes to 34 % or 35 % (reproducable!) and then the IO goes VERY slow. I guess without the RAM we would see the crash as before.

Now again copy operations with windows explorer are extremly slow...

Edit: One more thing: We have now set all the registry settings for the REFS problem to check if they will change behaviour after the next reboot. If not we will be forced to go back to NTFS
mkretzer
Expert
 
Posts: 342
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby Gostev » Tue May 30, 2017 2:23 pm

Ah, now I remember - so you're the customer I was referring to anyway ;) so we haven't seen this issue reproduced elsewhere yet.
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Slow backup file merge with REFS

Veeam Logoby suprnova » Tue May 30, 2017 4:23 pm

I seem to have a very similar issue, although mine only took a few days to start, and it seems to be only when the larger backups started hitting their retention. I'm still trying to find a correlation but while the fast clone merging is happening, the repository drives are up and down from a monitoring perspective. They show up in Windows explorer, but you can't browse them and no storage data is reported. While there are no merges in progress, jobs are fine.

These are two extreme examples of the larger backups.
Example1:
5/25: 4h:46m
5/26: 10h:4m
5/27: 10h:31m
5/28: 5h:52m
5/29: 34h46m

Example2:
5/25: 3h11m
5/26: 26h58m
5/28: 29h10m

Previous to the above two starting:
5/20: 42s
5/21: 1m
5/23: 21m
5/24: 5m48s

after:
5/25: 13m37s
5/26: 1h13m
5/27: 4h15m
5/28: 3h32m
suprnova
Service Provider
 
Posts: 21
Liked: never
Joined: Fri Apr 08, 2016 5:15 pm

Re: Slow backup file merge with REFS

Veeam Logoby Gostev » Tue May 30, 2017 9:07 pm

suprnova wrote:I'm still trying to find a correlation but while the fast clone merging is happening, the repository drives are up and down from a monitoring perspective. They show up in Windows explorer, but you can't browse them and no storage data is reported.

That sounds like an issue discussed in this mega thread. This impacts a portion of our customers and is being investigated by Microsoft.
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Slow backup file merge with REFS

Veeam Logoby suprnova » Wed May 31, 2017 10:26 am

I am trying to figure out if it is related. I did put in the test Microsoft fix from support but it made no difference. There was only one merge occurring when the gaps started. Either way, the merging is then taking forever, I am assuming because the repository is unstable.
suprnova
Service Provider
 
Posts: 21
Liked: never
Joined: Fri Apr 08, 2016 5:15 pm

Re: Slow backup file merge with REFS

Veeam Logoby sconley » Wed May 31, 2017 4:45 pm

So is the thinking that the slow merge issue may be related to the issue in the REFS 4k thread even given that I have formatted the repo volume using 64 kb blocks? I am curious because if that is the case I will start following that thread more closely. I also see that at least one person has been provided with a test fix that they have, at least in VERY early testing, had positive results with in addressing that issue.
sconley
Enthusiast
 
Posts: 28
Liked: never
Joined: Fri Mar 18, 2011 7:36 pm
Full Name: Sean Conley

Re: Slow backup file merge with REFS

Veeam Logoby Gostev » Thu Jun 01, 2017 1:03 am

Right. Since the core issue puts an extremely heavy load on ReFS (to the point where the volume becomes unreachable sometimes), it is perfectly expected that all other I/O activities involving this volume come to crawl as the result.
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Slow backup file merge with REFS

Veeam Logoby sconley » Thu Jun 01, 2017 1:08 am

That makes sense. In that case I will follow the progress of the other thread to see how it plays out over the next few days/weeks. In my case I am lucky that it is not crippling to our daily operations, possibly since we have a relatively small deployment. So at least I am not at the point of contemplating rebuilding the repository yet again. Thanks for the input.
sconley
Enthusiast
 
Posts: 28
Liked: never
Joined: Fri Mar 18, 2011 7:36 pm
Full Name: Sean Conley

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Sat Jun 03, 2017 10:11 pm

So merges are taking longer and longer. This time we disabled all scheduled synthetic fulls, no copies and no tape backups were running. Here are the merge times of the first backup that merges on weekends:

Without fast-Clone (same Storage): 5:04
4 weeks before: 0:33
3 weeks before: 0:12
2 weeks before: 2:44
1 week before: 3:12
This week: 3:59

So if this continues REFS is no longer faster than NTFS. Furthermore, NTFS does not have the same issues with active fulls at the same time on a fast storage.
So what changed? The only thing that happened 2 weeks ago was that for the first time the day before the merge retention points were deleted.

@Gostev: Do you have an idea what we can do next? Perhaps increase retention so there are no points deleted and see if backup merges get faster again next week?
mkretzer
Expert
 
Posts: 342
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google Feedfetcher, michael-itnow and 1 guest