Availability for the Always-On Enterprise
Gostev
Veeam Software
Posts: 23112
Liked: 2916 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow backup file merge with REFS

Post by Gostev » May 29, 2017 11:33 am

JimmyO wrote:My first daily merge took 1 hour
With fast clone???

JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: Slow backup file merge with REFS

Post by JimmyO » May 29, 2017 11:44 am

Yes! (I have approx. 1TB of increments /job)

mkretzer
Expert
Posts: 403
Liked: 80 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Slow backup file merge with REFS

Post by mkretzer » May 29, 2017 7:08 pm

Is fragmentation really relevant while fast cloning?
And why is active full slow when there are still 70 TB avaible which have never been written to (we see that in the storage)

Edit:
No matter how high the fragmentation is, we have 96 disks and the system is still "fast-cloning" with only 150 IOPS... This is the speed two disks in a RAID 0 could do random

Gostev
Veeam Software
Posts: 23112
Liked: 2916 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow backup file merge with REFS

Post by Gostev » May 29, 2017 9:32 pm

Fast cloning does not move the actual data around, it's about updating metadata so IOPS requirements are very low.

mkretzer
Expert
Posts: 403
Liked: 80 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Slow backup file merge with REFS

Post by mkretzer » May 30, 2017 8:44 am

Ok yesterday we had to increase the size of a disk of one of our VMs so this morning the entire disk had to be read. The backup normally takes 30 minutes but now this and the other backups running at the same time basically write with less than 1 MB/s and snapshots are getting bigger and bigger. Repo drive disk queue is basically 0 all the time...

@Gostev: If Veeam wants to look at the situation there is now a chance. But i guess only MS can really solve this and i know first level support cannot help us. So we will move back to NTFS very soon....

Gostev
Veeam Software
Posts: 23112
Liked: 2916 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow backup file merge with REFS

Post by Gostev » May 30, 2017 12:35 pm

Did you try rebooting the repository server? I've only heard about such issue (slow writes) once in the past half a year since 9.5 release, and that one time it was fixed by server reboot, so this is not necessarily related to ReFS usage.

mkretzer
Expert
Posts: 403
Liked: 80 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Slow backup file merge with REFS

Post by mkretzer » May 30, 2017 1:16 pm

@gostev Yes, last week when i opened a Veeam case... Your support told me to copy a file with windows explorer and the copy speed was also close to zero.
It is definately REFS: When we had 128 GB RAM we had the server crash, now since we have 384 GB our free RAM goes to 34 % or 35 % (reproducable!) and then the IO goes VERY slow. I guess without the RAM we would see the crash as before.

Now again copy operations with windows explorer are extremly slow...

Edit: One more thing: We have now set all the registry settings for the REFS problem to check if they will change behaviour after the next reboot. If not we will be forced to go back to NTFS

Gostev
Veeam Software
Posts: 23112
Liked: 2916 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow backup file merge with REFS

Post by Gostev » May 30, 2017 2:23 pm

Ah, now I remember - so you're the customer I was referring to anyway ;) so we haven't seen this issue reproduced elsewhere yet.

suprnova
Service Provider
Posts: 33
Liked: never
Joined: Apr 08, 2016 5:15 pm
Contact:

Re: Slow backup file merge with REFS

Post by suprnova » May 30, 2017 4:23 pm

I seem to have a very similar issue, although mine only took a few days to start, and it seems to be only when the larger backups started hitting their retention. I'm still trying to find a correlation but while the fast clone merging is happening, the repository drives are up and down from a monitoring perspective. They show up in Windows explorer, but you can't browse them and no storage data is reported. While there are no merges in progress, jobs are fine.

These are two extreme examples of the larger backups.
Example1:
5/25: 4h:46m
5/26: 10h:4m
5/27: 10h:31m
5/28: 5h:52m
5/29: 34h46m

Example2:
5/25: 3h11m
5/26: 26h58m
5/28: 29h10m

Previous to the above two starting:
5/20: 42s
5/21: 1m
5/23: 21m
5/24: 5m48s

after:
5/25: 13m37s
5/26: 1h13m
5/27: 4h15m
5/28: 3h32m

Gostev
Veeam Software
Posts: 23112
Liked: 2916 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow backup file merge with REFS

Post by Gostev » May 30, 2017 9:07 pm

suprnova wrote:I'm still trying to find a correlation but while the fast clone merging is happening, the repository drives are up and down from a monitoring perspective. They show up in Windows explorer, but you can't browse them and no storage data is reported.
That sounds like an issue discussed in this mega thread. This impacts a portion of our customers and is being investigated by Microsoft.

suprnova
Service Provider
Posts: 33
Liked: never
Joined: Apr 08, 2016 5:15 pm
Contact:

Re: Slow backup file merge with REFS

Post by suprnova » May 31, 2017 10:26 am

I am trying to figure out if it is related. I did put in the test Microsoft fix from support but it made no difference. There was only one merge occurring when the gaps started. Either way, the merging is then taking forever, I am assuming because the repository is unstable.

sconley
Enthusiast
Posts: 38
Liked: 2 times
Joined: Mar 18, 2011 7:36 pm
Full Name: Sean Conley

Re: Slow backup file merge with REFS

Post by sconley » May 31, 2017 4:45 pm

So is the thinking that the slow merge issue may be related to the issue in the REFS 4k thread even given that I have formatted the repo volume using 64 kb blocks? I am curious because if that is the case I will start following that thread more closely. I also see that at least one person has been provided with a test fix that they have, at least in VERY early testing, had positive results with in addressing that issue.

Gostev
Veeam Software
Posts: 23112
Liked: 2916 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow backup file merge with REFS

Post by Gostev » Jun 01, 2017 1:03 am

Right. Since the core issue puts an extremely heavy load on ReFS (to the point where the volume becomes unreachable sometimes), it is perfectly expected that all other I/O activities involving this volume come to crawl as the result.

sconley
Enthusiast
Posts: 38
Liked: 2 times
Joined: Mar 18, 2011 7:36 pm
Full Name: Sean Conley

Re: Slow backup file merge with REFS

Post by sconley » Jun 01, 2017 1:08 am

That makes sense. In that case I will follow the progress of the other thread to see how it plays out over the next few days/weeks. In my case I am lucky that it is not crippling to our daily operations, possibly since we have a relatively small deployment. So at least I am not at the point of contemplating rebuilding the repository yet again. Thanks for the input.

mkretzer
Expert
Posts: 403
Liked: 80 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Slow backup file merge with REFS

Post by mkretzer » Jun 03, 2017 10:11 pm

So merges are taking longer and longer. This time we disabled all scheduled synthetic fulls, no copies and no tape backups were running. Here are the merge times of the first backup that merges on weekends:

Without fast-Clone (same Storage): 5:04
4 weeks before: 0:33
3 weeks before: 0:12
2 weeks before: 2:44
1 week before: 3:12
This week: 3:59

So if this continues REFS is no longer faster than NTFS. Furthermore, NTFS does not have the same issues with active fulls at the same time on a fast storage.
So what changed? The only thing that happened 2 weeks ago was that for the first time the day before the merge retention points were deleted.

@Gostev: Do you have an idea what we can do next? Perhaps increase retention so there are no points deleted and see if backup merges get faster again next week?

Post Reply

Who is online

Users browsing this forum: daniel.triplehorn, Google [Bot] and 46 guests