Slow backup file merge with REFS

Availability for the Always-On Enterprise

Slow backup file merge with REFS

Veeam Logoby sconley » Thu May 04, 2017 4:05 pm

I have been seeing a significant slow down in the time the merge operations. One of our production jobs was taking less than 15 minutes to complete the merge process when we initially did the upgrade to 9.5 with REFS. This operation is now regularly taking over 2 hours. The action log still shows fast clone, so I'm unsure what could be causing this. Our environment is fairly simple. It is entirely virtual running vSphere 6/vCenter 6, with our B&R server running server 2012 R2, and a single proxy/repository VM running server 2016 with the repository space directly attached to the OS via iSCSI. We use backup copy jobs to copy offsite to a proxy/repository VM that has an identical configuration and I am not seeing the same issues on the remote end when it goes through the merge process.

I am also going to open a case with support and get them to look into this. I just wanted to post here and see if anyone has experienced anything similar.
sconley
Enthusiast
 
Posts: 28
Liked: never
Joined: Fri Mar 18, 2011 7:36 pm
Full Name: Sean Conley

Re: Slow backup file merge with REFS

Veeam Logoby Gostev » Thu May 04, 2017 11:30 pm

If you don't see the same with Backup Copy job, then this could potentially be an impact from fragmentation due to the primary backup job mode you have chosen. Which one is it?
Gostev
Veeam Software
 
Posts: 21442
Liked: 2362 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Slow backup file merge with REFS

Veeam Logoby sconley » Thu May 04, 2017 11:41 pm

All backup jobs are forever incremental with monthly compact and defragment maintenance scheduled. Looking at the job history the compact operations seem to have been running as expected.
sconley
Enthusiast
 
Posts: 28
Liked: never
Joined: Fri Mar 18, 2011 7:36 pm
Full Name: Sean Conley

Re: Slow backup file merge with REFS

Veeam Logoby tsightler » Fri May 05, 2017 12:25 am 2 people like this post

I'm curious if your ReFS volume is formatted with 4K clusters or the Veeam recommended 64K. I've seen this issue with 4K clusters in a number of cases, although on a smaller scale, but similar ratio (for example a merge that was taking 5 minutes slowly degrades to taking 40+ minutes).
tsightler
Veeam Software
 
Posts: 4802
Liked: 1760 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Slow backup file merge with REFS

Veeam Logoby sconley » Fri May 05, 2017 3:25 pm

The volume is formatted with 64k clusters. I actually rebuilt the repository at one point as I started with the default 4k cluster size before reading about some of the nightmare stories with 4k allocation size. Here is the output from fsutil:

C:\Windows\system32>fsutil fsinfo refsinfo e:
REFS Volume Serial Number : 0x5c2eae7f2eae51b6
REFS Version : 3.1
Number Sectors : 0x000000117ffa0000
Total Clusters : 0x0000000022fff400
Free Clusters : 0x00000000083d8585
Total Reserved : 0x00000000001ec09b
Bytes Per Sector : 512
Bytes Per Physical Sector : 4096
Bytes Per Cluster : 65536
Checksum Type: CHECKSUM_TYPE_NONE
sconley
Enthusiast
 
Posts: 28
Liked: never
Joined: Fri Mar 18, 2011 7:36 pm
Full Name: Sean Conley

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Sun May 28, 2017 4:42 am

Ok our system now is no longer crashing (as we discussed in the "REFS 4K horror story") but is also extremly slow merging. Also, other backups doing incrementals at the same time are very slow as well.

We opened a Veeam ticket (02163118) but when Veeam support found out that the issue also happens with normal file copies to the REFS they basically told us that it is not a Veeam issue. Ressource monitor show that the disk are in no way at maximum load.

We are thinking of going back to NTFS...
mkretzer
Expert
 
Posts: 330
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby haslund » Sun May 28, 2017 10:44 am

Can you share more details of your storage used for this backup repository? Is it local DAS or some remote storage such as iSCSI/FC?
Rasmus Haslund
Principal Technologist, Global Education Services @ Veeam Software
Veeam Certified Architect #1 | Veeam Certified Trainer #4 [v7,v8,v9] | Veeam Certified Trainer Mentor #1
Twitter: @haslund
Blog: www.perfectcloud.org
haslund
Veeam Software
 
Posts: 275
Liked: 50 times
Joined: Thu Feb 16, 2012 7:35 am
Location: Denmark
Full Name: Rasmus Haslund

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Sun May 28, 2017 12:10 pm

Storage is a dedicated Hitachi HUS110 FC SAN system with 96 disks in a RAID 60 Configuration (8 Disks per RAID set). The system is quite fast. Latency in perfmon shows as 2-8 ms, so the storage itself is not the issue. Disk queue is 0 all the time.
Initial backups went on with 600 - 800 MB/s. Yesterday a periodic active full only reached 52 MB/s with 93% target bottleneck. Right now there is a synthetic running and nothing else and it is at 78 % after nearly 7 hours.
Two weeks ago before the issues started Synthetic full took little more than an hour - with other jobs running in parallel.

The storage itself received only ~100 IO/s so this could be handled by one disk - but there are 96 disks avaiable just for this one job and the system does not much.

REFS is 64 k, we added RAM so we have 384 GB now and latest MS hotfixes with registry setting 1 for REFS is applied. RAM free is 65 %.
mkretzer
Expert
 
Posts: 330
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby haslund » Sun May 28, 2017 12:13 pm

I assume you are utilizing multipathing to access the Hitachi disk system, are you using only Windows MPIO or did you install any special software from Hitachi? Would it be possible to test with only a single path active?
Rasmus Haslund
Principal Technologist, Global Education Services @ Veeam Software
Veeam Certified Architect #1 | Veeam Certified Trainer #4 [v7,v8,v9] | Veeam Certified Trainer Mentor #1
Twitter: @haslund
Blog: www.perfectcloud.org
haslund
Veeam Software
 
Posts: 275
Liked: 50 times
Joined: Thu Feb 16, 2012 7:35 am
Location: Denmark
Full Name: Rasmus Haslund

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Sun May 28, 2017 1:02 pm

We use windows MPIO. But I know this system for 4 years now. It is capable of much IO single path or not.

The problem is definately REFS. We never had this with NTFS. This is our third try to implement REFS. With our second try we used a new Fujitsu Eternus DX60 S3 and the exact same thing happened. We also thought the storage is the problem and even let the vendor check that.

As i said latencies show very good values in perfmon.

Merges have finished now and i am going to test normal incremental backups to see if everything behaves normal again.
mkretzer
Expert
 
Posts: 330
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby haslund » Sun May 28, 2017 7:14 pm

I completely understand and respect your thoughts here. I am trying to look across customer posts and see quite a few are using FC or iSCSI and just wonder if anything is connected to the multipathing. Is there any chance it could be tested - just for the purpose of confirming it does not have any impact?
Rasmus Haslund
Principal Technologist, Global Education Services @ Veeam Software
Veeam Certified Architect #1 | Veeam Certified Trainer #4 [v7,v8,v9] | Veeam Certified Trainer Mentor #1
Twitter: @haslund
Blog: www.perfectcloud.org
haslund
Veeam Software
 
Posts: 275
Liked: 50 times
Joined: Thu Feb 16, 2012 7:35 am
Location: Denmark
Full Name: Rasmus Haslund

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Sun May 28, 2017 8:28 pm

We could do this - but i think there is another reason: Those using FC or ISCSI have external arrays which often means much bigger repositories. And since the problem was not there right away it is very likely that the number of used/fast cloned blocks has something to do with if the problem occours...
mkretzer
Expert
 
Posts: 330
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby haslund » Sun May 28, 2017 8:52 pm

mkretzer wrote:Since the problem was not there right away it is very likely that the number of used/fast cloned blocks has something to do with if the problem occours...

This seems to align with what the good @tsightler commented almost a month ago here https://forums.veeam.com/post239915.html#p239915
Rasmus Haslund
Principal Technologist, Global Education Services @ Veeam Software
Veeam Certified Architect #1 | Veeam Certified Trainer #4 [v7,v8,v9] | Veeam Certified Trainer Mentor #1
Twitter: @haslund
Blog: www.perfectcloud.org
haslund
Veeam Software
 
Posts: 275
Liked: 50 times
Joined: Thu Feb 16, 2012 7:35 am
Location: Denmark
Full Name: Rasmus Haslund

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Mon May 29, 2017 4:33 am

Yes! Problem is at the same time we started to do periodic active fulls again also the first backups went out of retention. That is the reason we now disabled active full to see if it works better.

Furthermore, we did not do a reboot after the weekend but the system seems to have returned to normal speed. One thing i found after checking our monitoring system was that memory usage while active full was running went up 120 GB in the matter of half an hour. Shortly after, WMI cut out for 15 minutes as the memory value recovered.
Is there perhaps a fixed value for the REFS filesystem how much memory the driver (?) can use and if that is reached system becomes kind of unstable?
mkretzer
Expert
 
Posts: 330
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby JimmyO » Mon May 29, 2017 7:19 am

Having the exact same issue - ReFS seems to cause a lot of fragmentation. My first daily merge took 1 hour, now a month or so later, it takes up to 30 hours, meaning that I will no longer be able to do daily backups.....
JimmyO
Enthusiast
 
Posts: 37
Liked: 2 times
Joined: Sun Apr 27, 2014 8:19 pm

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: No registered users and 29 guests