Slow backup file merge with REFS

Availability for the Always-On Enterprise

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Sat Jun 03, 2017 11:11 pm

One more thing: Our write speed is extemly slow again just because two synthetics (fast clone) are running. Disk latency is still between 2 and 4 ms so disk is not loaded. Before the synthetics started speed was normal.
mkretzer
Expert
 
Posts: 386
Liked: 77 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby suprnova » Sun Jun 04, 2017 2:31 pm

That's really strange, we've had basically all of the other issues, but our write speeds are normal while fast clone synthetics are running. We are running the experimental refs.sys driver though so maybe that is a potential fix. But Veeam support is unable to report what exactly the new driver fixes.
suprnova
Service Provider
 
Posts: 33
Liked: never
Joined: Fri Apr 08, 2016 5:15 pm

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Sun Jun 04, 2017 2:53 pm

I mean it must be something in the REFS driver. With NTFS there is REAL LOAD on the volume while merging but we never ever had any issues getting the data to disk while all the merges ran. We basically never had to care if the active fulls were going at the same time as synthetics - it just worked (same backend storage).

Where do i get that REFS driver and is it usable for production?
mkretzer
Expert
 
Posts: 386
Liked: 77 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby suprnova » Sun Jun 04, 2017 10:42 pm

Our issue has always been merges with NTFS, just way too slow, but we are also a service provider where there may be 100+ merges overnight. Veeam support can provide the driver. It's tough to say if it is production worthy or not, but I can say it has not made things worse.
suprnova
Service Provider
 
Posts: 33
Liked: never
Joined: Fri Apr 08, 2016 5:15 pm

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Thu Jun 08, 2017 5:32 am

Ok setting all the REFS Reg settings Microsoft recommends now leads to even slower merges and also slower write speed to the paralell running backups. Memory usage is lower but now our backup window is too long... We will remove the settings again and increase retention as a test.
mkretzer
Expert
 
Posts: 386
Liked: 77 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Sun Jun 11, 2017 7:11 am

Without deleting retention points our merges doubled in speed! We are not at the original "fast clone" speed from the beginning but its much better. Even an active full does not immediately stall when a merge is running in parallel.

Right now we have enough space to host even another week so we will see if the merges will even get faster next weekend.

I wonder: is there anything i can optimize with better hardware for the repo server? I simply cannot see which ressource is limiting the REFS to merge faster/delete restore points without issue. RAM is no issue anymore since we increased to 384 GB, CPU usage is low and disk queue of the REPO is 0. Buying a new server would still be cheaper than buying more storage and going back to NTFS :-)
mkretzer
Expert
 
Posts: 386
Liked: 77 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Sun Jun 18, 2017 9:11 am

Ok even without deletion of restore points everything got even slower with 6 weeks of backups on disk. We will go back to NTFS as we see no way to get this working right now...
mkretzer
Expert
 
Posts: 386
Liked: 77 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby mikegodwin » Mon Jul 24, 2017 4:50 pm

Has anyone tried disabling FileIntegrity in ReFS to see if it improves performance?
mikegodwin
Enthusiast
 
Posts: 53
Liked: 1 time
Joined: Fri Oct 12, 2012 12:28 am
Full Name: Mike Godwin

Re: Slow backup file merge with REFS

Veeam Logoby Lunatic Magnet » Fri Nov 24, 2017 9:12 pm

Several people have stated that they were switching to NTFS. For anyone that has, have you notice improvements and has the issue re-appeared?

We are facing a similar issue. Veeam B&R 9.5.0.1038, repository is 64K REFS on a physical Dell R730XD in a RAID 6. Backups are running between 150-350MB/s but a merge of a 330GB incremental to a 1.45TB full took 18 hours and brought the system to an almost standstill. Previously this was tremendously faster. I've had to disable health checks and defrags because of the time it was taking. Strangely backup copies are running normally to a secondary repository on identical hardware. OS is Windows 2016 build 1607 (14393.1884) with all latest patches. I'm working with support (02386355) and just read through this and the mega 4k thread hoping for answers.
Lunatic Magnet
Influencer
 
Posts: 15
Liked: 3 times
Joined: Wed Oct 18, 2017 6:40 pm

Re: Slow backup file merge with REFS

Veeam Logoby JaySt » Sun Nov 26, 2017 6:17 pm

Interesting. I also have a customer running 730xd hardware and raid6 with Windows server 2016 refs . No issues so far. Running a few TBs of hyperv vms with weekly synth fulls currently i believe

What are the exact drive types used in your case? 12Gbps NLSAS or 6Gbps Sata?
Veeam Certified Engineer
JaySt
Service Provider
 
Posts: 84
Liked: 18 times
Joined: Tue Jun 09, 2015 7:08 pm
Full Name: JaySt

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Mon Nov 27, 2017 5:21 am

NTFS is not fast but predicatable. The issue cannot "re-appear" there because it never was there with NTFS because in NTFS the blocks are just copied. So if that would be an issue NTFS would had an issue in any application.

BTW for REFS the type of storage does not matter for the merge issue - the issue is not the backend speed (look at the disk queue lenght it should be very low)!

Do you use synthetic fulls for your primary backup? If so you have more block-cloned blocks and that should mean the issue is more likely to appear.
mkretzer
Expert
 
Posts: 386
Liked: 77 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Slow backup file merge with REFS

Veeam Logoby JaySt » Mon Nov 27, 2017 11:38 am

i'm not that sure about stating "type of storage does not matter" regarding the backend storage type. Performance of the backend isn't an issue during ReFS merge, that's quite clear. But other features like cache or the handling of certain types of IO could differ between technologies. I see different ReFS user experiences in this forum where almost in every case backend performance shouldn't be an issue, but something else is causing issues. For example, the use of storage spaces has some sort of (negative?) effect on this as it seems.
Veeam Certified Engineer
JaySt
Service Provider
 
Posts: 84
Liked: 18 times
Joined: Tue Jun 09, 2015 7:08 pm
Full Name: JaySt

Re: Slow backup file merge with REFS

Veeam Logoby Lunatic Magnet » Mon Nov 27, 2017 2:32 pm

JaySt wrote:Interesting. I also have a customer running 730xd hardware and raid6 with Windows server 2016 refs . No issues so far. Running a few TBs of hyperv vms with weekly synth fulls currently i believe

What are the exact drive types used in your case? 12Gbps NLSAS or 6Gbps Sata?


They're 12Gps drives. Our backups are forever-forward incremental with 30 restore points. No active or synthetic fulls.
Lunatic Magnet
Influencer
 
Posts: 15
Liked: 3 times
Joined: Wed Oct 18, 2017 6:40 pm

Re: Slow backup file merge with REFS

Veeam Logoby JaySt » Tue Nov 28, 2017 8:41 pm

So same drive types it seems.
Strange. No issues here so far. Have you tried running a synth full? same merge times?
Veeam Certified Engineer
JaySt
Service Provider
 
Posts: 84
Liked: 18 times
Joined: Tue Jun 09, 2015 7:08 pm
Full Name: JaySt

Re: Slow backup file merge with REFS

Veeam Logoby mkretzer » Wed Nov 29, 2017 5:54 am

Again: Why should the drive type matter for an issue which happens after a certain amount of block cloned data is on the disk?
mkretzer
Expert
 
Posts: 386
Liked: 77 times
Joined: Thu Dec 17, 2015 7:17 am

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: No registered users and 70 guests