REFS 4k horror story

Availability for the Always-On Enterprise

Re: REFS 4k horror story

Veeam Logoby mkretzer » Thu Jul 20, 2017 9:35 am 2 people like this post

@ferrus We have seen three issues:

- Crash by high memory usage: Especially with 4k, Per-VM. Fixed in our system by increasing RAM to 384 GB. Should be fixed by the update as well.
- Slow merges, every week it gets slower: Seems to be fixed by the update.
- Active full at the same time as merges are running is EXTREMLY slow (~1-2 MB/s instead of 400+ MB/s): We will test this tomorrow.
- Deletes cause REFS to get slower in general: The first test seems to suggest that this is also fixed by the update.
mkretzer
Expert
 
Posts: 338
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS 4k horror story

Veeam Logoby ferrus » Thu Jul 20, 2017 10:03 am

The server we're migrating is a single job/single VM repository, albeit 18TB in size. So it looks as though the only untested issue won't be a problem for us.
This gives me a lot more confidence.

I wonder if this (after testing), will affect Veeams recommendation of 64 vs 4k?
ferrus
Veeam ProPartner
 
Posts: 137
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: REFS 4k horror story

Veeam Logoby dellock6 » Thu Jul 20, 2017 10:36 am

Any comment on the 3 registry keys that came with the test hotfix? Are they still needed with this latest update?
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5118
Liked: 1361 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: REFS 4k horror story

Veeam Logoby JimmyO » Thu Jul 20, 2017 10:48 am

mkretzer wrote:I can't believe it but in the first two tests this hotfix fixed our slow merge speed!

We did not do any deletes yet on the volume (this will be our next test), but backup merge speed got slower with every week nevertheless:

1 week after new REFS+active full: 0:08:56
2 week after new REFS+active full: 0:18:24
3 week after new REFS+active full: 0:31:59
4 week after new REFS+active full: 4:17:41

After Patch / 5 weeks after new REFS+active full: 0:09:03

And this after we just migrated 90 % of our backups back to NTFS :-(


Sounds good - I´m testing right now. Do you use any refs registry settings?
JimmyO
Enthusiast
 
Posts: 37
Liked: 2 times
Joined: Sun Apr 27, 2014 8:19 pm

Re: REFS 4k horror story

Veeam Logoby mkretzer » Thu Jul 20, 2017 11:55 am

Only the most basic RefsEnableLargeWorkingSetTrim.
mkretzer
Expert
 
Posts: 338
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS 4k horror story

Veeam Logoby suprnova » Thu Jul 20, 2017 1:10 pm

Initial results with the MS update have been positive for us as well, overall incremental backup speeds have improved. I removed the hotfix registry keys and I am also just using RefsEnableLargeWorkingSetTrim.
suprnova
Service Provider
 
Posts: 21
Liked: never
Joined: Fri Apr 08, 2016 5:15 pm

Re: REFS 4k horror story

Veeam Logoby Cicadymn » Thu Jul 20, 2017 3:58 pm

ferrus wrote:I'm just about to migrate one of our Veeam repositories over to ReFS out of necessity - so this really is good news.
There's quite a few ReFS threads on here - I need to find out how many other existing issues there are to look out for.

Just be sure to use 64K block size on ReFS. I believe 4K is the default, so you'll need to change it to 64K. They've had a lot less issues than those of us on 4K.
Cicadymn
Influencer
 
Posts: 20
Liked: 5 times
Joined: Mon Jan 30, 2017 7:42 pm
Full Name: Sam

Re: REFS 4k horror story

Veeam Logoby kubimike » Thu Jul 20, 2017 4:51 pm

Anyone else end up with ReFS version '10.0.14393.1532 ' ??
kubimike
Expert
 
Posts: 243
Liked: 23 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby SBarrett847 » Thu Jul 20, 2017 4:59 pm

Yep that's what i have too.
SBarrett847
Service Provider
 
Posts: 174
Liked: 28 times
Joined: Tue Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett

Re: REFS 4k horror story

Veeam Logoby kubimike » Thu Jul 20, 2017 5:02 pm

Well I guess msft typed the wrong number in the CSV file, it lists '10.0.14393.1408'
kubimike
Expert
 
Posts: 243
Liked: 23 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby kubimike » Fri Jul 21, 2017 1:37 am

New patch is a flop, as soon as retention hits, say goodnight dick. Going back to the beta driver. UGHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

The computer has rebooted from a bugcheck. The bugcheck was: 0x00000133 (0x0000000000000001, 0x0000000000001e00, 0xfffff801e1019540, 0x0000000000000000). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 0819dd2a-6290-44e4-9bf4-25363b2282d6.
kubimike
Expert
 
Posts: 243
Liked: 23 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby JimmyO » Fri Jul 21, 2017 7:07 am

Same for me - the patch makes little or no difference when doing daily merge of my increments. :x
JimmyO
Enthusiast
 
Posts: 37
Liked: 2 times
Joined: Sun Apr 27, 2014 8:19 pm

Re: REFS 4k horror story

Veeam Logoby JaySt » Fri Jul 21, 2017 7:25 am

if this patch would fix the problems seen in this thread and for which Veeam is so actively trying come up with a fix with MS , i think we would have seen some sort of announcement through Gostev maybe, something like "we're getting close to a fix.. hold on...". Would be quite special to have a patch (or THE patch) drop down from the sky like this.
Or did i miss something (possible... with this large thread).
Veeam Certified Engineer
JaySt
Service Provider
 
Posts: 56
Liked: 3 times
Joined: Tue Jun 09, 2015 7:08 pm
Full Name: JaySt

Re: REFS 4k horror story

Veeam Logoby mkretzer » Fri Jul 21, 2017 8:43 am

@JimmyO What kind of issue did you have - Slow merges or crashes?

For us the patch seems to really fix the slowness, even when an active full is running at the same time! For the first time, the target storage shows 100 % utilisation on all drives - in the past the backend did not write much and REFS just hang there at very low write speed
mkretzer
Expert
 
Posts: 338
Liked: 74 times
Joined: Thu Dec 17, 2015 7:17 am

Re: REFS 4k horror story

Veeam Logoby JimmyO » Fri Jul 21, 2017 9:25 am

No crashes - only slow merges an unresponsive disks (for about 30 sec at the time).
I can see some difference in merge time. I estimate 25% increase in performance. What used to take 20 hours now takes 15 (at least after the first run).
Still - when I started to use ReFS (first week) it took 1 hour...
JimmyO
Enthusiast
 
Posts: 37
Liked: 2 times
Joined: Sun Apr 27, 2014 8:19 pm

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 58 guests