REFS 4k horror story

Availability for the Always-On Enterprise

Re: REFS 4k horror story

Veeam Logoby kubimike » Tue Jun 20, 2017 3:48 pm

@nmdange, cool ok so you're running 14393.1198 then ..

@thomas.raabo You're running the experimental fix from msft ??"14393.1100" ?
kubimike
Expert
 
Posts: 243
Liked: 23 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby Cicadymn » Tue Jun 20, 2017 6:29 pm

I've got 14393.1198 on my main backup VM and 14393.1100 (experimental) on my backup copy vm.

I haven't re-enabled synthetic fulls on my main backup VM yet as I haven't heard whether or not it includes the fixes from the experimental driver. On one note, the registry keys added when I set up the experimental fix are not present in the current 1198 driver.
Cicadymn
Influencer
 
Posts: 20
Liked: 5 times
Joined: Mon Jan 30, 2017 7:42 pm
Full Name: Sam

Re: REFS 4k horror story

Veeam Logoby andersgustaf » Wed Jun 21, 2017 6:53 am

Hi,

I have the same problem, but Im not using Veeam.. Im using DPM :)

I have also got the refs.sys file from a MS technician, but my issue is when Im replacing the existing refs.sys the disk is reported RAW and can not be read.

Any special order/rutine to change the file that my technician has not mentioned?


Best regards,
Anders
andersgustaf
Novice
 
Posts: 8
Liked: never
Joined: Thu Oct 15, 2015 11:08 am
Full Name: Anders

Re: REFS 4k horror story

Veeam Logoby kubimike » Wed Jun 21, 2017 11:57 am

Welcome to Veeam land. What happens when you put the refs.sys file back you had originally ? Don't quote me on this but when windows 2016 was released and an refs volume was formatted it wasn't the same version we have now. I could be totally wrong here but I've read that somewhere. Maybe that's the issue ? Version 3.0 vs 3.1
kubimike
Expert
 
Posts: 243
Liked: 23 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby lepphce1 » Wed Jun 21, 2017 2:18 pm

An update from me. I installed the beta driver provided by Veeam 14393.1100 with the associated registry entries. I continue to have problems with BSOD/reboots. In my case it was necessary to delete a corrupt backup chain (caused by this very issue), which caused the server to go into a series of reboots. This is the same behavior as the official production driver. Hopefully there will be an updated driver soon.
lepphce1
Enthusiast
 
Posts: 30
Liked: 2 times
Joined: Tue Jun 28, 2016 4:40 pm

Re: REFS 4k horror story

Veeam Logoby thomas.raabo » Wed Jun 21, 2017 8:59 pm

kubimike wrote:@nmdange, cool ok so you're running 14393.1198 then ..

@thomas.raabo You're running the experimental fix from msft ??"14393.1100" ?


correct.
thomas.raabo
Service Provider
 
Posts: 14
Liked: 5 times
Joined: Mon Oct 31, 2016 6:27 pm
Location: infrastructure guy
Full Name: Thomas Raabo

Re: REFS 4k horror story

Veeam Logoby thomas.raabo » Wed Jun 21, 2017 9:01 pm

andersgustaf wrote:Hi,

I have the same problem, but Im not using Veeam.. Im using DPM :)

I have also got the refs.sys file from a MS technician, but my issue is when Im replacing the existing refs.sys the disk is reported RAW and can not be read.

Any special order/rutine to change the file that my technician has not mentioned?


Best regards,
Anders



Did you put your system into test signing mode?
thomas.raabo
Service Provider
 
Posts: 14
Liked: 5 times
Joined: Mon Oct 31, 2016 6:27 pm
Location: infrastructure guy
Full Name: Thomas Raabo

Re: REFS 4k horror story

Veeam Logoby andersgustaf » Thu Jun 22, 2017 6:40 am

thomas.raabo wrote:Hi,

I have the same problem, but Im not using Veeam.. Im using DPM :)

I have also got the refs.sys file from a MS technician, but my issue is when Im replacing the existing refs.sys the disk is reported RAW and can not be read.

Any special order/rutine to change the file that my technician has not mentioned?


Best regards,
Anders


Did you put your system into test signing mode?


No, I got no instructions of doing so.
But I tried that on a test VM that I created and after a reboot it felt like the server became 100 times faster and I was able to access the disk.
I have asked the technician for a new file but I guess that wont help since the file isnt signed. Also, if you check properties on the refs.sys you will see a "certificate error" and that is since it isnt signed (duh).

Thanks for the help!

by the way, I opened my case with MS late April and it took a while to get here... :)

//Anders
andersgustaf
Novice
 
Posts: 8
Liked: never
Joined: Thu Oct 15, 2015 11:08 am
Full Name: Anders

Re: REFS 4k horror story

Veeam Logoby thomas.raabo » Thu Jun 22, 2017 9:26 am

In order to apply the hotfix please use “Microsoft” password to unzip the folder.

Afterwards please rename the original "refs.sys" in C:\Windows\System32\drivers, for example to refs.sys_original, copy the contents of the archive into the same folder.

After copying the new refs.sys please execute this command:
bcdedit /set testsigning

Then please create the following registry keys on the server in question:

- RefsDisableCachedPins (DWORD) = 1
in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem

- RefsProcessedDeleteQueueEntryCountThreshold (DWORD) = 2048 in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
First of all let's give it a try with 2048, then we will change it to 1024 and 512 after if needed.

Also, let's increase the following timeout:

TimeOutValue (DWORD)
in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Disk
Please set it to 120 (decimal value).

Please note that these changes will require the server restart.
thomas.raabo
Service Provider
 
Posts: 14
Liked: 5 times
Joined: Mon Oct 31, 2016 6:27 pm
Location: infrastructure guy
Full Name: Thomas Raabo

Re: REFS 4k horror story

Veeam Logoby andy51585 » Thu Jun 22, 2017 12:08 pm

People who are using the new refs driver released in the latest cumulative update. Are you still utilizing the registry tweaks originally proposed?

The version of my installed refs.sys is 10.0.14393.1198 and still having issues with near daily server lockups with 4k. Unfortunately I don't have enough space at this time to move things around so I have to make this work somehow.

I'm currently using
RefsEnableInlineTrim =1
RefsEnableLargeWorkingSetTrim = 1
RefsNumberOfChunksToTrim = 128 (decimal)

On the Veeam Server, I have RefsVirtualSyntheticDisabled set to 1. I don't even care about the block clone stuff at this time. I just need backups that reliably run for a couple more months until one of our old NAS boxes has enough space on it to move stuff around and reformat w 64k.

I've even gone to running weekly active fulls instead of synthetics as a test, and still have the issues.

The server hosting the iscsi repository has 40GB of RAM and 4 vcpu.

Any advice would be great.
andy51585
Novice
 
Posts: 8
Liked: never
Joined: Mon Jul 28, 2014 9:13 pm
Full Name: Andrew

Re: REFS 4k horror story

Veeam Logoby tsightler » Thu Jun 22, 2017 1:43 pm

andy51585 wrote:The version of my installed refs.sys is 10.0.14393.1198 and still having issues with near daily server lockups with 4k. Unfortunately I don't have enough space at this time to move things around so I have to make this work somehow.


I don't think I've seen anyone in this thread have success with 4K, regardless of hotfixes or anything else. The thread title is probably a misnomer at this point because most of the discussions for the last month or two has been around people that were still having various issues even after moving to 64K. I'm sure at least some of the issues are the same, but the problem with 4K is that it needs a lot more memory to survive.
tsightler
Veeam Software
 
Posts: 4839
Liked: 1787 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: REFS 4k horror story

Veeam Logoby lepphce1 » Thu Jun 22, 2017 2:21 pm

I'm still on 4K and memory has never been an issue. Unfortunately I don't have the storage to reformat to 64K. But my server does crash for days, with low resource utilization, when I do a mass delete. Seems there are a variety of symptoms and I wouldn't be surprised if there are multiple ReFS issues that are being addressed here.

It's really cool technology and I hope they work it out at some point, but right now I'm actively shopping for disk shelves so I can end this nightmare.
lepphce1
Enthusiast
 
Posts: 30
Liked: 2 times
Joined: Tue Jun 28, 2016 4:40 pm

Re: REFS 4k horror story

Veeam Logoby kubimike » Thu Jun 22, 2017 3:39 pm

@lepphche1 hello I've been doing backup copies to another source in preparation to use the new refs.sys experimental driver. I've had great success with a USB 3.0 Western Digital DUO 16 TB. See screengrab below, those are real results!
Image

Also I ordered a QNAP 4bat NAS. Its the TVS-471 w/ 4 10TB drives. Going to benchmark that guy too .

Depending on how much data you have those are great cheap solutions to tide you over. Hope this info helps you out! :D
kubimike
Expert
 
Posts: 243
Liked: 23 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby lepphce1 » Thu Jun 22, 2017 3:52 pm

@kubimike, thanks for the suggestions. I'm struggling with my backup copies and GFS retention, so those solutions are still a bit small for my situation... I'm looking at a Dell MD1200 with 12 10TB disks-- it should keep me going for a few years, and after discounts I'm hoping to pay less than $10,000.
lepphce1
Enthusiast
 
Posts: 30
Liked: 2 times
Joined: Tue Jun 28, 2016 4:40 pm

Re: REFS 4k horror story

Veeam Logoby kubimike » Thu Jun 22, 2017 5:15 pm

I see, yeah I have a lot more data but Im only doing backup copies of critical data. If my veeam box blows up again at least I have my SQL data and exchange. It just so happens I can fit those two VMs on the 16TB box. And I just ordered a new badass HP LTO7 MSL2024 ! :twisted:
kubimike
Expert
 
Posts: 243
Liked: 23 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google Feedfetcher, nmdange, PTide, Shestakov and 98 guests