REFS issues (server lockups, high CPU, high RAM)

[bryand82487]

KB4057142 (https://support.microsoft.com/en-us/help/4057142) now available from Windows Update and Microsoft Catalog (but not it seems on WSUS) includes Refs.sys 10.0.14393.2035 dated 11 January 2018.
The notes say "Addresses synchronization issue where backing up large Resilient File System (ReFS) volumes may lead to errors 0xc2 and 7E." There is no word on setting Registry keys.

Is this what everyone is waiting for?

I don't think this is it but I'm going to give it a try anyways. With Veeam everyone is "backing up to an REFS volume" not "backing up an REFS volume", so I don't think it will be the fix unless there verbiage is wrong.

[suprnova]

I don't think this is it but I'm going to give it a try anyways. With Veeam everyone is "backing up to an REFS volume" not "backing up an REFS volume", so I don't think it will be the fix unless there verbiage is wrong.

Didn't change anything for me...synthetic full for a 400GB full backup only got to 58% after 24 hours and when I looked on disk the file size was 32MB. Had to cancel it so I'd stop missing backups. Hope you have better luck!

[soehl]

It seems there is no stable solution for the problem, the MS Premier Support don´t answer back for about a week.
I´m now ready for disable Block Clone......

I found this registry key:

Code: Select all

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
RefsVirtualSyntheticDisabled
Type: REG_DWORD (0 False, 1 True)
Default value: 0
Description: By default, Veeam Backup & Replication uses Fast Clone for all backup repositories that meet the specified requirements. You can disable this option with this value.

But where I have to set this key? On the managment servers or on the repositories servers?

Thanks!

[tsightler]

My suspicion is that the build number isn't enough to determine if a given file contains the fixes. The build provided by MS Premier Support is likely a special build, with additional fixes that not have made their way into the mainline builds, even if they have a newer number. Note that this happens at Veeam as well, for example, if you get a hotfix for 9.5U2 that solves a specific problem, but then 9.5U3 comes out but that specific hotfix wasn't integrated into the main build. In that case the newer files may still have a higher version number, but not contain the fixes you need.

[Mike Resseler]

Folks on this very long thread

If you read the digest of Anton, you will most likely look forward to the CU of February. (So am I ). Let's hope that there is no last minute issue and that all the issues are solved.

Cheers
Mike

[JimmyO]

Yes - let´s hope.. I migrated back to NTFS a while ago (took 2 weeks), but if the Feb CU is heaven sent as thay say, I´d be prepared to give ReFS another try.

[kubimike]

Longest running uptime so far. Was never this stable before.
New '10.0.14393.1934 ' FTW

Nothing in my backup schedule has changed, still doing daily synthetics linking 6TB vbks, mind you it can do 4 synthetic fulls at once at night!

[Cicadymn]

I spoke to the ReFS team via Email and they said the following:

The fix will be officially released in the optional update on 2/20, and in the following patch Tuesday (3/13).

I've talked my management into holding off going back to NTFS until we give this new one a test, so I'm hoping that this really is the one!

[Mike Resseler]

Nice MikeO!

@Sam: Thanks for giving us that extra information. And glad that you could convince your management. Obviously, my fingers are crossed!

[vishal.chotoe]

KB4057142 (https://support.microsoft.com/en-us/help/4057142) now available from Windows Update and Microsoft Catalog (but not it seems on WSUS) includes Refs.sys 10.0.14393.2035 dated 11 January 2018.
The notes say "Addresses synchronization issue where backing up large Resilient File System (ReFS) volumes may lead to errors 0xc2 and 7E." There is no word on setting Registry keys.

Is this what everyone is waiting for?

I don't think this is it but I'm going to give it a try anyways. With Veeam everyone is "backing up to an REFS volume" not "backing up an REFS volume", so I don't think it will be the fix unless there verbiage is wrong.

This update didn't change a thing.... I guess we have to wait for the update coming in march as read above.

[Bacon]

Hey Guys, do you have this problem with virtual servers or physical servers which are running ReFS on Server2016.
I have this only with virtual machines running on vmware. Physical servers seems not affected.

[dellock6]

Longest running uptime so far. Was never this stable before.
New '10.0.14393.1934 ' FTW

Nothing in my backup schedule has changed, still doing daily synthetics linking 6TB vbks, mind you it can do 4 synthetic fulls at once at night!

Happy to hear that! Could you give us more information about memory consumption? For what I understood, it should be lower with the "stable" driver.
Thanks,
Luca

[bonzovt]

Hey Guys, do you have this problem with virtual servers or physical servers which are running ReFS on Server2016.
I have this only with virtual machines running on vmware. Physical servers seems not affected.

Good question. I have a customer looking to deploy new beefy physical servers, and we are talking about the possibility installing Win 2k16 on the OS and installing Veeam B&R and the repository directly, versus installing ESXi, creating a virtual B&R server and a virtual repository server so they can manage directly from their vCenter. I'm wondering if the ReFS direct attached via local Windows OS versus disk attached to a Win VM either through VMFS or RDM play any role in how ReFS performs? Has anyone out there had any experience doing this through a VM?

[nedcompbeheer]

Hey Guys, do you have this problem with virtual servers or physical servers which are running ReFS on Server2016.
I have this only with virtual machines running on vmware. Physical servers seems not affected.

Indeed a good question, I'm also on the point of configuration a physical server for Veeam back-up (40TB and 128GB RAM) and in doubt to choose between NTFS and REFS. Mainly because when chosen voor REFS the migration path to NTFS is as good as impossible because i don't have another server onsite to migratie the Veeam back-ups to.

[Ctek]

Thanks for the heads-up on KB4057142, I will deploy this update on one of our problematic repository servers which has quite a bit of volumes and problematic RAM consumption and lockups and will report back the results later this week.

[bryand82487]

Good question. I have a customer looking to deploy new beefy physical servers, and we are talking about the possibility installing Win 2k16 on the OS and installing Veeam B&R and the repository directly, versus installing ESXi, creating a virtual B&R server and a virtual repository server so they can manage directly from their vCenter. I'm wondering if the ReFS direct attached via local Windows OS versus disk attached to a Win VM either through VMFS or RDM play any role in how ReFS performs? Has anyone out there had any experience doing this through a VM?

We have some of these issues and are using all virtual. Our REFS backup repository is a 2016 VM with a 36 TB repository made up of four 9 TB luns attached to the VM using RDM's. I wouldn't go physical and can't think of any reason you would want to if the ESXi licensing costs are not an issue. A virtual environment is much easier to monitor and manage.

[kubimike]

Happy to hear that! Could you give us more information about memory consumption? For what I understood, it should be lower with the "stable" driver.
Thanks,
Luca

I'll run RAMMAP on tonights backup and post.

[Ctek]

Thanks for the heads-up on KB4057142, I will deploy this update on one of our problematic repository servers which has quite a bit of volumes and problematic RAM consumption and lockups and will report back the results later this week.

Update: KB4057142 Does not seem to address any of the ReFS issues that we are suffering from. I've tested it on 3 servers with different workloads, no change in behavior. Waiting for the Feb. Cumulative update

[jayscarff]

Tried to get support through MS - this was the reply...

"Hello Jason,
Thank you for the response.

We do not have a TAT from the internal team for the release of the hotfix yet.

I will contact you between 2 PM – 4 PM EST once I am in the office tomorrow to discuss regarding it in detail.

With Best Regards,
Siddharth Tagra "

[firewall68]

Case-ID: #02599159

Hi together,
same here. The update did not fix the REFS issue.
After reconfigure the whole backup (23 TB) from my synology smb share to a iscsi connteced target (formated 2016 ReFS, Targed Size 50 TB), it looks like, after booting the server,
i`ve lost the target because it^s only a RAW Disk without any access.....

the interesting thing is that the gui says RAW but with powershell (get-disk) everything is alright.

Number Friendly Name Serial Number HealthStatus OperationalStatus Total Size Partition
Style
------ ------------- ------------- ------------ ----------------- ---------- ----------
0 HP LOGICAL... 0014380225BD040 Healthy Online 136.7 GB MBR
1 HP LOGICAL... 0014380225BD040 Healthy Online 136.7 GB GPT
2 HP LOGICAL... PACCTID11190KMH Healthy Online 12.55 TB GPT
3 HP LOGICAL... PACCTID11190KMH Healthy Online 10.64 TB GPT
4 SYNOLOGY i... 879b6689-07f7-4121-a7a5-292ba... Healthy Online 49.08 TB GPT

[adapterer]

if you have the beta driver, run "Bcdedit.exe -set TESTSIGNING OFF" and partition will not be raw.

[firewall68]

if you have the beta driver, run "Bcdedit.exe -set TESTSIGNING OFF" and partition will not be raw.

Is there any source were you can get the beta driver?

found this article by the way (maybe helpful)
http://original-network.com/microsoft-r ... ed-issues/

[Mildur]

We are a Cloud Provider for our customers.
Implementing the registry key "refsVirtualSyntheticDisabled=1" on the backup Repo is not working.
Server does not let me logging in
Do we have to implement this key on the Tenant Site?

I hope, the CU 2018-02 is coming soon with the new refs driver.

[crollins]

Am I missing something!? I don't any mention of REFS fixes in the February CU. Is this going to be another month?
https://support.microsoft.com/eu-es/hel ... -kb4074590

[SteelContainer]

There is a CU 2018-02 out now but it doesn't specifically mention ReFS in the improvments/Fixes.
https://support.microsoft.com/en-ph/hel ... -kb4074590

Will need to test it to see if the driver is included.

[SteelContainer]

Seems to still be Refs.sys 10.0.14393.2035 11-Jan-18

[jayscarff]

Thanks Cullan, was hoping a 1gig CU would change something!

[firewall68]

Hello together,
we have open a case with microsoft, and they give us the feedback that the CU2018-02 will include the REFS Driver. just right now (Austria in the morning 7am) i try to install it and after reboot hopefully the 50 TB repo will show up and not RAW.

[Mike Resseler]

Keep us informed gentleman. I am not in the possibility to install updates on a workload the next couple of days but I am eager to know if it is included. From the list of files, it does indeed seem that ReFS has not received an update compared to the January CU.

[firewall68]

Ok i`ve did it, but .......
According MS the update will fix the issue but you have to format the disk again with REFS. therfor all data on the RAW Device are lost.
the only way is to buy a software which can recover REFS.
what i now will do is. Format the disk with REFS again, restart the server (2 - 3 Time) and check the stability of the REFS Disk (ISCSI Target)
I have not a big issue with my Data (23TB) because i have Backup Copy Jobs to another Target.
Keep you informed