Comprehensive data protection for all workloads
Locked
bryand82487
Influencer
Posts: 14
Liked: 2 times
Joined: Dec 13, 2017 3:31 pm
Full Name: Bryan Dennison
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by bryand82487 »

Neil Flanagan wrote:KB4057142 (https://support.microsoft.com/en-us/help/4057142) now available from Windows Update and Microsoft Catalog (but not it seems on WSUS) includes Refs.sys 10.0.14393.2035 dated 11 January 2018.
The notes say "Addresses synchronization issue where backing up large Resilient File System (ReFS) volumes may lead to errors 0xc2 and 7E." There is no word on setting Registry keys.

Is this what everyone is waiting for?
I don't think this is it but I'm going to give it a try anyways. With Veeam everyone is "backing up to an REFS volume" not "backing up an REFS volume", so I don't think it will be the fix unless there verbiage is wrong.
suprnova
Enthusiast
Posts: 38
Liked: never
Joined: Apr 08, 2016 5:15 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by suprnova »

bryand82487 wrote:I don't think this is it but I'm going to give it a try anyways. With Veeam everyone is "backing up to an REFS volume" not "backing up an REFS volume", so I don't think it will be the fix unless there verbiage is wrong.
Didn't change anything for me...synthetic full for a 400GB full backup only got to 58% after 24 hours and when I looked on disk the file size was 32MB. Had to cancel it so I'd stop missing backups. Hope you have better luck!
soehl
Enthusiast
Posts: 57
Liked: 8 times
Joined: May 09, 2011 12:43 pm
Full Name: Sebastian
Location: Germany
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by soehl »

It seems there is no stable solution for the problem, the MS Premier Support don´t answer back for about a week. :(
I´m now ready for disable Block Clone......

I found this registry key:

Code: Select all

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
RefsVirtualSyntheticDisabled
Type: REG_DWORD (0 False, 1 True)
Default value: 0
Description: By default, Veeam Backup & Replication uses Fast Clone for all backup repositories that meet the specified requirements. You can disable this option with this value.
But where I have to set this key? On the managment servers or on the repositories servers?

Thanks!
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by tsightler » 1 person likes this post

My suspicion is that the build number isn't enough to determine if a given file contains the fixes. The build provided by MS Premier Support is likely a special build, with additional fixes that not have made their way into the mainline builds, even if they have a newer number. Note that this happens at Veeam as well, for example, if you get a hotfix for 9.5U2 that solves a specific problem, but then 9.5U3 comes out but that specific hotfix wasn't integrated into the main build. In that case the newer files may still have a higher version number, but not contain the fixes you need.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mike Resseler »

Folks on this very long thread :-)

If you read the digest of Anton, you will most likely look forward to the CU of February. (So am I :-)). Let's hope that there is no last minute issue and that all the issues are solved.

Cheers
Mike
JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by JimmyO »

Yes - let´s hope..;) I migrated back to NTFS a while ago (took 2 weeks), but if the Feb CU is heaven sent as thay say, I´d be prepared to give ReFS another try.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike » 1 person likes this post

Longest running uptime so far. Was never this stable before.
New '10.0.14393.1934 ' FTW

Nothing in my backup schedule has changed, still doing daily synthetics linking 6TB vbks, mind you it can do 4 synthetic fulls at once at night!

Image
Cicadymn
Enthusiast
Posts: 26
Liked: 12 times
Joined: Jan 30, 2017 7:42 pm
Full Name: Sam
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Cicadymn » 4 people like this post

I spoke to the ReFS team via Email and they said the following:
The fix will be officially released in the optional update on 2/20, and in the following patch Tuesday (3/13).
I've talked my management into holding off going back to NTFS until we give this new one a test, so I'm hoping that this really is the one!
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mike Resseler »

Nice MikeO!

@Sam: Thanks for giving us that extra information. And glad that you could convince your management. Obviously, my fingers are crossed!
vishal.chotoe
Lurker
Posts: 2
Liked: never
Joined: Dec 04, 2017 3:02 pm
Full Name: Vishal Chotoe
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by vishal.chotoe »

Neil Flanagan wrote:KB4057142 (https://support.microsoft.com/en-us/help/4057142) now available from Windows Update and Microsoft Catalog (but not it seems on WSUS) includes Refs.sys 10.0.14393.2035 dated 11 January 2018.
The notes say "Addresses synchronization issue where backing up large Resilient File System (ReFS) volumes may lead to errors 0xc2 and 7E." There is no word on setting Registry keys.

Is this what everyone is waiting for?

I don't think this is it but I'm going to give it a try anyways. With Veeam everyone is "backing up to an REFS volume" not "backing up an REFS volume", so I don't think it will be the fix unless there verbiage is wrong.
This update didn't change a thing.... I guess we have to wait for the update coming in march as read above.
Bacon
Novice
Posts: 3
Liked: 1 time
Joined: Jan 26, 2018 9:35 am
Full Name: Alexander
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Bacon »

Hey Guys, do you have this problem with virtual servers or physical servers which are running ReFS on Server2016.
I have this only with virtual machines running on vmware. Physical servers seems not affected.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by dellock6 »

kubimike wrote:Longest running uptime so far. Was never this stable before.
New '10.0.14393.1934 ' FTW

Nothing in my backup schedule has changed, still doing daily synthetics linking 6TB vbks, mind you it can do 4 synthetic fulls at once at night!
Happy to hear that! Could you give us more information about memory consumption? For what I understood, it should be lower with the "stable" driver.
Thanks,
Luca
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
bonzovt
Enthusiast
Posts: 42
Liked: 1 time
Joined: Sep 09, 2016 6:15 pm
Full Name: Adam Fisher
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by bonzovt »

Bacon wrote:Hey Guys, do you have this problem with virtual servers or physical servers which are running ReFS on Server2016.
I have this only with virtual machines running on vmware. Physical servers seems not affected.
Good question. I have a customer looking to deploy new beefy physical servers, and we are talking about the possibility installing Win 2k16 on the OS and installing Veeam B&R and the repository directly, versus installing ESXi, creating a virtual B&R server and a virtual repository server so they can manage directly from their vCenter. I'm wondering if the ReFS direct attached via local Windows OS versus disk attached to a Win VM either through VMFS or RDM play any role in how ReFS performs? Has anyone out there had any experience doing this through a VM?
nedcompbeheer
Service Provider
Posts: 1
Liked: never
Joined: Jan 29, 2018 8:47 am
Full Name: Ralf Brand
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by nedcompbeheer »

Bacon wrote:Hey Guys, do you have this problem with virtual servers or physical servers which are running ReFS on Server2016.
I have this only with virtual machines running on vmware. Physical servers seems not affected.
Indeed a good question, I'm also on the point of configuration a physical server for Veeam back-up (40TB and 128GB RAM) and in doubt to choose between NTFS and REFS. Mainly because when chosen voor REFS the migration path to NTFS is as good as impossible because i don't have another server onsite to migratie the Veeam back-ups to.
Ctek
Service Provider
Posts: 83
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Ctek »

Thanks for the heads-up on KB4057142, I will deploy this update on one of our problematic repository servers which has quite a bit of volumes and problematic RAM consumption and lockups and will report back the results later this week.
VMCE
bryand82487
Influencer
Posts: 14
Liked: 2 times
Joined: Dec 13, 2017 3:31 pm
Full Name: Bryan Dennison
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by bryand82487 »

afishernwn wrote:Good question. I have a customer looking to deploy new beefy physical servers, and we are talking about the possibility installing Win 2k16 on the OS and installing Veeam B&R and the repository directly, versus installing ESXi, creating a virtual B&R server and a virtual repository server so they can manage directly from their vCenter. I'm wondering if the ReFS direct attached via local Windows OS versus disk attached to a Win VM either through VMFS or RDM play any role in how ReFS performs? Has anyone out there had any experience doing this through a VM?
We have some of these issues and are using all virtual. Our REFS backup repository is a 2016 VM with a 36 TB repository made up of four 9 TB luns attached to the VM using RDM's. I wouldn't go physical and can't think of any reason you would want to if the ESXi licensing costs are not an issue. A virtual environment is much easier to monitor and manage.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike »

dellock6 wrote:
Happy to hear that! Could you give us more information about memory consumption? For what I understood, it should be lower with the "stable" driver.
Thanks,
Luca
I'll run RAMMAP on tonights backup and post.
Ctek
Service Provider
Posts: 83
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Ctek » 1 person likes this post

Ctek wrote:Thanks for the heads-up on KB4057142, I will deploy this update on one of our problematic repository servers which has quite a bit of volumes and problematic RAM consumption and lockups and will report back the results later this week.
Update: KB4057142 Does not seem to address any of the ReFS issues that we are suffering from. I've tested it on 3 servers with different workloads, no change in behavior. Waiting for the Feb. Cumulative update :?
VMCE
jayscarff
Service Provider
Posts: 114
Liked: 12 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jayscarff »

Tried to get support through MS - this was the reply...

"Hello Jason,
Thank you for the response.

We do not have a TAT from the internal team for the release of the hotfix yet.

I will contact you between 2 PM – 4 PM EST once I am in the office tomorrow to discuss regarding it in detail.

With Best Regards,
Siddharth Tagra "
Jason
VMCE
firewall68
Influencer
Posts: 12
Liked: never
Joined: Jan 04, 2017 1:16 pm
Full Name: Mike
Location: Austria
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by firewall68 »

Case-ID: #02599159

Hi together,
same here. The update did not fix the REFS issue.
After reconfigure the whole backup (23 TB) from my synology smb share to a iscsi connteced target (formated 2016 ReFS, Targed Size 50 TB), it looks like, after booting the server,
i`ve lost the target because it^s only a RAW Disk without any access..... :shock: :(

the interesting thing is that the gui says RAW but with powershell (get-disk) everything is alright.

Number Friendly Name Serial Number HealthStatus OperationalStatus Total Size Partition
Style
------ ------------- ------------- ------------ ----------------- ---------- ----------
0 HP LOGICAL... 0014380225BD040 Healthy Online 136.7 GB MBR
1 HP LOGICAL... 0014380225BD040 Healthy Online 136.7 GB GPT
2 HP LOGICAL... PACCTID11190KMH Healthy Online 12.55 TB GPT
3 HP LOGICAL... PACCTID11190KMH Healthy Online 10.64 TB GPT
4 SYNOLOGY i... 879b6689-07f7-4121-a7a5-292ba... Healthy Online 49.08 TB GPT
adapterer
Expert
Posts: 227
Liked: 46 times
Joined: Oct 12, 2015 11:24 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by adapterer » 1 person likes this post

if you have the beta driver, run "Bcdedit.exe -set TESTSIGNING OFF" and partition will not be raw.
firewall68
Influencer
Posts: 12
Liked: never
Joined: Jan 04, 2017 1:16 pm
Full Name: Mike
Location: Austria
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by firewall68 »

adapterer wrote:if you have the beta driver, run "Bcdedit.exe -set TESTSIGNING OFF" and partition will not be raw.
Is there any source were you can get the beta driver?

found this article by the way (maybe helpful)
http://original-network.com/microsoft-r ... ed-issues/
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mildur »

We are a Cloud Provider for our customers.
Implementing the registry key "refsVirtualSyntheticDisabled=1" on the backup Repo is not working.
Server does not let me logging in :-)
Do we have to implement this key on the Tenant Site?

I hope, the CU 2018-02 is coming soon with the new refs driver.
Product Management Analyst @ Veeam Software
crollins
Service Provider
Posts: 1
Liked: never
Joined: Dec 13, 2017 3:28 pm
Full Name: Chance Rollins
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by crollins »

Am I missing something!? I don't any mention of REFS fixes in the February CU. Is this going to be another month?
https://support.microsoft.com/eu-es/hel ... -kb4074590
SteelContainer
Service Provider
Posts: 146
Liked: 21 times
Joined: May 21, 2014 8:47 am
Location: New Zealand
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by SteelContainer »

There is a CU 2018-02 out now but it doesn't specifically mention ReFS in the improvments/Fixes.
https://support.microsoft.com/en-ph/hel ... -kb4074590

Will need to test it to see if the driver is included.
SteelContainer
Service Provider
Posts: 146
Liked: 21 times
Joined: May 21, 2014 8:47 am
Location: New Zealand
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by SteelContainer »

Seems to still be Refs.sys 10.0.14393.2035 11-Jan-18 :(
jayscarff
Service Provider
Posts: 114
Liked: 12 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jayscarff »

Thanks Cullan, was hoping a 1gig CU would change something!
Jason
VMCE
firewall68
Influencer
Posts: 12
Liked: never
Joined: Jan 04, 2017 1:16 pm
Full Name: Mike
Location: Austria
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by firewall68 »

Hello together,
we have open a case with microsoft, and they give us the feedback that the CU2018-02 will include the REFS Driver. just right now (Austria in the morning 7am) i try to install it and after reboot hopefully the 50 TB repo will show up and not RAW.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mike Resseler »

Keep us informed gentleman. I am not in the possibility to install updates on a workload the next couple of days but I am eager to know if it is included. From the list of files, it does indeed seem that ReFS has not received an update compared to the January CU. :-(
firewall68
Influencer
Posts: 12
Liked: never
Joined: Jan 04, 2017 1:16 pm
Full Name: Mike
Location: Austria
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by firewall68 »

Ok i`ve did it, but .......
According MS the update will fix the issue but you have to format the disk again with REFS. therfor all data on the RAW Device are lost.
the only way is to buy a software which can recover REFS.
what i now will do is. Format the disk with REFS again, restart the server (2 - 3 Time) and check the stability of the REFS Disk (ISCSI Target)
I have not a big issue with my Data (23TB) because i have Backup Copy Jobs to another Target.
Keep you informed
Locked

Who is online

Users browsing this forum: Google [Bot] and 165 guests