Comprehensive data protection for all workloads
Post Reply
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt »

Can someone running 2019 build 1903 post the version of the refs.sys file?
Veeam Certified Engineer
DonZoomik
Service Provider
Posts: 368
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: Windows 2019, large REFS and deletes

Post by DonZoomik »

According to patch notes it hasn't been patched so it's probaby 10.0.18362.0
I don't have an 1903 system right now to check but it's an educated guess from patch info's file list CSV.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

Correct, 10.0.18362.0. Thats the version working perfectly for us. It is working so good i even understand why they never patched anything in that version.
WorkingHardInIt
Veeam Vanguard
Posts: 39
Liked: 11 times
Joined: Feb 14, 2014 1:27 pm
Full Name: Didier Van Hoye
Contact:

Re: Windows 2019, large REFS and deletes

Post by WorkingHardInIt »

Any one tested 10.0.17763.771 yet on Windows Server 2019 1809?
dasfliege
Service Provider
Posts: 238
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by dasfliege »

@WorkingHardInIt
Not yet. When and how has it been released?

@mkretzer / @gostev
Would you may share the Microsoft case number, where you already had contact with the ReFS Devs? MS is asking me for it.
GACcc
Novice
Posts: 8
Liked: 1 time
Joined: Jun 06, 2018 1:06 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by GACcc »

WorkingHardInIt wrote: Oct 11, 2019 7:23 am Any one tested 10.0.17763.771 yet on Windows Server 2019 1809?
Yes I have... and its not working for us... its horrible.
The first backups are "pretty" fust but then it takes hours and hours
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt »

I did some testing as well. We had two repository servers with Server 2019- 1809. We do copy jobs from the first to the second one. Only on the second one, the one doing GFS stuff, things went bad with merging times. Way too long. Had to restart everything to get it going again each time.
I read through this thread, wiped some tears when i concluded MS created a seemingly stable refs.sys driver a while back, but not fixing this for 1809 customers, and we decided to rip out a refs.sys file from a Windows 10 1903 patched desktop and place it on the second repository only. Refs.sys version coming from the windows 10 1903 machine was 10.0.18362.1 , modified date 19-3-2019.
I have to see how it goes in the coming days, will report back.
Veeam Certified Engineer
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

@dasfliege I could give you one of our case numbers but i do not know if that would help as we really had direct contact with the devs - without a ticket system inbetween...
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt »

JaySt wrote: Oct 11, 2019 8:50 pm I did some testing as well. We had two repository servers with Server 2019- 1809. We do copy jobs from the first to the second one. Only on the second one, the one doing GFS stuff, things went bad with merging times. Way too long. Had to restart everything to get it going again each time.
I read through this thread, wiped some tears when i concluded MS created a seemingly stable refs.sys driver a while back, but not fixing this for 1809 customers, and we decided to rip out a refs.sys file from a Windows 10 1903 patched desktop and place it on the second repository only. Refs.sys version coming from the windows 10 1903 machine was 10.0.18362.1 , modified date 19-3-2019.
I have to see how it goes in the coming days, will report back.
unfortunately, this did not work. GFS restorepoint creation still running. job takes more than 80 hours now. Strange, i had good hopes replacing refs.sys would help. Either this is a different issue not related to ReFS.sys somehow or just replacing refs.sys is not the complete story to fix these kind of issues.
Veeam Certified Engineer
steggles
Novice
Posts: 5
Liked: never
Joined: Jul 10, 2019 11:52 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by steggles »

Hi All,

We have just built 2019 Veeam Repositories using the Long Term Servicing Branch of Windows Server 2019. We have been having a few issues with high CPU on ReFS jobs, eventually they cause the server to lock up, and all the other jobs wait on them. It tends to happen on Health Check and Compact operations on the jobs. But from the thread here it would seem to indicate I need to have disks with size greater than 100TB, is that right? As I only have a number of ~74TB drives as part of a bigger SOBR. I wouldn't have thought I would be impacted by this.

I have opened a ticket with Veeam. Should I upgrade to 1903? or is there likely another cause?

Regards,
Thomas Higgins
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt »

I also have disk sizes below or around 100TB and seeing problems.
you can't upgrade to 1903. It's a new install. it's all just such a sad story.
Veeam Certified Engineer
dasfliege
Service Provider
Posts: 238
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by dasfliege » 2 people like this post

Another week, another try:
After MS analysed the latest memory dumps i've collected, while our ReFS performance was almost down to zero, they came up with another possible solution. I've already applied it and so far it looks quite promising. Block Clone activities are now running since 1-2 hours and disk activity never dropped to 0 so far.

Would be nice if some other guys could test it and give a feedback. All you have to do is creating the following regkey and reboot the server:
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
REG_DWORD value: "RefsEnableLargeWorkingSetTrim"=1

There are some KB describing the issue for WS2016. But it seems like it has reappeared on WS2019:
https://support.microsoft.com/en-us/hel ... is-running
https://support.microsoft.com/en-us/hel ... windows-10
TheJourney
Influencer
Posts: 22
Liked: 4 times
Joined: Dec 10, 2009 8:44 pm
Full Name: Sam Journagan
Contact:

Re: Windows 2019, large REFS and deletes

Post by TheJourney » 2 people like this post

I can confirm REFS on windows 2019 is Junk.

Our Exchange server merges were stalling after 8 hours or so, many times not completing at all. The last successful merge on 2019 took over 17 hours. Rebooting the repo would help for a day or so then the issue would come right back.

I reinstalled with Windows Server 1903, kept the data volume (no ReFs reformat) attached it to the server with the same drive letter keeping all the backup files. Kept the same IP address, server name, etc. Opened up Veeam > Backup Infrastructure > MS Windows > Properties of the existing server object in Veeam. This installs the Transport Agent. Then ran through the properties dialog under backup Repos, this installs the other Veeam components needed for Repos. I was then able to run backup jobs without changing anything else.

Immediately retried a failed Exchange merge and it completed in 30 mins. The next exchange merge took less than 2 hours. All backup jobs are completing normally after the Repo migration. Not sure if this is a supported move, but working great so far. I hope it stays this way!!!

Cheers!
jayscarff
Service Provider
Posts: 114
Liked: 12 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: Windows 2019, large REFS and deletes

Post by jayscarff »

dasfliege wrote: Oct 21, 2019 11:32 am Another week, another try:
After MS analysed the latest memory dumps i've collected, while our ReFS performance was almost down to zero, they came up with another possible solution. I've already applied it and so far it looks quite promising. Block Clone activities are now running since 1-2 hours and disk activity never dropped to 0 so far.

Would be nice if some other guys could test it and give a feedback. All you have to do is creating the following regkey and reboot the server:
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
REG_DWORD value: "RefsEnableLargeWorkingSetTrim"=1

There are some KB describing the issue for WS2016. But it seems like it has reappeared on WS2019:
https://support.microsoft.com/en-us/hel ... is-running
https://support.microsoft.com/en-us/hel ... windows-10
Just seeing all clients useing our Cloud Repo (the refs one) slowing down completely. While NTFS cloud repo has no issues. Only thing that has changed are windows updates. Im applying this reg key now and will reboot.
Jason
VMCE
dasfliege
Service Provider
Posts: 238
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by dasfliege » 1 person likes this post

Update from my side:
Since i've applied the regkey two days ago, i have all our Copy Jobs (30 Jobs running continously) enabled again and i had no problems at all. Merges and GFS creations runs faster then they do on our 2016 repo and i had no locks at all. Seems like this really did the trick. Hope that it don't get worse again.

@jayscarff
Please report back your experience after you tried it.
nikpolini
Enthusiast
Posts: 33
Liked: 2 times
Joined: May 28, 2015 3:23 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by nikpolini »

Hi All.
Can some one on 1903 check to see if the RefsEnableLargeWorkingSetTrim entry exists by default and its value?

Cheers.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » 1 person likes this post

RefsEnableLargeWorkingSetTrim is not there in 1903 - why should it, the driver is so efficient now...
GACcc
Novice
Posts: 8
Liked: 1 time
Joined: Jun 06, 2018 1:06 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by GACcc »

dasfliege wrote: Oct 21, 2019 11:32 am Another week, another try:
After MS analysed the latest memory dumps i've collected, while our ReFS performance was almost down to zero, they came up with another possible solution. I've already applied it and so far it looks quite promising. Block Clone activities are now running since 1-2 hours and disk activity never dropped to 0 so far.

Would be nice if some other guys could test it and give a feedback. All you have to do is creating the following regkey and reboot the server:
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
REG_DWORD value: "RefsEnableLargeWorkingSetTrim"=1

There are some KB describing the issue for WS2016. But it seems like it has reappeared on WS2019:
https://support.microsoft.com/en-us/hel ... is-running
https://support.microsoft.com/en-us/hel ... windows-10
Just tried it right now. Set the reg key and rebooted. First backup was "fast" (not as back in times). As soon as I start multiple jobs simultanously its slow again... even on the graph it goes down to 0.0 KB/s for xxx minutes.

Server 2019 - 1809 - 17763.775
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt »

i also tried RefsEnableLargeWorkingSetTrim on a Server 2019 1809 system with problems. First copy jobs after the reboot seemed to be fast and completed. When i checked after the weekend, all jobs had problems once again. So it did not help in our case.
Veeam Certified Engineer
dasfliege
Service Provider
Posts: 238
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by dasfliege » 1 person likes this post

@GaCcc
@JaySt

Did you also applied the modification i mentioned earlier?
Check if Trim is enabled by firing the command: "fsutil behavior query DisableDeleteNotify"

If DisableDeleteNotifiy is disabled (0) for ReFS on the 2019 server, enable it with the following commands:
"fsutil behavior set DisableDeleteNotify 1"
"fsutil behavior set DisableDeleteNotify ReFS"
I only did these two modifications and i'm on refs.sys 10.0.17763.771. Our Backup are now running in parallel (30 Backup- and 30 Copy Jobs) without any problems since more then a week. Even faster then before on 2016. Sad to hear that it didn't worked out so far for you guys.
GACcc
Novice
Posts: 8
Liked: 1 time
Joined: Jun 06, 2018 1:06 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by GACcc »

@dasfliege

I did only the part below.
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
REG_DWORD value: "RefsEnableLargeWorkingSetTrim"=1

I'll test the one with the DeleteNotify right now. PS. there is a missing "1" in your example:

If DisableDeleteNotifiy is disabled (0) for ReFS on the 2019 server, enable it with the following commands:
"fsutil behavior set DisableDeleteNotify 1"
"fsutil behavior set DisableDeleteNotify ReFS 1" <=========
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt »

@dasfliege
same as GACcc, i only set the RefsEnableLargeWorkingSetTrim=1 value
i'll try the other two as well. will report back, might take a few days.
Veeam Certified Engineer
rhys.hammond
Veeam Software
Posts: 72
Liked: 15 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond »

Just applied the reg key (RefsEnableLargeWorkingSetTrim) and the 'fsutil behavior set DisableDeleteNotify ReFS 1'.
Will report back in a few days.
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt »

@rhis.hammond.
You did not use "fsutil behavior set DisableDeleteNotify 1" ? may i ask why? i had doubts about the need for that one, as i'm using a NTFS bootvolume on SSD on this server as well. But i'm not sure i'm making the right assumption.
Veeam Certified Engineer
rhys.hammond
Veeam Software
Posts: 72
Liked: 15 times
Joined: Apr 07, 2013 10:36 pm
Full Name: Rhys Hammond
Location: Brisbane , Australia
Contact:

Re: Windows 2019, large REFS and deletes

Post by rhys.hammond »

dasfliege wrote: Oct 04, 2019 1:28 pm If DisableDeleteNotifiy is disabled (0) for ReFS on the 2019 server, enable it with the following commands:
"fsutil behavior set DisableDeleteNotify 1"
"fsutil behavior set DisableDeleteNotify ReFS"
@JaySt,
Simply because DisableDeleteNotifiy was indeed disabled (0) for ReFS on this machine so I "enabled" it.
I didn't see the benefit in disabling delete notify for NTFS.

Also,our performance has fallen off a cliff, I'm still offloading 100's of TBs worth of VBKs but unfortunately, it seems the capacity tier offload process also triggers the poor ReFS issue.
Currently down from 700TB to 400TB, another 200Tb to offload (The telco stuffed around this customer, increasing their WAN pipes took months longer than anticipated, fortunately, they didn't have to reduce their retention due to benefits of ReFS)

I'm likley just going to wipe 1809 and install 1903, the customer doesn't want SAC but the alternative is downgrading to 2016 and then we'd have to wipe the ReFS volume.
Veeam Certified Architect | Author of http://rhyshammond.com | Veeam Vanguard | vExpert
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt »

Tested for some days. Unfortunately no improvements after changing DisableDeleteNotify settings to enabled (1)
Veeam Certified Engineer
GACcc
Novice
Posts: 8
Liked: 1 time
Joined: Jun 06, 2018 1:06 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by GACcc »

same here, I started a backup for all backupsets yesterday evening, before this whole story it took approx 1.5h and for now its about 4.5 - 5 hours.
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt »

In my case, the task "compacting full backup file (xx% done) [fast clone] " takes too much time. This task runs for each VM in the job after the copy jobs copies the most recent restore point for every VM.
It's not that the task has problems or errors out, it just takes too long. I don't know why. I would assume the fast cloning should be much much faster. I'm curious to know by which aspects of a Veeam configuration(GFS, datasize, restorepoints, #VMs) fast clone performance / run time would be impacted the most.

I've got this job for 100 VMs, now running that compacting fast clone task for 6,5 hours. Data size of the job s 7.7TB.
GFS is 7 daily, 4 weekly, 3 monthly 4 quarterly, 1 yearly.
GFS selection is the same day of the week for all.
I have 5 copyjobs besides this one, so I reduced the repository tasks from 16 to max 3. This way the server is more dedicated to a single job when it runs. Just to rule out some things regarding runngiun
That seems to improve some things regarding stability, but not regarding speed of fast clone actions. I barely seem to make it within a 24 hour interval.
Veeam Certified Engineer
poulpreben
Certified Trainer
Posts: 1024
Liked: 448 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: Windows 2019, large REFS and deletes

Post by poulpreben »

I recently remounted ReFS from a Server 2016 to Server 2019 (LTSC), which caused performance to tank. Then I attached the volumes to a Server 1903 Core, and I'm still experiencing periodical slow performance.

@mkretzer: Did you just attach existing volumes to your Server 1903, or did you create new ones? I am wondering if it could be trying to run trim/unmap on the 2016 data or something.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » 1 person likes this post

We recreated the volume and data. But according to microsoft it should not be necesarry. Is your backend storage trim/unmap capable and did you disable that in windows?
Post Reply

Who is online

Users browsing this forum: Bing [Bot], divertisity, ybarrap2003 and 145 guests