Comprehensive data protection for all workloads
JaySt
Service Provider
Posts: 162
Liked: 24 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt » Oct 10, 2019 5:21 pm

Can someone running 2019 build 1903 post the version of the refs.sys file?
Veeam Certified Engineer

DonZoomik
Expert
Posts: 126
Liked: 32 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by DonZoomik » Oct 10, 2019 7:12 pm

According to patch notes it hasn't been patched so it's probaby 10.0.18362.0
I don't have an 1903 system right now to check but it's an educated guess from patch info's file list CSV.

mkretzer
Expert
Posts: 556
Liked: 124 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Oct 10, 2019 10:20 pm

Correct, 10.0.18362.0. Thats the version working perfectly for us. It is working so good i even understand why they never patched anything in that version.

WorkingHardInIt
Veeam Vanguard
Posts: 25
Liked: 4 times
Joined: Feb 14, 2014 1:27 pm
Full Name: Didier Van Hoye
Contact:

Re: Windows 2019, large REFS and deletes

Post by WorkingHardInIt » Oct 11, 2019 7:23 am

Any one tested 10.0.17763.771 yet on Windows Server 2019 1809?

dasfliege
Service Provider
Posts: 34
Liked: 6 times
Joined: Nov 17, 2014 1:48 pm
Location: Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by dasfliege » Oct 11, 2019 9:05 am

@WorkingHardInIt
Not yet. When and how has it been released?

@mkretzer / @gostev
Would you may share the Microsoft case number, where you already had contact with the ReFS Devs? MS is asking me for it.

GACcc
Novice
Posts: 4
Liked: never
Joined: Jun 06, 2018 1:06 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by GACcc » Oct 11, 2019 12:52 pm

WorkingHardInIt wrote:
Oct 11, 2019 7:23 am
Any one tested 10.0.17763.771 yet on Windows Server 2019 1809?
Yes I have... and its not working for us... its horrible.
The first backups are "pretty" fust but then it takes hours and hours

JaySt
Service Provider
Posts: 162
Liked: 24 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt » Oct 11, 2019 8:50 pm

I did some testing as well. We had two repository servers with Server 2019- 1809. We do copy jobs from the first to the second one. Only on the second one, the one doing GFS stuff, things went bad with merging times. Way too long. Had to restart everything to get it going again each time.
I read through this thread, wiped some tears when i concluded MS created a seemingly stable refs.sys driver a while back, but not fixing this for 1809 customers, and we decided to rip out a refs.sys file from a Windows 10 1903 patched desktop and place it on the second repository only. Refs.sys version coming from the windows 10 1903 machine was 10.0.18362.1 , modified date 19-3-2019.
I have to see how it goes in the coming days, will report back.
Veeam Certified Engineer

mkretzer
Expert
Posts: 556
Liked: 124 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » Oct 12, 2019 11:29 am

@dasfliege I could give you one of our case numbers but i do not know if that would help as we really had direct contact with the devs - without a ticket system inbetween...

JaySt
Service Provider
Posts: 162
Liked: 24 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt » Oct 14, 2019 1:50 pm

JaySt wrote:
Oct 11, 2019 8:50 pm
I did some testing as well. We had two repository servers with Server 2019- 1809. We do copy jobs from the first to the second one. Only on the second one, the one doing GFS stuff, things went bad with merging times. Way too long. Had to restart everything to get it going again each time.
I read through this thread, wiped some tears when i concluded MS created a seemingly stable refs.sys driver a while back, but not fixing this for 1809 customers, and we decided to rip out a refs.sys file from a Windows 10 1903 patched desktop and place it on the second repository only. Refs.sys version coming from the windows 10 1903 machine was 10.0.18362.1 , modified date 19-3-2019.
I have to see how it goes in the coming days, will report back.
unfortunately, this did not work. GFS restorepoint creation still running. job takes more than 80 hours now. Strange, i had good hopes replacing refs.sys would help. Either this is a different issue not related to ReFS.sys somehow or just replacing refs.sys is not the complete story to fix these kind of issues.
Veeam Certified Engineer

steggles
Novice
Posts: 5
Liked: never
Joined: Jul 10, 2019 11:52 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by steggles » Oct 14, 2019 4:31 pm

Hi All,

We have just built 2019 Veeam Repositories using the Long Term Servicing Branch of Windows Server 2019. We have been having a few issues with high CPU on ReFS jobs, eventually they cause the server to lock up, and all the other jobs wait on them. It tends to happen on Health Check and Compact operations on the jobs. But from the thread here it would seem to indicate I need to have disks with size greater than 100TB, is that right? As I only have a number of ~74TB drives as part of a bigger SOBR. I wouldn't have thought I would be impacted by this.

I have opened a ticket with Veeam. Should I upgrade to 1903? or is there likely another cause?

Regards,
Thomas Higgins

JaySt
Service Provider
Posts: 162
Liked: 24 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: Windows 2019, large REFS and deletes

Post by JaySt » Oct 15, 2019 2:18 pm

I also have disk sizes below or around 100TB and seeing problems.
you can't upgrade to 1903. It's a new install. it's all just such a sad story.
Veeam Certified Engineer

dasfliege
Service Provider
Posts: 34
Liked: 6 times
Joined: Nov 17, 2014 1:48 pm
Location: Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by dasfliege » Oct 21, 2019 11:32 am 1 person likes this post

Another week, another try:
After MS analysed the latest memory dumps i've collected, while our ReFS performance was almost down to zero, they came up with another possible solution. I've already applied it and so far it looks quite promising. Block Clone activities are now running since 1-2 hours and disk activity never dropped to 0 so far.

Would be nice if some other guys could test it and give a feedback. All you have to do is creating the following regkey and reboot the server:
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
REG_DWORD value: "RefsEnableLargeWorkingSetTrim"=1

There are some KB describing the issue for WS2016. But it seems like it has reappeared on WS2019:
https://support.microsoft.com/en-us/hel ... is-running
https://support.microsoft.com/en-us/hel ... windows-10

TheJourney
Influencer
Posts: 14
Liked: 4 times
Joined: Dec 10, 2009 8:44 pm
Full Name: Sam Journagan
Contact:

Re: Windows 2019, large REFS and deletes

Post by TheJourney » Oct 22, 2019 2:29 pm 2 people like this post

I can confirm REFS on windows 2019 is Junk.

Our Exchange server merges were stalling after 8 hours or so, many times not completing at all. The last successful merge on 2019 took over 17 hours. Rebooting the repo would help for a day or so then the issue would come right back.

I reinstalled with Windows Server 1903, kept the data volume (no ReFs reformat) attached it to the server with the same drive letter keeping all the backup files. Kept the same IP address, server name, etc. Opened up Veeam > Backup Infrastructure > MS Windows > Properties of the existing server object in Veeam. This installs the Transport Agent. Then ran through the properties dialog under backup Repos, this installs the other Veeam components needed for Repos. I was then able to run backup jobs without changing anything else.

Immediately retried a failed Exchange merge and it completed in 30 mins. The next exchange merge took less than 2 hours. All backup jobs are completing normally after the Repo migration. Not sure if this is a supported move, but working great so far. I hope it stays this way!!!

Cheers!

jayscarff
Service Provider
Posts: 93
Liked: 6 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: Windows 2019, large REFS and deletes

Post by jayscarff » Oct 22, 2019 10:52 pm

dasfliege wrote:
Oct 21, 2019 11:32 am
Another week, another try:
After MS analysed the latest memory dumps i've collected, while our ReFS performance was almost down to zero, they came up with another possible solution. I've already applied it and so far it looks quite promising. Block Clone activities are now running since 1-2 hours and disk activity never dropped to 0 so far.

Would be nice if some other guys could test it and give a feedback. All you have to do is creating the following regkey and reboot the server:
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
REG_DWORD value: "RefsEnableLargeWorkingSetTrim"=1

There are some KB describing the issue for WS2016. But it seems like it has reappeared on WS2019:
https://support.microsoft.com/en-us/hel ... is-running
https://support.microsoft.com/en-us/hel ... windows-10
Just seeing all clients useing our Cloud Repo (the refs one) slowing down completely. While NTFS cloud repo has no issues. Only thing that has changed are windows updates. Im applying this reg key now and will reboot.
Jason
VMCE v9

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], vhernandez and 27 guests