Comprehensive data protection for all workloads
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev »

PeterC wrote: Apr 13, 2021 1:54 pmIn the past we have tested a little bit with synthetic fulls, but we noticed a significant increase of storage usage so returned back to the forward incremental.
Synthetic fulls don't take up any physical disk space on ReFS (and never did) so whatever storage consumption increase you saw was unrelated to the backup mode change.
ITP-Stan
Service Provider
Posts: 201
Liked: 55 times
Joined: Feb 18, 2013 10:45 am
Full Name: Stan (IF-IT4U)
Contact:

Re: Windows 2019, large REFS and deletes

Post by ITP-Stan »

With ReFS repo set-up correctly synthetic fulls should not really increase storage usage.
Maybe only a little bit depending on the number of restore points you keep and the closing of the chain.
jeffshead
Enthusiast
Posts: 71
Liked: 3 times
Joined: May 05, 2016 1:07 pm
Full Name: Jeff
Contact:

Re: Windows 2019, large REFS and deletes

Post by jeffshead »

Does anyone know if this issue still exists with Server 2019 LTSC (1809) with the April 9th, 2021 refs.sys version 10.0.17763.1728?

I'm not sure if I should use NTFS or ReFS for the repository.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

I'll ask Microsoft - i wanted to ask them anyway about 21H1
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev »

jeffshead wrote: May 21, 2021 2:21 amDoes anyone know if this issue still exists
Can you clarify which issue are you talking about? There haven't really been any around Server 2019 for a long time now.
jeffshead
Enthusiast
Posts: 71
Liked: 3 times
Joined: May 05, 2016 1:07 pm
Full Name: Jeff
Contact:

Re: Windows 2019, large REFS and deletes

Post by jeffshead »

Gostev wrote: May 21, 2021 11:39 amCan you clarify which issue are you talking about? There haven't really been any around Server 2019 for a long time now.
Thanks for chiming in. My inquiry was for any issues listed in this thread or any issues in general. I haven't formatted the repository drive yet. Wanted to have confidence that ReFS is going to be at least as reliable/robust as NTFS on Server 2019 LTSC (1809).
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

How big is your repo? With 2019 issues only started to come up with a repo way over 300 TB in one extend - even that system runs normally without issues for months!
All that with a very old repo hardware (Dell R720).
jeffshead
Enthusiast
Posts: 71
Liked: 3 times
Joined: May 05, 2016 1:07 pm
Full Name: Jeff
Contact:

Re: Windows 2019, large REFS and deletes

Post by jeffshead »

mkretzer wrote:How big is your repo?
36TB on Dell R720XD, Windows Server 2019 LTSC (1809). The main reason for my post was because some posted that there were ReFS driver issues with 1809 but not with newer versions of Windows.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » 2 people like this post

I am 98 % certain you will have no issues at that scale. Our 2019 LTSC repo (with newest updates!) is an old R720 with 300 TB. It ran nearly flawless for a year now and only showed issues when very large backups were deleted.
jbcap
Influencer
Posts: 18
Liked: never
Joined: May 24, 2018 4:12 am
Full Name: Jim Bell
Contact:

Re: Windows 2019, large REFS and deletes

Post by jbcap »

With 2019 1809, are there currently any recommended registry settings or hotfixes? Or is a vanilla install fine at the moment?
Are there any benefits to moving off the LTSB to Semi-Annual release?

refs.sys is 10.0.17763.1971

Thanks.
jeffshead
Enthusiast
Posts: 71
Liked: 3 times
Joined: May 05, 2016 1:07 pm
Full Name: Jeff
Contact:

Re: Windows 2019, large REFS and deletes

Post by jeffshead »

jbcap wrote:Or is a vanilla install fine at the moment?
I think these issues have been resolved.

I made sure my 1809 server had the latest updates and I recreated a repository about a month ago -- Switched from NTFS to ReFS and have encountered zero issues. However, my full backup went from taking 4.25 hrs on NTFS to taking over 6 hrs to complete on ReFS. Synthetic fulls take no time to create on ReFS but I'm still considering going back to NTFS and active fulls.
DerOest
Enthusiast
Posts: 71
Liked: 42 times
Joined: Oct 30, 2015 10:10 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by DerOest »

@jeffshead, from the previous page:
dasfliege wrote: Jan 05, 2021 8:04 am Hi Adrian_C

If you run any cumulative later then apil 19, you should also have the ReFS fix in place. Check the version of your refs.sys driver. Latest LTS version is 10.0.17763.1369

Did you also applied largeworkingsettrim and deletenotify? If not, execute the following two commands in an elevated command prompt:
fsutil behavior set DisableDeleteNotify ReFS 1
REG ADD HKLM\System\CurrentControlSet\Control\FileSystem /v RefsEnableLargeWorkingSetTrim /t REG_DWORD /d 1
jeffshead
Enthusiast
Posts: 71
Liked: 3 times
Joined: May 05, 2016 1:07 pm
Full Name: Jeff
Contact:

Re: Windows 2019, large REFS and deletes

Post by jeffshead »

DerOest wrote:@jeffshead, from the previous page:
Thanks but I think that suggestion applies only to repositories that are on SSD's, correct? That will not help if the repository is on mechanical drives, will it?
DerOest
Enthusiast
Posts: 71
Liked: 42 times
Joined: Oct 30, 2015 10:10 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by DerOest »

These options have nothing to do with SSD/HDD, they are ReFS-internal settings and generally valid
jeffshead
Enthusiast
Posts: 71
Liked: 3 times
Joined: May 05, 2016 1:07 pm
Full Name: Jeff
Contact:

Re: Windows 2019, large REFS and deletes

Post by jeffshead »

DerOest wrote:These options have nothing to do with SSD/HDD, they are ReFS-internal settings and generally valid
I'm not second-guessing you but all of the results turned up from my previous Internet searches state that those settings relate to trim; hence SSDs. So that I have a better understanding, can you explain or point me to any articles that describe how/why those settings improve performance regardless of the hard drive type(s) that are installed in the system?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev »

Trim is a generic term, not necessarily exclusive to SSD. For example, memory usage can also be "trimmed" as is the case with RefsEnableLargeWorkingSetTrim value, which controls how aggressively the working set of the ReFS metadata management process is reduced.

I don't know if Microsoft has public articles with the level of details you require, as they would have to go deep into ReFS internals to explain "how/why those settings improve performance". Perhaps try contacting Microsoft Support to see if they would be able to provide this information to you.
jbcap
Influencer
Posts: 18
Liked: never
Joined: May 24, 2018 4:12 am
Full Name: Jim Bell
Contact:

Re: Windows 2019, large REFS and deletes

Post by jbcap »

So, is the consensus that we should apply the RefsEnableLargeWorkingSetTrim registry setting at the moment with an up-to-date 1809?
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

Last information i have from microsoft is that we should activate it...
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

Does anyone else have bad ReFS write performance with the latest hotfixes (CU6 + Printernightmare fix)? We have much more target bottleneck and our bigger jobs take 2-3 hours longer.
There is a new ReFS Driver in one of these two updates (at least for SAC 2004 and 21H1, checking LTSC currently).

Markus
poulpreben
Certified Trainer
Posts: 1024
Liked: 448 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: Windows 2019, large REFS and deletes

Post by poulpreben »

We are on LTSC with latest patches and are currently troubleshooting some performance issues. Throughput goes to near zero as soon as just one job starts merging.

Subscribing for updates…
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

The driver on LTSC is from 12.05.2021... So its unlikely the same happens there...

I asked MS, will keep you updated.
poulpreben
Certified Trainer
Posts: 1024
Liked: 448 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: Windows 2019, large REFS and deletes

Post by poulpreben »

Does that mean you installed it on that date, or that the driver got signed there?

Patch Tuesday in May was on 11.05.2021, so if it was signed the following day, it would've been included in 2021-06 CU instead.

[Edit]: I just checked it. It was indeed signed on 12.05.2021, and we did not deploy it until 15.06.2021 (one week after Patch Tuesday according to the patching policy for those systems). I will try to correlate this with our performance regressions.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

12.05. is the signing date.
MS wants a memory dump. We will try to provide them one ASAP
Captnspdr
Veeam Vanguard
Posts: 27
Liked: 3 times
Joined: Apr 16, 2019 4:50 pm
Full Name: Jonah May
Contact:

Re: Windows 2019, large REFS and deletes

Post by Captnspdr »

We are seeing performance issues with the latest 2019 cumulative updates as well. We have a cloud host that ReFS.sys seems to be eating the CPU on the server to the point that it has to be rebooted every 5 days or so.

Right now, one job is writing to the server at about 20 Mbps and ReFS.sys is using between 40 and 60% CPU on a Dell R720. Uptime is a little over 3 days.

We have already applied RefsEnableLargeWorkingSetTrim and DisableDeleteNotify.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

Its strange - last week our backups were just slower. This week they work nearly normal. I just wanted inform microsoft that its all good when i saw you message Captnspdr - indeed, our CPU usage seems to be 15-20 % higher after the patch. I will try to verify that with the next backup run.

Which process causes the CPU usage in your system?
Captnspdr
Veeam Vanguard
Posts: 27
Liked: 3 times
Joined: Apr 16, 2019 4:50 pm
Full Name: Jonah May
Contact:

Re: Windows 2019, large REFS and deletes

Post by Captnspdr »

I had to run Windows Performance Analyzer to find out, but the culprit is ReFS.sys.
Captnspdr
Veeam Vanguard
Posts: 27
Liked: 3 times
Joined: Apr 16, 2019 4:50 pm
Full Name: Jonah May
Contact:

Re: Windows 2019, large REFS and deletes

Post by Captnspdr »

Looks like the ReFS driver was updated again in mid-July, but we're still seeing the high CPU usage from ReFS.sys.
LaurentZ
Novice
Posts: 3
Liked: never
Joined: Feb 05, 2013 3:46 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by LaurentZ »

We have 2 Veeam v11 repositories, both Windows Server 2019 v1809 and the behavior is totally different.
One with refs.sys 10.0.17763.1971 : work great
One with refs.sys 10.0.17763.2090 : catastrophic behavior. Copy jobs worked great, but CPU at 100% when merge (fast clone). Even with 1 single job...
Server is totally unresponsive after a couple of hours
eee777
Service Provider
Posts: 5
Liked: never
Joined: Jul 28, 2016 6:47 am
Full Name: Evgeni Stanev
Location: Bulgaria
Contact:

Re: Windows 2019, large REFS and deletes

Post by eee777 »

We moved from Server 2016 / Veeam v10 to Server 2019 build 1809 / Veeam v11 / refs.sys is 10.0.17763.2090. Copy and Tape Jobs are performing much faster now (almost 2x speed) but regular backups got worse. Sometimes there is not throughput for 10-15 mins , then it goes back to normal ... then back to 0. CPU usage is 2-10% without any spikes during the backup / synthetic full jobs. We are using the same storage as we used with Server 2016 , only the standalone backup server was upgraded with 2x more memory - 256GB for single 80TB ReFS repo. The registry fix suggested here is also applied.
Captnspdr
Veeam Vanguard
Posts: 27
Liked: 3 times
Joined: Apr 16, 2019 4:50 pm
Full Name: Jonah May
Contact:

Re: Windows 2019, large REFS and deletes

Post by Captnspdr »

@LaurentZ

We are on 2019 LTSC and it looks like our driver version giving us the issue is 10.0.17763.2090.
Post Reply

Who is online

Users browsing this forum: Baidu [Spider], K-SOLdier, kyle.shuberg, robert.vonmehren, ybarrap2003 and 216 guests