Windows 2019, large REFS and deletes

[infused]

Just a note, you will notice (maybe) Windows Defender will have a big impact on writing files. I've noticed this on a couple of backup boxes. Removing Windows Defender seems to be the only way. Disabling real-time scanning does not seem to make a difference.

Uninstall-WindowsFeature -Name Windows-Defender

[SteelContainer]

We are running the following for most of our reps.
OS: Server 2019 '1809' and the latest 2020-05 CU.
Hardware: Lenovo SR590
Disks: OS on 120GB SSD mirror kit, backups on 150TB ReFS volume with 64k block size (14 x 14TB disks in RAID6 512K RAID striping)
CPU: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095 Mhz, 8 Core(s), 16 Logical Processor(s)
RAM: 64GB
NIC: 10G fibre SFP

Veeam: Veeam 10 - 10.0.0.4461 P1
In terms of jobs most are 28 daily forever forward incremental - no synthetic fulls (but looking at enabling this)
Repository task limits recently set to 14 (same as number of disks as recommended by Veeam support)

We have never really had issues with the CPU load or memory usage, typically no higher than 40% for CPU and 15GB for RAM but I applied the following two changes to two of our reps a few days ago and the CPU dropped right down (MAX is now 6%), the processor queue length also had a significant drop, almost non existent. Memory usage dropped a little but not as significant. So perhaps even 40% CPU usage was a sign something was wrong.

I'm currently making the same changes to more of our reps:
fsutil behavior set DisableDeleteNotify ReFS 1
REG ADD HKLM\System\CurrentControlSet\Control\FileSystem /v RefsEnableLargeWorkingSetTrim /t REG_DWORD /d 1

[JRRW]

Many experience this and it's a real shame that there is no solution. It appears over time what used to be mostly-sequential reads become more like mostly-random reads.

Has anyone tried accelerating a Veeam ReFS repository with Flash?

Yes. Mixed bag.

It's fast, if your SSD tier is large enough. The issue is de-staging - particularly with what I'm suspecting more and more is a bug in Storage Spaces ReFS where a fast-clone that is attempted with the Performance Tier > 90% full, it literally crashes the volume and won't be recoverable until a reboot. This is true for both a repository AND a source, if you're using Hyper-V.

i.e. backups were working fine on a new system I'm running (1.2TiB SSD Mirror Tier / 50TiB parity tier via 10x8TB NL-SAS) but when it went to do a synthetic full, zoink, ded.

I'm currently setting my destage to 15% which means it should start to destage at anything over ~ 187GiB in the SSD Tier - this is the reason:
When under load, Storage Spaces MAP (mirror accelerated parity) doesn't not seem to destage until the load is not on it anymore. I can see this to be true when i start a backup job while it's de-staging, and suddenly the de-staging stops. Once i kill the backup job (or even file copy job to that volume) the de-stage starts again.

Which means, you have to ensure your incremental never fill the performance tier, because it will need to de-stage them before the Synthetic Full runs, or at least have at least 15% free to not crash the fast clone.

I'm still doing some checking on this but that's what i've found thus far. I have a few extra SSD i had intended on using for a replication repository - i might steal two from that to bump the SSD tier up to a larger size.


That all having been said: Microsoft recommends no smaller than 10% Mirror size to the Parity - which means technically I'm not running what I should be, it should be 5TiB of Performance, not 1.2TiB.

Really SSD isn't that expensive, but as this has to be Mirror (not parity) for the performance tier, it becomes a lot more expensive than if it were just a straight Raid-5 SSD.


I'd go the route of SOBR, and just make a raid-5 SSD 'performance' veeam tier for incremental - but you lose the Fast-Clone Synthetic Fulls by not having them on the same ReFS volume, from what I can tell.

[JRRW]

Just a note, you will notice (maybe) Windows Defender will have a big impact on writing files. I've noticed this on a couple of backup boxes. Removing Windows Defender seems to be the only way. Disabling real-time scanning does not seem to make a difference.

Uninstall-WindowsFeature -Name Windows-Defender

Did you exclude all of the proper directories?

Code: Select all

Write-Host "Adding Recommended Veeam Anti-Virus Exclusions" -ForegroundColor Green 
Write-Host "Excluding Recomended File Extensions" -ForegroundColor Yellow 

Add-MpPreference -ExclusionExtension ".vmdk"
Add-MpPreference -ExclusionExtension ".flat"
Add-MpPreference -ExclusionExtension ".vbm"
Add-MpPreference -ExclusionExtension ".vbk"
Add-MpPreference -ExclusionExtension ".vib"
Add-MpPreference -ExclusionExtension ".vrb"
Add-MpPreference -ExclusionExtension ".vsb"
Add-MpPreference -ExclusionExtension ".vlb"
Add-MpPreference -ExclusionExtension ".erm"

Write-Host "Excluding Recommended Folders used by Veeam" -ForegroundColor Yellow 
Add-MpPreference -ExclusionPath "C:\Program Files\Veeam"
Add-MpPreference -ExclusionPath "C:\Program Files (x86)\Veeam"
Add-MpPreference -ExclusionPath "C:\Program Files\Common Files\Veeam"
Add-MpPreference -ExclusionPath "C:\Program Files (x86)\Common Files\Veeam"
Add-MpPreference -ExclusionPath "C:\ProgramData\Veeam"
Add-MpPreference -ExclusionPath  "C:\VeeamFLR" 
Add-MpPreference -ExclusionPath "C:\Windows\Veeam"
#This may vary based on your repository path, rep path, etc
Add-MpPreference -ExclusionPath "D:\Backup"
Add-MpPreference -ExclusionPath "D:\VBRCatalog"
Add-MpPreference -ExclusionPath "V:\InstantRecoveryCache"
Add-MpPreference -ExclusionPath "V:\VeeamReplication"

[popjls]

Yes. The registry tweaks are still recommended.

• Ensure Trim is disabled

Code: Select all

 fsutil behavior set DisableDeleteNotify ReFS 1

• Set RefsEnableLargeWorkingSetTrim = 1

Code: Select all

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem
Value Name: RefsEnableLargeWorkingSetTrim
Value Type: REG_DWORD
Value Data: 1

My backup repository is thin provisioned iscsi all ssd with trim enabled on the backend. Do we still need to disable trim in Windows? I did my first full fast clone of a single 15TB VM and it did it in one hour exactly. Not sure if that is good or bad in regards to the speed of the clone but if it's not great, should we still enable these tweaks?

[Gostev]

Yes, you should.

[backupquestions]

As of today, with all the refs 2019 issues.... I've got several sites with 2016 refs and they all work flawlessly so far.

Is it recommended to go to 2019 on any "new" set ups? Most of me wants to just keep standing up 2016 as I think the extended support is over 2027 so we have plenty of time.

This site is only say 20TB full, not huge. Will be doing weekly synthetic fulls refs for offloading to object storage etc.

If it is for sure safe I can try 2019 on this new setup, but otherwise I will just stick with ole trusty 2016.

[Gostev]

Simply put, 2016 is proven to be rock solid.

2019 cannot be considered proven yet, because the final fixes are fairly recent. But from technical perspective, there are not outstanding issues with 2019 that we're waiting for Microsoft to fix. So, it just needs more time now before I can confidently state that it's proven too.

[expert]

We are running the following for most of our reps.
OS: Server 2019 '1809' and the latest 2020-05 CU.
Hardware: Lenovo SR590
Disks: OS on 120GB SSD mirror kit, backups on 150TB ReFS volume with 64k block size (14 x 14TB disks in RAID6 512K RAID striping)
CPU: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095 Mhz, 8 Core(s), 16 Logical Processor(s)
RAM: 64GB
NIC: 10G fibre SFP

Veeam: Veeam 10 - 10.0.0.4461 P1

Hi!

We have a similar configuration, 2 completely new Veeam Servers (Veeam role not installed yet). I'm insecure about the Windows Server Version: Windows Server 1809 (actually installed) or Windows Server 1909 Core

So you did not had any issues with ReFS and Windows Server 2019 '1809'? In the forum there are reports that Windows Server 2019 prior '1903' can cause several problems.

best regards,

Juergen

[mkretzer]

With the latest ReFS Patches windows 2019 is quite unable and stable. 1909 should have some performance optimizations but our 400 TB repo on 2019 does not run much worse than our 600 TB 1909 repo...

[AuGL]

With the latest ReFS Patches windows 2019 is quite unable and stable.

Did you mean "usable and stable"?

[mkretzer]

Yep, thats what i meant

[Gostev]

This typo is absolutely epic!

[AuGL]

What is everyone using then for their ReFS Repository Servers .... 2019 LTSC or the Semi-Annual build like 1909?

[mkretzer]

@AuGL how many TB of backups do you have? I think for most customers the difference is not "feelable", we only use both to have different ReFS drivers for backup and backup copy locations - if there is a major bug we might be lucky and it only affects one repo...

[AuGL]

Around 90TB at the moment ... but that will grow to double that over the next 12 months.

[DonZoomik]

I have 3 relatively large repos. 65T, 232T, 291T, all on WS2019 1809 LTSC. With RefsEnableLargeWorkingSetTrim and patches, it's... fine. Before March patches it occasionally slowed down (but never crashed) with very large (~30T) synthetics - clearing system working set always made it resume. File system fragmentation slows things down a bit but nothing too dramatic.

[AuGL]

Thanks for replying and sharing some info.

clearing system working set always made it resume

What is this?

[DonZoomik]

ReFS driver leaks (not exactly the correct term in this context but close enough) some memory and with very large synthetics (well, mostly used to leak), it seems to drown in it's own memory management. If you go back in this thread, you'll find a tool that regularly tells the system (eg kernel, not user-mode processes) to reduce working set (effectively to drop some memory usage, flush caches, page to disk...). This causes ReFS to flush a lot of memory and performance goes up again. You can do this manually from Sysinternals RAMMap tool as well.
I haven't had to do this since spring.

[AuGL]

Thanks, good to hear.

[Steve-nIP]

What about 2004? It'd be interesting to hear how ReFS performs in the newest build compared to 1607, 1809 and say, 1909, or at least what changes have occurred in the ReFS stack

[mkretzer]

I did not yet ask microsoft. Will do now...

[PeterC]

It looks like we are back to square 1, at this moment our 2019 LTSC repo is crawling to the finishline during backups.

We are using an HPE Apollo 4200 with 1 200 TB volume as repo for our Veeam VBR 10a.
We have had a lot of trouble in the past which seemed to have been solved after ReFS patch KB 4531331.
After setting the task limit on that repo to 30 (32 core cpu) and installing the patch backups were running normal again.

Our backups used to be done at around 15:00, but after this they were finished between 06:00 - 07:00. We almost cracked open a bottle of champagne.
Lately backups started to run a bit longer, nothing alarming.
But to days ago suddenly backups wer slowing down a lot more, they never finished before the afternoon.
No jobs were added or anything else (except Windows updates), and again we were in trouble.

What we see is that when jobs start the I/O to the repo is between 1 - 6 GB/s for the combined jobs. But when one of the jobs is ready and starts the merge, the I/O for the other jobs just drops to a few MB/s and sometimes KB/s. When the merges are finished the I/O returns to normal values.
But because we have multiple jobs running from 20:30 to 03:00, the later the job start the longer it takes for the job to finish.
We see some jobs waiting for infrastructue availability more than 10 hours.
We have actually no idea why this is suddenly happening. As i said before, we haven't changed anything on this repo.

Just to make sure all other repos (windows 2016) are performing like clockwork.

[PeterC]

We have opened a new Case # 04331174. Just noticed that the ReFS.sys is newer than the one in KB 4531331.
The version is now 10.0.17763.1192 (date modified 7/14/2020).
Could this be the issue?

[Fiqsz]

Not sure if it helps, but I've just updated our Windows 2019 Server with REFS and refs.sys version is now 10.0.17763.1369.
Windows CU 2020.08

[DonZoomik]

Are you sure that you have RefsEnableLargeWorkingSetTrim registry value set? I was just testing if this flag caused some other problems and after removing it and restarting, I immediately hit similar problems during merge.

[PeterStam]

Yes it is set DonZoomik (colleague of PeterC).

[mkretzer]

@peterc Did you install update 10a?

For us after installing update 10a our synthetics for some jobs started to take 10+ hours instead of < 1 hour! ReFS does not have any "strange" messages in event log, still speed is MUCH slower now.

@GOSTEV:
Case #04337327

[koenteugels]

So for a new installation veeam 10a with 4 physical proxy servers 200 TB source data, do you recommend windows 2016 in stead of 2019

[PeterC]

@DonZoomik, yes the RefsEnableLargeWorkingSetTrim is still set to 1.
@mkretzer, yes we have installed update 10a, we had a lot of trouble with the management console on our VBR after update 10. It was tremendously slow, almost unworkable. Have had a case for this for several months and it was mostly fixed in update 10a.
I cannot really tell you if all of this happened after update 10a. We also opened a case with MS (Case 120081425000237) for this, so hope that Veeam or MS can find the cause.
@koenteugels, at this moment i would definitely choose for Server 2016. We have a lot of (virtual) 2016 repos that work without any problem. Got 1 (physical) 2019 repo and it almost has been one (big) nightmare. We have no experience with the 2019 Core versions (1909, 2004), so can not advice on these versions.