Comprehensive data protection for all workloads
Post Reply
infused
Service Provider
Posts: 132
Liked: 6 times
Joined: Apr 20, 2013 9:25 am
Full Name: Hayden Kirk
Contact:

Re: Windows 2019, large REFS and deletes

Post by infused » 2 people like this post

Just a note, you will notice (maybe) Windows Defender will have a big impact on writing files. I've noticed this on a couple of backup boxes. Removing Windows Defender seems to be the only way. Disabling real-time scanning does not seem to make a difference.

Uninstall-WindowsFeature -Name Windows-Defender
https://www.tecfused.com - my blog

Cullan
Service Provider
Posts: 119
Liked: 17 times
Joined: May 21, 2014 8:47 am
Location: New Zealand
Contact:

Re: Windows 2019, large REFS and deletes

Post by Cullan » 1 person likes this post

We are running the following for most of our reps.
OS: Server 2019 '1809' and the latest 2020-05 CU.
Hardware: Lenovo SR590
Disks: OS on 120GB SSD mirror kit, backups on 150TB ReFS volume with 64k block size (14 x 14TB disks in RAID6 512K RAID striping)
CPU: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095 Mhz, 8 Core(s), 16 Logical Processor(s)
RAM: 64GB
NIC: 10G fibre SFP

Veeam: Veeam 10 - 10.0.0.4461 P1
In terms of jobs most are 28 daily forever forward incremental - no synthetic fulls (but looking at enabling this)
Repository task limits recently set to 14 (same as number of disks as recommended by Veeam support)

We have never really had issues with the CPU load or memory usage, typically no higher than 40% for CPU and 15GB for RAM but I applied the following two changes to two of our reps a few days ago and the CPU dropped right down (MAX is now 6%), the processor queue length also had a significant drop, almost non existent. Memory usage dropped a little but not as significant. So perhaps even 40% CPU usage was a sign something was wrong.

I'm currently making the same changes to more of our reps:
fsutil behavior set DisableDeleteNotify ReFS 1
REG ADD HKLM\System\CurrentControlSet\Control\FileSystem /v RefsEnableLargeWorkingSetTrim /t REG_DWORD /d 1

JRRW
Novice
Posts: 9
Liked: 1 time
Joined: Dec 10, 2019 3:59 pm
Full Name: Ryan Walker
Contact:

Re: Windows 2019, large REFS and deletes

Post by JRRW » 1 person likes this post

evilaedmin wrote:
Feb 22, 2020 3:35 am
Many experience this and it's a real shame that there is no solution. It appears over time what used to be mostly-sequential reads become more like mostly-random reads.

Has anyone tried accelerating a Veeam ReFS repository with Flash?
Yes. Mixed bag.

It's fast, if your SSD tier is large enough. The issue is de-staging - particularly with what I'm suspecting more and more is a bug in Storage Spaces ReFS where a fast-clone that is attempted with the Performance Tier > 90% full, it literally crashes the volume and won't be recoverable until a reboot. This is true for both a repository AND a source, if you're using Hyper-V.

i.e. backups were working fine on a new system I'm running (1.2TiB SSD Mirror Tier / 50TiB parity tier via 10x8TB NL-SAS) but when it went to do a synthetic full, zoink, ded.

I'm currently setting my destage to 15% which means it should start to destage at anything over ~ 187GiB in the SSD Tier - this is the reason:
When under load, Storage Spaces MAP (mirror accelerated parity) doesn't not seem to destage until the load is not on it anymore. I can see this to be true when i start a backup job while it's de-staging, and suddenly the de-staging stops. Once i kill the backup job (or even file copy job to that volume) the de-stage starts again.

Which means, you have to ensure your incremental never fill the performance tier, because it will need to de-stage them before the Synthetic Full runs, or at least have at least 15% free to not crash the fast clone.

I'm still doing some checking on this but that's what i've found thus far. I have a few extra SSD i had intended on using for a replication repository - i might steal two from that to bump the SSD tier up to a larger size.


That all having been said: Microsoft recommends no smaller than 10% Mirror size to the Parity - which means technically I'm not running what I should be, it should be 5TiB of Performance, not 1.2TiB.

Really SSD isn't that expensive, but as this has to be Mirror (not parity) for the performance tier, it becomes a lot more expensive than if it were just a straight Raid-5 SSD.


I'd go the route of SOBR, and just make a raid-5 SSD 'performance' veeam tier for incremental - but you lose the Fast-Clone Synthetic Fulls by not having them on the same ReFS volume, from what I can tell.

JRRW
Novice
Posts: 9
Liked: 1 time
Joined: Dec 10, 2019 3:59 pm
Full Name: Ryan Walker
Contact:

Re: Windows 2019, large REFS and deletes

Post by JRRW »

infused wrote:
May 25, 2020 4:46 am
Just a note, you will notice (maybe) Windows Defender will have a big impact on writing files. I've noticed this on a couple of backup boxes. Removing Windows Defender seems to be the only way. Disabling real-time scanning does not seem to make a difference.

Uninstall-WindowsFeature -Name Windows-Defender
Did you exclude all of the proper directories?

Code: Select all

Write-Host "Adding Recommended Veeam Anti-Virus Exclusions" -ForegroundColor Green 
Write-Host "Excluding Recomended File Extensions" -ForegroundColor Yellow 

Add-MpPreference -ExclusionExtension ".vmdk"
Add-MpPreference -ExclusionExtension ".flat"
Add-MpPreference -ExclusionExtension ".vbm"
Add-MpPreference -ExclusionExtension ".vbk"
Add-MpPreference -ExclusionExtension ".vib"
Add-MpPreference -ExclusionExtension ".vrb"
Add-MpPreference -ExclusionExtension ".vsb"
Add-MpPreference -ExclusionExtension ".vlb"
Add-MpPreference -ExclusionExtension ".erm"

Write-Host "Excluding Recommended Folders used by Veeam" -ForegroundColor Yellow 
Add-MpPreference -ExclusionPath "C:\Program Files\Veeam"
Add-MpPreference -ExclusionPath "C:\Program Files (x86)\Veeam"
Add-MpPreference -ExclusionPath "C:\Program Files\Common Files\Veeam"
Add-MpPreference -ExclusionPath "C:\Program Files (x86)\Common Files\Veeam"
Add-MpPreference -ExclusionPath "C:\ProgramData\Veeam"
Add-MpPreference -ExclusionPath  "C:\VeeamFLR" 
Add-MpPreference -ExclusionPath "C:\Windows\Veeam"
#This may vary based on your repository path, rep path, etc
Add-MpPreference -ExclusionPath "D:\Backup"
Add-MpPreference -ExclusionPath "D:\VBRCatalog"
Add-MpPreference -ExclusionPath "V:\InstantRecoveryCache"
Add-MpPreference -ExclusionPath "V:\VeeamReplication"


popjls
Influencer
Posts: 14
Liked: 1 time
Joined: Jun 25, 2018 3:41 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by popjls »

Andrew@MSFT wrote:
Mar 18, 2020 12:43 am
Yes. The registry tweaks are still recommended.

• Ensure Trim is disabled

Code: Select all

 fsutil behavior set DisableDeleteNotify ReFS 1
• Set RefsEnableLargeWorkingSetTrim = 1

Code: Select all

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem
Value Name: RefsEnableLargeWorkingSetTrim
Value Type: REG_DWORD
Value Data: 1
My backup repository is thin provisioned iscsi all ssd with trim enabled on the backend. Do we still need to disable trim in Windows? I did my first full fast clone of a single 15TB VM and it did it in one hour exactly. Not sure if that is good or bad in regards to the speed of the clone but if it's not great, should we still enable these tweaks?

Gostev
SVP, Product Management
Posts: 26498
Liked: 4150 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev » 1 person likes this post

Yes, you should.

backupquestions
Expert
Posts: 181
Liked: 20 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Windows 2019, large REFS and deletes

Post by backupquestions »

As of today, with all the refs 2019 issues.... I've got several sites with 2016 refs and they all work flawlessly so far.

Is it recommended to go to 2019 on any "new" set ups? Most of me wants to just keep standing up 2016 as I think the extended support is over 2027 so we have plenty of time.

This site is only say 20TB full, not huge. Will be doing weekly synthetic fulls refs for offloading to object storage etc.

If it is for sure safe I can try 2019 on this new setup, but otherwise I will just stick with ole trusty 2016.

Gostev
SVP, Product Management
Posts: 26498
Liked: 4150 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev »

Simply put, 2016 is proven to be rock solid.

2019 cannot be considered proven yet, because the final fixes are fairly recent. But from technical perspective, there are not outstanding issues with 2019 that we're waiting for Microsoft to fix. So, it just needs more time now before I can confidently state that it's proven too.

expert
Novice
Posts: 5
Liked: never
Joined: May 20, 2016 7:05 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by expert »

Cullan wrote:
May 28, 2020 3:00 am
We are running the following for most of our reps.
OS: Server 2019 '1809' and the latest 2020-05 CU.
Hardware: Lenovo SR590
Disks: OS on 120GB SSD mirror kit, backups on 150TB ReFS volume with 64k block size (14 x 14TB disks in RAID6 512K RAID striping)
CPU: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095 Mhz, 8 Core(s), 16 Logical Processor(s)
RAM: 64GB
NIC: 10G fibre SFP

Veeam: Veeam 10 - 10.0.0.4461 P1
Hi!

We have a similar configuration, 2 completely new Veeam Servers (Veeam role not installed yet). I'm insecure about the Windows Server Version: Windows Server 1809 (actually installed) or Windows Server 1909 Core

So you did not had any issues with ReFS and Windows Server 2019 '1809'? In the forum there are reports that Windows Server 2019 prior '1903' can cause several problems.

best regards,

Juergen

mkretzer
Expert
Posts: 667
Liked: 153 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » 1 person likes this post

With the latest ReFS Patches windows 2019 is quite unable and stable. 1909 should have some performance optimizations but our 400 TB repo on 2019 does not run much worse than our 600 TB 1909 repo...

AuGL
Enthusiast
Posts: 45
Liked: 1 time
Joined: May 07, 2019 12:22 am
Full Name: Glenn
Contact:

Re: Windows 2019, large REFS and deletes

Post by AuGL »

With the latest ReFS Patches windows 2019 is quite unable and stable.
Did you mean "usable and stable"? :D

mkretzer
Expert
Posts: 667
Liked: 153 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

Yep, thats what i meant :-)

Gostev
SVP, Product Management
Posts: 26498
Liked: 4150 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows 2019, large REFS and deletes

Post by Gostev »

This typo is absolutely epic!

AuGL
Enthusiast
Posts: 45
Liked: 1 time
Joined: May 07, 2019 12:22 am
Full Name: Glenn
Contact:

Re: Windows 2019, large REFS and deletes

Post by AuGL »

What is everyone using then for their ReFS Repository Servers .... 2019 LTSC or the Semi-Annual build like 1909?

mkretzer
Expert
Posts: 667
Liked: 153 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer »

@AuGL how many TB of backups do you have? I think for most customers the difference is not "feelable", we only use both to have different ReFS drivers for backup and backup copy locations - if there is a major bug we might be lucky and it only affects one repo...

AuGL
Enthusiast
Posts: 45
Liked: 1 time
Joined: May 07, 2019 12:22 am
Full Name: Glenn
Contact:

Re: Windows 2019, large REFS and deletes

Post by AuGL »

Around 90TB at the moment ... but that will grow to double that over the next 12 months.

DonZoomik
Expert
Posts: 216
Liked: 53 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by DonZoomik » 1 person likes this post

I have 3 relatively large repos. 65T, 232T, 291T, all on WS2019 1809 LTSC. With RefsEnableLargeWorkingSetTrim and patches, it's... fine. Before March patches it occasionally slowed down (but never crashed) with very large (~30T) synthetics - clearing system working set always made it resume. File system fragmentation slows things down a bit but nothing too dramatic.

AuGL
Enthusiast
Posts: 45
Liked: 1 time
Joined: May 07, 2019 12:22 am
Full Name: Glenn
Contact:

Re: Windows 2019, large REFS and deletes

Post by AuGL »

Thanks for replying and sharing some info.
clearing system working set always made it resume
What is this?

DonZoomik
Expert
Posts: 216
Liked: 53 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: Windows 2019, large REFS and deletes

Post by DonZoomik »

ReFS driver leaks (not exactly the correct term in this context but close enough) some memory and with very large synthetics (well, mostly used to leak), it seems to drown in it's own memory management. If you go back in this thread, you'll find a tool that regularly tells the system (eg kernel, not user-mode processes) to reduce working set (effectively to drop some memory usage, flush caches, page to disk...). This causes ReFS to flush a lot of memory and performance goes up again. You can do this manually from Sysinternals RAMMap tool as well.
I haven't had to do this since spring.

AuGL
Enthusiast
Posts: 45
Liked: 1 time
Joined: May 07, 2019 12:22 am
Full Name: Glenn
Contact:

Re: Windows 2019, large REFS and deletes

Post by AuGL »

Thanks, good to hear.

Steve-nIP
Service Provider
Posts: 68
Liked: 18 times
Joined: Feb 06, 2018 10:08 am
Full Name: Steve
Contact:

Re: Windows 2019, large REFS and deletes

Post by Steve-nIP »

What about 2004? It'd be interesting to hear how ReFS performs in the newest build compared to 1607, 1809 and say, 1909, or at least what changes have occurred in the ReFS stack

mkretzer
Expert
Posts: 667
Liked: 153 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Windows 2019, large REFS and deletes

Post by mkretzer » 2 people like this post

I did not yet ask microsoft. Will do now...

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 29 guests