-
- Service Provider
- Posts: 178
- Liked: 13 times
- Joined: Apr 20, 2013 9:25 am
- Full Name: Hayden Kirk
- Contact:
Re: Windows 2019, large REFS and deletes
Just a note, you will notice (maybe) Windows Defender will have a big impact on writing files. I've noticed this on a couple of backup boxes. Removing Windows Defender seems to be the only way. Disabling real-time scanning does not seem to make a difference.
Uninstall-WindowsFeature -Name Windows-Defender
Uninstall-WindowsFeature -Name Windows-Defender
-
- Service Provider
- Posts: 157
- Liked: 22 times
- Joined: May 21, 2014 8:47 am
- Location: New Zealand
- Contact:
Re: Windows 2019, large REFS and deletes
We are running the following for most of our reps.
OS: Server 2019 '1809' and the latest 2020-05 CU.
Hardware: Lenovo SR590
Disks: OS on 120GB SSD mirror kit, backups on 150TB ReFS volume with 64k block size (14 x 14TB disks in RAID6 512K RAID striping)
CPU: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095 Mhz, 8 Core(s), 16 Logical Processor(s)
RAM: 64GB
NIC: 10G fibre SFP
Veeam: Veeam 10 - 10.0.0.4461 P1
In terms of jobs most are 28 daily forever forward incremental - no synthetic fulls (but looking at enabling this)
Repository task limits recently set to 14 (same as number of disks as recommended by Veeam support)
We have never really had issues with the CPU load or memory usage, typically no higher than 40% for CPU and 15GB for RAM but I applied the following two changes to two of our reps a few days ago and the CPU dropped right down (MAX is now 6%), the processor queue length also had a significant drop, almost non existent. Memory usage dropped a little but not as significant. So perhaps even 40% CPU usage was a sign something was wrong.
I'm currently making the same changes to more of our reps:
fsutil behavior set DisableDeleteNotify ReFS 1
REG ADD HKLM\System\CurrentControlSet\Control\FileSystem /v RefsEnableLargeWorkingSetTrim /t REG_DWORD /d 1
OS: Server 2019 '1809' and the latest 2020-05 CU.
Hardware: Lenovo SR590
Disks: OS on 120GB SSD mirror kit, backups on 150TB ReFS volume with 64k block size (14 x 14TB disks in RAID6 512K RAID striping)
CPU: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095 Mhz, 8 Core(s), 16 Logical Processor(s)
RAM: 64GB
NIC: 10G fibre SFP
Veeam: Veeam 10 - 10.0.0.4461 P1
In terms of jobs most are 28 daily forever forward incremental - no synthetic fulls (but looking at enabling this)
Repository task limits recently set to 14 (same as number of disks as recommended by Veeam support)
We have never really had issues with the CPU load or memory usage, typically no higher than 40% for CPU and 15GB for RAM but I applied the following two changes to two of our reps a few days ago and the CPU dropped right down (MAX is now 6%), the processor queue length also had a significant drop, almost non existent. Memory usage dropped a little but not as significant. So perhaps even 40% CPU usage was a sign something was wrong.
I'm currently making the same changes to more of our reps:
fsutil behavior set DisableDeleteNotify ReFS 1
REG ADD HKLM\System\CurrentControlSet\Control\FileSystem /v RefsEnableLargeWorkingSetTrim /t REG_DWORD /d 1
-
- Enthusiast
- Posts: 78
- Liked: 46 times
- Joined: Dec 10, 2019 3:59 pm
- Full Name: Ryan Walker
- Contact:
Re: Windows 2019, large REFS and deletes
Yes. Mixed bag.evilaedmin wrote: ↑Feb 22, 2020 3:35 am Many experience this and it's a real shame that there is no solution. It appears over time what used to be mostly-sequential reads become more like mostly-random reads.
Has anyone tried accelerating a Veeam ReFS repository with Flash?
It's fast, if your SSD tier is large enough. The issue is de-staging - particularly with what I'm suspecting more and more is a bug in Storage Spaces ReFS where a fast-clone that is attempted with the Performance Tier > 90% full, it literally crashes the volume and won't be recoverable until a reboot. This is true for both a repository AND a source, if you're using Hyper-V.
i.e. backups were working fine on a new system I'm running (1.2TiB SSD Mirror Tier / 50TiB parity tier via 10x8TB NL-SAS) but when it went to do a synthetic full, zoink, ded.
I'm currently setting my destage to 15% which means it should start to destage at anything over ~ 187GiB in the SSD Tier - this is the reason:
When under load, Storage Spaces MAP (mirror accelerated parity) doesn't not seem to destage until the load is not on it anymore. I can see this to be true when i start a backup job while it's de-staging, and suddenly the de-staging stops. Once i kill the backup job (or even file copy job to that volume) the de-stage starts again.
Which means, you have to ensure your incremental never fill the performance tier, because it will need to de-stage them before the Synthetic Full runs, or at least have at least 15% free to not crash the fast clone.
I'm still doing some checking on this but that's what i've found thus far. I have a few extra SSD i had intended on using for a replication repository - i might steal two from that to bump the SSD tier up to a larger size.
That all having been said: Microsoft recommends no smaller than 10% Mirror size to the Parity - which means technically I'm not running what I should be, it should be 5TiB of Performance, not 1.2TiB.
Really SSD isn't that expensive, but as this has to be Mirror (not parity) for the performance tier, it becomes a lot more expensive than if it were just a straight Raid-5 SSD.
I'd go the route of SOBR, and just make a raid-5 SSD 'performance' veeam tier for incremental - but you lose the Fast-Clone Synthetic Fulls by not having them on the same ReFS volume, from what I can tell.
-
- Enthusiast
- Posts: 78
- Liked: 46 times
- Joined: Dec 10, 2019 3:59 pm
- Full Name: Ryan Walker
- Contact:
Re: Windows 2019, large REFS and deletes
Did you exclude all of the proper directories?infused wrote: ↑May 25, 2020 4:46 am Just a note, you will notice (maybe) Windows Defender will have a big impact on writing files. I've noticed this on a couple of backup boxes. Removing Windows Defender seems to be the only way. Disabling real-time scanning does not seem to make a difference.
Uninstall-WindowsFeature -Name Windows-Defender
Code: Select all
Write-Host "Adding Recommended Veeam Anti-Virus Exclusions" -ForegroundColor Green
Write-Host "Excluding Recomended File Extensions" -ForegroundColor Yellow
Add-MpPreference -ExclusionExtension ".vmdk"
Add-MpPreference -ExclusionExtension ".flat"
Add-MpPreference -ExclusionExtension ".vbm"
Add-MpPreference -ExclusionExtension ".vbk"
Add-MpPreference -ExclusionExtension ".vib"
Add-MpPreference -ExclusionExtension ".vrb"
Add-MpPreference -ExclusionExtension ".vsb"
Add-MpPreference -ExclusionExtension ".vlb"
Add-MpPreference -ExclusionExtension ".erm"
Write-Host "Excluding Recommended Folders used by Veeam" -ForegroundColor Yellow
Add-MpPreference -ExclusionPath "C:\Program Files\Veeam"
Add-MpPreference -ExclusionPath "C:\Program Files (x86)\Veeam"
Add-MpPreference -ExclusionPath "C:\Program Files\Common Files\Veeam"
Add-MpPreference -ExclusionPath "C:\Program Files (x86)\Common Files\Veeam"
Add-MpPreference -ExclusionPath "C:\ProgramData\Veeam"
Add-MpPreference -ExclusionPath "C:\VeeamFLR"
Add-MpPreference -ExclusionPath "C:\Windows\Veeam"
#This may vary based on your repository path, rep path, etc
Add-MpPreference -ExclusionPath "D:\Backup"
Add-MpPreference -ExclusionPath "D:\VBRCatalog"
Add-MpPreference -ExclusionPath "V:\InstantRecoveryCache"
Add-MpPreference -ExclusionPath "V:\VeeamReplication"
-
- Enthusiast
- Posts: 57
- Liked: 5 times
- Joined: Jun 25, 2018 3:41 am
- Contact:
Re: Windows 2019, large REFS and deletes
My backup repository is thin provisioned iscsi all ssd with trim enabled on the backend. Do we still need to disable trim in Windows? I did my first full fast clone of a single 15TB VM and it did it in one hour exactly. Not sure if that is good or bad in regards to the speed of the clone but if it's not great, should we still enable these tweaks?Andrew@MSFT wrote: ↑Mar 18, 2020 12:43 am Yes. The registry tweaks are still recommended.
• Ensure Trim is disabled• Set RefsEnableLargeWorkingSetTrim = 1Code: Select all
fsutil behavior set DisableDeleteNotify ReFS 1
Code: Select all
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem Value Name: RefsEnableLargeWorkingSetTrim Value Type: REG_DWORD Value Data: 1
-
- Chief Product Officer
- Posts: 31806
- Liked: 7299 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Windows 2019, large REFS and deletes
Yes, you should.
-
- Expert
- Posts: 186
- Liked: 22 times
- Joined: Mar 13, 2019 2:30 pm
- Full Name: Alabaster McJenkins
- Contact:
Re: Windows 2019, large REFS and deletes
As of today, with all the refs 2019 issues.... I've got several sites with 2016 refs and they all work flawlessly so far.
Is it recommended to go to 2019 on any "new" set ups? Most of me wants to just keep standing up 2016 as I think the extended support is over 2027 so we have plenty of time.
This site is only say 20TB full, not huge. Will be doing weekly synthetic fulls refs for offloading to object storage etc.
If it is for sure safe I can try 2019 on this new setup, but otherwise I will just stick with ole trusty 2016.
Is it recommended to go to 2019 on any "new" set ups? Most of me wants to just keep standing up 2016 as I think the extended support is over 2027 so we have plenty of time.
This site is only say 20TB full, not huge. Will be doing weekly synthetic fulls refs for offloading to object storage etc.
If it is for sure safe I can try 2019 on this new setup, but otherwise I will just stick with ole trusty 2016.
-
- Chief Product Officer
- Posts: 31806
- Liked: 7299 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Windows 2019, large REFS and deletes
Simply put, 2016 is proven to be rock solid.
2019 cannot be considered proven yet, because the final fixes are fairly recent. But from technical perspective, there are not outstanding issues with 2019 that we're waiting for Microsoft to fix. So, it just needs more time now before I can confidently state that it's proven too.
2019 cannot be considered proven yet, because the final fixes are fairly recent. But from technical perspective, there are not outstanding issues with 2019 that we're waiting for Microsoft to fix. So, it just needs more time now before I can confidently state that it's proven too.
-
- Enthusiast
- Posts: 31
- Liked: 3 times
- Joined: May 20, 2016 7:05 am
- Contact:
Re: Windows 2019, large REFS and deletes
Hi!Cullan wrote: ↑May 28, 2020 3:00 am We are running the following for most of our reps.
OS: Server 2019 '1809' and the latest 2020-05 CU.
Hardware: Lenovo SR590
Disks: OS on 120GB SSD mirror kit, backups on 150TB ReFS volume with 64k block size (14 x 14TB disks in RAID6 512K RAID striping)
CPU: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz, 2095 Mhz, 8 Core(s), 16 Logical Processor(s)
RAM: 64GB
NIC: 10G fibre SFP
Veeam: Veeam 10 - 10.0.0.4461 P1
We have a similar configuration, 2 completely new Veeam Servers (Veeam role not installed yet). I'm insecure about the Windows Server Version: Windows Server 1809 (actually installed) or Windows Server 1909 Core
So you did not had any issues with ReFS and Windows Server 2019 '1809'? In the forum there are reports that Windows Server 2019 prior '1903' can cause several problems.
best regards,
Juergen
-
- Veeam Legend
- Posts: 1203
- Liked: 417 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: Windows 2019, large REFS and deletes
With the latest ReFS Patches windows 2019 is quite unable and stable. 1909 should have some performance optimizations but our 400 TB repo on 2019 does not run much worse than our 600 TB 1909 repo...
-
- Enthusiast
- Posts: 51
- Liked: 3 times
- Joined: May 07, 2019 12:22 am
- Full Name: Glenn
- Contact:
Re: Windows 2019, large REFS and deletes
Did you mean "usable and stable"?With the latest ReFS Patches windows 2019 is quite unable and stable.
-
- Veeam Legend
- Posts: 1203
- Liked: 417 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: Windows 2019, large REFS and deletes
Yep, thats what i meant
-
- Chief Product Officer
- Posts: 31806
- Liked: 7299 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Windows 2019, large REFS and deletes
This typo is absolutely epic!
-
- Enthusiast
- Posts: 51
- Liked: 3 times
- Joined: May 07, 2019 12:22 am
- Full Name: Glenn
- Contact:
Re: Windows 2019, large REFS and deletes
What is everyone using then for their ReFS Repository Servers .... 2019 LTSC or the Semi-Annual build like 1909?
-
- Veeam Legend
- Posts: 1203
- Liked: 417 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: Windows 2019, large REFS and deletes
@AuGL how many TB of backups do you have? I think for most customers the difference is not "feelable", we only use both to have different ReFS drivers for backup and backup copy locations - if there is a major bug we might be lucky and it only affects one repo...
-
- Enthusiast
- Posts: 51
- Liked: 3 times
- Joined: May 07, 2019 12:22 am
- Full Name: Glenn
- Contact:
Re: Windows 2019, large REFS and deletes
Around 90TB at the moment ... but that will grow to double that over the next 12 months.
-
- Service Provider
- Posts: 372
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: Windows 2019, large REFS and deletes
I have 3 relatively large repos. 65T, 232T, 291T, all on WS2019 1809 LTSC. With RefsEnableLargeWorkingSetTrim and patches, it's... fine. Before March patches it occasionally slowed down (but never crashed) with very large (~30T) synthetics - clearing system working set always made it resume. File system fragmentation slows things down a bit but nothing too dramatic.
-
- Enthusiast
- Posts: 51
- Liked: 3 times
- Joined: May 07, 2019 12:22 am
- Full Name: Glenn
- Contact:
Re: Windows 2019, large REFS and deletes
Thanks for replying and sharing some info.
What is this?clearing system working set always made it resume
-
- Service Provider
- Posts: 372
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: Windows 2019, large REFS and deletes
ReFS driver leaks (not exactly the correct term in this context but close enough) some memory and with very large synthetics (well, mostly used to leak), it seems to drown in it's own memory management. If you go back in this thread, you'll find a tool that regularly tells the system (eg kernel, not user-mode processes) to reduce working set (effectively to drop some memory usage, flush caches, page to disk...). This causes ReFS to flush a lot of memory and performance goes up again. You can do this manually from Sysinternals RAMMap tool as well.
I haven't had to do this since spring.
I haven't had to do this since spring.
-
- Enthusiast
- Posts: 51
- Liked: 3 times
- Joined: May 07, 2019 12:22 am
- Full Name: Glenn
- Contact:
Re: Windows 2019, large REFS and deletes
Thanks, good to hear.
-
- Service Provider
- Posts: 129
- Liked: 59 times
- Joined: Feb 06, 2018 10:08 am
- Full Name: Steve
- Contact:
Re: Windows 2019, large REFS and deletes
What about 2004? It'd be interesting to hear how ReFS performs in the newest build compared to 1607, 1809 and say, 1909, or at least what changes have occurred in the ReFS stack
-
- Veeam Legend
- Posts: 1203
- Liked: 417 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: Windows 2019, large REFS and deletes
I did not yet ask microsoft. Will do now...
-
- Enthusiast
- Posts: 46
- Liked: 12 times
- Joined: Apr 10, 2018 2:24 pm
- Full Name: Peter Camps
- Contact:
Re: Windows 2019, large REFS and deletes
It looks like we are back to square 1, at this moment our 2019 LTSC repo is crawling to the finishline during backups.
We are using an HPE Apollo 4200 with 1 200 TB volume as repo for our Veeam VBR 10a.
We have had a lot of trouble in the past which seemed to have been solved after ReFS patch KB 4531331.
After setting the task limit on that repo to 30 (32 core cpu) and installing the patch backups were running normal again.
Our backups used to be done at around 15:00, but after this they were finished between 06:00 - 07:00. We almost cracked open a bottle of champagne.
Lately backups started to run a bit longer, nothing alarming.
But to days ago suddenly backups wer slowing down a lot more, they never finished before the afternoon.
No jobs were added or anything else (except Windows updates), and again we were in trouble.
What we see is that when jobs start the I/O to the repo is between 1 - 6 GB/s for the combined jobs. But when one of the jobs is ready and starts the merge, the I/O for the other jobs just drops to a few MB/s and sometimes KB/s. When the merges are finished the I/O returns to normal values.
But because we have multiple jobs running from 20:30 to 03:00, the later the job start the longer it takes for the job to finish.
We see some jobs waiting for infrastructue availability more than 10 hours.
We have actually no idea why this is suddenly happening. As i said before, we haven't changed anything on this repo.
Just to make sure all other repos (windows 2016) are performing like clockwork.
We are using an HPE Apollo 4200 with 1 200 TB volume as repo for our Veeam VBR 10a.
We have had a lot of trouble in the past which seemed to have been solved after ReFS patch KB 4531331.
After setting the task limit on that repo to 30 (32 core cpu) and installing the patch backups were running normal again.
Our backups used to be done at around 15:00, but after this they were finished between 06:00 - 07:00. We almost cracked open a bottle of champagne.
Lately backups started to run a bit longer, nothing alarming.
But to days ago suddenly backups wer slowing down a lot more, they never finished before the afternoon.
No jobs were added or anything else (except Windows updates), and again we were in trouble.
What we see is that when jobs start the I/O to the repo is between 1 - 6 GB/s for the combined jobs. But when one of the jobs is ready and starts the merge, the I/O for the other jobs just drops to a few MB/s and sometimes KB/s. When the merges are finished the I/O returns to normal values.
But because we have multiple jobs running from 20:30 to 03:00, the later the job start the longer it takes for the job to finish.
We see some jobs waiting for infrastructue availability more than 10 hours.
We have actually no idea why this is suddenly happening. As i said before, we haven't changed anything on this repo.
Just to make sure all other repos (windows 2016) are performing like clockwork.
-
- Enthusiast
- Posts: 46
- Liked: 12 times
- Joined: Apr 10, 2018 2:24 pm
- Full Name: Peter Camps
- Contact:
Re: Windows 2019, large REFS and deletes
We have opened a new Case # 04331174. Just noticed that the ReFS.sys is newer than the one in KB 4531331.
The version is now 10.0.17763.1192 (date modified 7/14/2020).
Could this be the issue?
The version is now 10.0.17763.1192 (date modified 7/14/2020).
Could this be the issue?
-
- Novice
- Posts: 3
- Liked: never
- Joined: Feb 17, 2020 7:32 am
Re: Windows 2019, large REFS and deletes
Not sure if it helps, but I've just updated our Windows 2019 Server with REFS and refs.sys version is now 10.0.17763.1369.
Windows CU 2020.08
Windows CU 2020.08
-
- Service Provider
- Posts: 372
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: Windows 2019, large REFS and deletes
Are you sure that you have RefsEnableLargeWorkingSetTrim registry value set? I was just testing if this flag caused some other problems and after removing it and restarting, I immediately hit similar problems during merge.
-
- Influencer
- Posts: 14
- Liked: 2 times
- Joined: Feb 25, 2015 1:49 pm
- Full Name: Peter Stam
- Contact:
Re: Windows 2019, large REFS and deletes
Yes it is set DonZoomik (colleague of PeterC).
-
- Veeam Legend
- Posts: 1203
- Liked: 417 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: Windows 2019, large REFS and deletes
@peterc Did you install update 10a?
For us after installing update 10a our synthetics for some jobs started to take 10+ hours instead of < 1 hour! ReFS does not have any "strange" messages in event log, still speed is MUCH slower now.
@GOSTEV:
Case #04337327
For us after installing update 10a our synthetics for some jobs started to take 10+ hours instead of < 1 hour! ReFS does not have any "strange" messages in event log, still speed is MUCH slower now.
@GOSTEV:
Case #04337327
-
- Novice
- Posts: 7
- Liked: never
- Joined: Jan 31, 2017 1:50 pm
- Full Name: Koen Teugels
- Contact:
Re: Windows 2019, large REFS and deletes
So for a new installation veeam 10a with 4 physical proxy servers 200 TB source data, do you recommend windows 2016 in stead of 2019
-
- Enthusiast
- Posts: 46
- Liked: 12 times
- Joined: Apr 10, 2018 2:24 pm
- Full Name: Peter Camps
- Contact:
Re: Windows 2019, large REFS and deletes
@DonZoomik, yes the RefsEnableLargeWorkingSetTrim is still set to 1.
@mkretzer, yes we have installed update 10a, we had a lot of trouble with the management console on our VBR after update 10. It was tremendously slow, almost unworkable. Have had a case for this for several months and it was mostly fixed in update 10a.
I cannot really tell you if all of this happened after update 10a. We also opened a case with MS (Case 120081425000237) for this, so hope that Veeam or MS can find the cause.
@koenteugels, at this moment i would definitely choose for Server 2016. We have a lot of (virtual) 2016 repos that work without any problem. Got 1 (physical) 2019 repo and it almost has been one (big) nightmare. We have no experience with the 2019 Core versions (1909, 2004), so can not advice on these versions.
@mkretzer, yes we have installed update 10a, we had a lot of trouble with the management console on our VBR after update 10. It was tremendously slow, almost unworkable. Have had a case for this for several months and it was mostly fixed in update 10a.
I cannot really tell you if all of this happened after update 10a. We also opened a case with MS (Case 120081425000237) for this, so hope that Veeam or MS can find the cause.
@koenteugels, at this moment i would definitely choose for Server 2016. We have a lot of (virtual) 2016 repos that work without any problem. Got 1 (physical) 2019 repo and it almost has been one (big) nightmare. We have no experience with the 2019 Core versions (1909, 2004), so can not advice on these versions.
Who is online
Users browsing this forum: mbrzezinski and 179 guests