ReFS state post September 2018 Windows Updates

[Gostev]

I am starting the new thread for ReFS feedback now that all the teething issues with its stability and memory consumption seem to have been resolved as of refs.sys driver version 2457.

We want to keep this discussion consumable and on-topic, so before you post your feedback or issues in this thread, please be sure your backup repository meets the following requirements:

Operating System
Using the following OS versions will ensure you're not running into any known ReFS issues:
• Windows Server 2019 (not officially supported until Update 4, so use it for lab environments only please).
• Windows Server 2016 patched to at least September 2018 updates (KB4343884 or any later one, since Windows Updates are cumulative).
• Windows 10 Pro for Workstations
You can double-check that you're on the right patch level by verifying that your refs.sys driver version is 2457 or later.

Hardware
Make sure your backup repository server meets minimum system requirements, particularly around:
• CPU: we recommend allocating at least 1 core per each concurrent backup proxy task, and at least 1 core for each two concurrent repository tasks.
• RAM: if the backup repository server is running multiple Veeam roles, please add up memory requirements of each individual role.
• Storage: we recommend formatting the ReFS volume with 64KB block size.

3rd Party Software
Uninstall the following 3rd party software that have been reported to cause ReFS stability and/or performance issues:
• 3rd party antiviruses (but not Windows Defender)
• Microsoft Configuration Manager Client
If possible at all, we recommend that you start troubleshooting ReFS issues from performing a clean installation of an operating system from the original Windows installation ISO from MSDN, to ensure no bloatware is installed along from vendor-provided installation media. Please create the dedicated topics if you'd like to discuss some specific incompatibilities with the specific 3rd party software, and we will update this master post with results of the corresponding discussions.

For more information on what we know so far, I am reposting the snippet of my "Word" section of the Veeam forum digest from a few weeks ago:

We have now completed the stress testing of ReFS driver version 2457 (released 1 month ago as a part of KB4343884). As a reminder, this was a milestone update that brought the backport of ReFS driver memory management fixes from Windows Server 2019 branch. So we decided to really put this driver through its paces in our stress testing lab to validate those changes. Just in case you're wondering why we tested the original version, and not the most current from the latest Windows update - this is because this sort of stability testing takes a few weeks. But the future driver versions of course include all these changes too, anyway. And long story short, I'm happy to report that ReFS remained stable no matter what challenges we threw at it.

The last stress test was particularly impressive, because it was done on a 40TB ReFS repository with just 8GB RAM in the server – which is our minimum system requirements and way below the current recommendation of 1GB per 1TB. Backup job configuration was also the toughest for ReFS to handle with over a hundred VMs, per-VM backup file chains and weekly synthetic fulls enabled for – meaning, the retention process had to delete hundreds of backup files with the total virtual size of 60TB at once - configuration which pretty much guaranteed server lockups before. But the latest ReFS driver chew through this like a piece of cake, with no spikes for either CPU or RAM and no system freezes – very impressive with just 8GB physical RAM. In fact, top RAM usage was just 5.4GB (metafile maximums were 4.5 GB total and 2.1GB active), while CPU load of the 4 vCPU VM was hovering at around 10%.

Does it mean this is finally the end of the ReFS troubles, at least at a wide scale? Anecdotally, we're already starting to receive such confirmations anyway. For example, one of our solution architects has two identical ReFS repositories in his lab with vastly different behavior. The one on Windows 10 Pro for Workstations (which is based on a newer Windows build) has been rock sold - so much that he was quite tempted to suggest customers use Windows 10 instead of Windows 2016! But an identical setup with Windows 2016 was still locking up every now and then up until KB4343884 was installed. So, I'm very optimistic about it, even if of course only the field experience will be able to confirm the resolution for sure. But until then, I recommend "business as usual" particularly in regards to physical RAM on your backup repository servers.

Thanks!

[Mgamerz]

Maybe pin this thread so it's at the top?

[Iain_Green]

Running Windows Server 2016 OS Build 14393.2551 fully patched.

Yet refs.sys shows the following:

last modified 30/08/2018
File Version 10.0.14393.2515
Product version 10.0.14393.2515

If I try and run the KB4343884 found from https://www.catalog.update.microsoft.co ... =KB4343884 I get "The update is not applicable to your computer"?

How do I get to REFS version .2457?

[opg70]

.2515 is more recent and includes all the specified fixes, you don't need .2457

[Iain_Green]

@opg70 thanks for confirming.

[dimaslan]

I am checking the refs.sys properties and I see nothing in the details tab. Do I need to change something first?

[Gostev]

Wow! Something is seriously messed up with your OS then... I've no explanation to this.

[akornow]

Hi there

I´m new on forum, but every monday morning i take few minutes to read with very attention gostev news.
They are exactly what we expect from techincal news.Good job Gostev, Make me proud to be veeam partner.

Now back to real life... I wish to ask if someone faced after sudden windows server 2016 restart a REFS volume being marked as RAW (Is an 10TB VHDX attached to Veeam VM and used as backup repository) and if there is any way to bring it back to life.
As i said it is backup repository for long term backups so... there is no backup for backup.
We oppened a ticket to MS, wait, wait, wait a little bit more something like 28 Hours and after that they simply tell us in 5 minute call that there is nothing that they can do except restore from backup or try to find some third party rescue tool...

Thanks in advance

Alessandro

[Gostev]

Thank you for your kind words, Alessandro!

Yes, I heard about these types of complete volume loss situations from the ReFS team, they saw it in support with non-HCL repository server hardware (specifically, RAID controllers without battery-backed write cache). The issue is caused when certain critical ReFS metadata that is already in flight is lost due to a sudden restart, before it actually lands to disks. Enterprise-grade RAID controllers take care of this by automatically committing any pending/unconfirmed writes sitting in BBWC to disk once the server restarts.

[anton]

We've intalled KB4462928 on three ReFS based repository servers.
For two of them this didn't give any issues; but on our largest repository we noticed that everything slowed down after installation.

The response times on the ReFS volume went really high (>1000 ms)

After we removed the KB from this repository server the performance was OK again.
With KB4462928 the refs driver was on 2515, currently we have version 2363 which is stable.

We are using Windows Storage spaces (JBOD), the virtual disk has a Mirrored layout and thin provisioning.

Anyone else experiencing this after installing KB4462928?

[Gostev]

Version 2515 is from mid-September Windows update, so it's been around for a while and has no known issues with it.
However, not too many Veeam users use Storage Spaces for backup repository, so this can be the culprit in your case.

[Frenchyaz]

Hi everyone,

I am a new Veeam customer and of course during the installation Veeam recommends Refs 64K... It is the first server that we installed with Refs so I was a little bit anxious as to how it may mess up with the backups.

So far, I can't say if it has been good or bad as we haven't started yet production backup on it.

I just wanted to put some information on our configuration in case somebody has one similar and ran into issues.

ESXi host 6.7U1 in standalone server which contains only one VM

VM Guest: Windows server 2016 with up to date patches from Windows Update
Veeam Backup and Replication 9.5U3
SQL Server 2016 SP2 CU2 standard
64GB memory reserved
16vCPU
System OS on 1st Paravirtual controller, VMDK 4K
SQL on 2nd Paravirtual controller, VMDK 64K
Pass through HBA 16GB (*2)
RDM Disks (3 * 64TB volume formatted with Refs)

Preliminary tests show a sustain backup speed of 800MB/s up to 1GB, seems like our slowest link is our SAN...

We haven't tested multiple backups yet however we should pretty soon after we set up our backup policy. Coming from Netbackup, it is a bit of a learning curve however speedwise, Veeam blows away Netbackup.

So if you are using Veeam in this configuration and have issues with Refs, feel free to post here in order to see what can be done in order to avoid pitfalls.
I'll continue to update here after further backups, hopefully soon...

[olavl]

anyone with experience running Veeam + ReFS with deduplication?

[micha2k6]

Hi there

I´m new on forum, but every monday morning i take few minutes to read with very attention gostev news.
They are exactly what we expect from techincal news.Good job Gostev, Make me proud to be veeam partner.

Now back to real life... I wish to ask if someone faced after sudden windows server 2016 restart a REFS volume being marked as RAW (Is an 10TB VHDX attached to Veeam VM and used as backup repository) and if there is any way to bring it back to life.
As i said it is backup repository for long term backups so... there is no backup for backup.
We oppened a ticket to MS, wait, wait, wait a little bit more something like 28 Hours and after that they simply tell us in 5 minute call that there is nothing that they can do except restore from backup or try to find some third party rescue tool...

Thanks in advance

Alessandro

Hi Allessandro,

This happened to me twice so far. My Veeam Jobs failed with the error: "Synthetic full backup creation failed Error: The volume repair was not successful. Failed to create or open file [....vbk]." and after a reboot the volume is gone and marked as RAW.
We tried to resolve the issue with Microsoft and Dell (as the repository is on a Dell physical machine) but both were more or less pointing to the other one. We haven't been able to recover anything from the disks and we ended up updating all firmware versions and drivers to the latest ones and fully reinitialized the raid volume (not quick initialize) and afterwards did a full slow formatting of the whole disk. Now we're hoping for the best. Both incidents happened on our biggest repositories (~80TB) and they both were only roughly 50% filled, both server 2016 on identical HW configuration.
Repository servers are Dell R730 (2x Xeon E5-2630, 96GB RAM, 80TB raid volume on a Perc H730P Mini).
The second time this happened for me was just beginning of November, exactly 2 days after I installed the 2018-10 and 2018-09 Cumulative Updates but we still have AV from McAfee installed on the servers.

Yes, I heard about these types of complete volume loss situations from the ReFS team, they saw it in support with non-HCL repository server hardware (specifically, RAID controllers without battery-backed write cache). The issue is caused when certain critical ReFS metadata that is already in flight is lost due to a sudden restart, before it actually lands to disks. Enterprise-grade RAID controllers take care of this by automatically committing any pending/unconfirmed writes sitting in BBWC to disk once the server restarts.

After reading this I had a closer look at the event log of our repository server. There was a clean reboot after the patches got installed and ~ 2 hours later I have the first entries from ReFS in the event log. First checksum errors that the system was able to correct and right after that checksum errors on the same file that could not be corrected.

Michael

[l0stb@ackup]

3rd Party Software
Uninstall the following 3rd party software that have been reported to cause ReFS stability and/or performance issues:
• Microsoft Configuration Manager Client

Do we have more information about this issue, symptoms and/or workarounds?
We have two Veeam servers with ReFS repositories and use SCCM to patch all our Windows servers, we've been holding off on installing the agent on the Veeam servers and patching them manually so far because of this note. Thanks

[Gostev]

Do we have more information about this issue, symptoms and/or workarounds?
We have two Veeam servers with ReFS repositories and use SCCM to patch all our Windows servers, we've been holding off on installing the agent on the Veeam servers and patching them manually so far because of this note. Thanks

Not much really, we just had a few customers report this issue on the last few pages of the original thread.

Please do let us know your experience if you decide to give it a try.

[Seve CH]

Hello,

I confirm that since we removed the SCCM agent two months ago, the server works fine.
400TB in ReFS repos, 75GB RAM used (MS-SQL Server + Veeam Backup + Veeam One + Windows usage)

We do not plan to open a support case at Microsoft nor risking to install the SCCM agent. It is just one server so for us, it is not worth all the inconvenience.

Regards

[ctg49]

I wanted to give an update of some testing I've been doing with Server 2019. I suspect this won't come as a surprise to anyone using the Core version of 2016 and leveraging Dedupe over ReFS, but ReFS in 2019 is performing well with dedupe. I'm getting virtually identical numbers as far as percentages go, so don't expect some big change there (I presume it's the same engine that NTFS has historically leveraged). The actual process is *much* faster though, I was copying ~2.5TB of data (mixed large and small files, ~600k total count) into a VM at 1Gb speeds and dedupe was outpacing it, if I were to estimate a time to dedupe the entire volume, it probably would have been two hours or so. That would have probably taken 8h+ before, at minimum.

I look forward to upgrading our VEEAM repositories to leverage >64TB volume sizes on dedupe volumes, as well as actually being able to leverage the benefits of ReFS.

Side-note, does VEEAM actually support being installed/running on Server 2019? I wouldn't expect there to be any issues, but one never knows.

[Gostev]

This is answered in the first post of this topic official support for Server 2019 is coming in Update 4.

[ctg49]

Should have read more. Between this and 6.7U1 support, U4's gonna be big for us then. Thanks!

[NightBird]

And 6.7U1 is mandatory because of sesparse bug...

[DonZoomik]

The actual process is *much* faster though, I was copying ~2.5TB of data (mixed large and small files, ~600k total count) into a VM at 1Gb speeds and dedupe was outpacing it, if I were to estimate a time to dedupe the entire volume, it probably would have been two hours or so. That would have probably taken 8h+ before, at minimum.

It's actually quite the same as with NTFS. WS2012R2 could do 75-150MB/s (depending on dedupe ration) on midrange Xeon. WS2016 could do 75-150MB/s per thread.

[nmdange]

I believe you can't do dedupe and fastclone on the same ReFS volume. With that in mind, I'd almost always choose Fastclone over Dedupe.

[SBarrett847]

I Recently deployed a 72tb storage server running Server2019 w/ReFS for my Veeam backups, the server CPU is spiking and leads to eventual inability to access the server. I've seen the ReFS issue first hand elsewhere and this has all the hallmarks.

This server does only have 32gb installed, and I've read the recommendation is 1GB Ram per TB of Storage. I've also read in this thread that it can work with way less. I need some way to conclusively prove that this the known ReFS issue and that more RAM is the answer, before I go ask for it. My problem is that by the time I notice there is an issue the server has become inaccessible. Looking through PRTG I can see the CPU gradually climb to 100% and stay there, before PRTG loses the ability to get info from the server. I've no record of what process is causing the CPU spike though. The Event logs show nothing useful either.

Any Ideas on where I go from here?

[jorgedlcruz]

Hello SBarrett847,
I would recommend looking at the Windows Counters, look for the ReFS ones, and because you have PRTG you can easy monitor the activity of ReFS - https://www.paessler.com/manuals/prtg/p ... tom_sensor

I hope this helps

[SBarrett847]

Aha, I didn't know about those counters - perfect. Any ideas as to which are the most important to monitor? There is quite a few of them.

[SBarrett847]

Cant edit/add to the above post. This might be use full for anybody else using PRTG (even the free one). Create a new PerfCounter Custom Sensor in PRTG and paste in the below (edit for your own Drive letter) and it will pull all the Refs metrics into PRTG. Now I just have work out what they all mean

Code: Select all

\ReFS(V:)\Allocation of Data Clusters on Fast Tier/sec::Sec
\ReFS(V:)\Allocation of Data Clusters on Slow Tier/sec::sec
\ReFS(V:)\Allocation of Metadata Clusters on Fast Tier/sec::Sec
\ReFS(V:)\Allocation of Metadata Clusters on Slow Tier/sec::Sec
\ReFS(V:)\Checkpoint latency (100 ns)::100ns
\ReFS(V:)\Checkpoints/sec::Sec
\ReFS(V:)\Compacted Container Fill Ratio (%)::%
\ReFS(V:)\Compaction Failure Count
\ReFS(V:)\Compaction read latency (100 ns)::100ns
\ReFS(V:)\Compaction write latency (100 ns)::100ns
\ReFS(V:)\Compactions failed due to ineligible container
\ReFS(V:)\Compactions failed due to max fragmentation
\ReFS(V:)\Container Destages From Fast Tier/sec::Sec
\ReFS(V:)\Container Destages From Slow Tier/sec::Sec
\ReFS(V:)\Container Move Failure Count
\ReFS(V:)\Container Move Retry Count
\ReFS(V:)\Container moves failed due to ineligible container
\ReFS(V:)\Current Fast Tier Data Fill Percentage::%
\ReFS(V:)\Current Fast Tier Metadata Fill Percentage::%
\ReFS(V:)\Current Slow Tier Data Fill Percentage::%
\ReFS(V:)\Current Slow Tier Metadata Fill Percentage::%
\ReFS(V:)\Data Compactions/sec::Sec
\ReFS(V:)\Data In Place Write Clusters/sec::Sec
\ReFS(V:)\Delete Queue entries
\ReFS(V:)\Dirty metadata pages
\ReFS(V:)\Dirty table list entries
\ReFS(V:)\Fast tier destage read latency (100 ns)::100ns
\ReFS(V:)\Fast tier destage write latency (100 ns)::100ns
\ReFS(V:)\Fast Tier Destaged Container Fill Ratio (%)::%
\ReFS(V:)\Log fill percentage::%
\ReFS(V:)\Log writes/sec::Sec
\ReFS(V:)\Slow tier destage read latency (100 ns)::100ns
\ReFS(V:)\Slow tier destage write latency (100 ns)::100ns
\ReFS(V:)\Slow Tier Destaged Container Fill Ratio (%)::%
\ReFS(V:)\Total Allocation of Clusters/sec::Sec
\ReFS(V:)\Tree update latency (100 ns)::100ns
\ReFS(V:)\Tree updates/sec::Sec
\ReFS(V:)\Trim latency (100 ns)::100ns

[jaybios]

Hi,

I'm trying to find the right thread for this as it seems as though the problem I'm having is potentially ReFS related.

I get a similar problem to that others are seeing with ReFS, I'm on Server 2016, ReFS store on an iSCSI SAN with 4k block size as my backup repository, doing a backup of a 10TB VM. Server is up to date, SCCM client has been removed.

If i leave the "Defragment and compact" setting enabled, my server locks up completely at 55% on compacting full backup file [fast clone] consistently. I've disabled this and now I'm currently at 99% on transforming full backup, so it looks as though it's working fine with compacting turned off. I had this problem last year, had all sorts of headaches with it, but eventually i did something and the problem stopped (unfortunately i wasn't savvy enough to have made a list of the changes i was making...after the 20th time rebooting a HP server one gets frustrated...)

Do others see this problem occurring at this point? Or are the ReFS problems more related to the actual data transfer stage of backup?

[Gostev]

Actually, neither. The only scenario that ReFS ever had issues with was mass deletion of backup files. Not even in the worst days of ReFS there was any other scenario that impacted ReFS stability.

The only thing that jumps on me in your case is that you used 4KB block size for ReFS, as opposed to the recommended 64KB block size.

[mkretzer]

@gostev: Sorry that is not entirely true. We also had issues when doing active fulls. The system had major issues getting the data to disk, much slower compared to NTFS. Still, i agree these issues are completely gone now and only for us with W2019 there seems to be the delete issue again.