REFS issues (server lockups, high CPU, high RAM)

[rverhe]

Have the same issues with update: KB4103720 , REFS Fast clone merge time doubled...
Someone already have an solution for this?

[Gostev]

KB4103720 does not list any changes which could affect ReFS performance even remotely. Can you confirm it is indeed caused by this update by temporarily uninstalling one?

[scis]

I regretfully applied KB103720, KB4132216, & KB 4103723 to our disk backup servers, and I too am having issues with Veeam Fast Clone after applying these windows updates on 5/21/2018. I'm using ReFS on our primary disk backup server with 208 TB in our storage pool. The other disk backup servers are using NTFS in a SOBR configuration, and i only have a single small backup job with 4 objects and 1.4 TB of data to these servers. I'm not really noticing an issue with the other disk servers using NTFS, but they likely are using the processor as much either since they're setup with NTFS.

Exchange backup jobs for Synthetics backups.
Backup TypeStart TimeDurationProcessing Rate (MB/Sec)Data Size (GB)Transferred (GB)Total Backup Size (GB)
Synthetic Full 5/5/2018 4:00 PM 00:18:53 402.28 5131.00 56.12 6172.65
Synthetic Full 5/12/2018 4:00 PM 00:19:14 348.61 5131.00 51.18 6228.79
Synthetic Full 5/19/2018 4:00 PM 00:18:41 409.74 5131.00 62.69 6296.45
Synthetic Full 5/26/2018 4:00 PM 04:13:03 338.37 5131.00 50.33 6322.44

Another example with 61 objects and 17TB of data....
Synthetic Full 5/14/2018 10:00 PM 00:41:17 375.08 17069.82 211.66 28112.28
Synthetic Full 5/21/2018 10:00 PM 06:22:42 335.23 17085.82 210.25 28413.62

Laserfiche backup jobs
Synthetic Full 5/4/2018 7:00 PM 00:39:35 318.60 12667.40 34.92 25655.09
Synthetic Full 5/11/2018 7:00 PM 00:43:48 369.57 10772.00 45.05 25763.46
Synthetic Full 5/18/2018 7:00 PM 00:47:23 222.24 10772.00 25.36 25485.99
Synthetic Full 5/25/2018 7:00 PM 02 day(s) 15:59:21 330.44 10822.00 32.88 0.00

One large file server job....
Synthetic Full 5/5/2018 4:45 PM 00:27:50 123.38 7787.80 5.23 23235.91
Synthetic Full 5/12/2018 4:45 PM 00:26:45 128.98 7787.80 5.05 23250.56
Synthetic Full 5/19/2018 4:45 PM 00:25:40 132.72 7787.80 6.77 23421.17
Synthetic Full 5/26/2018 4:45 PM 01 day(s) 06:41:08 136.91 7787.80 5.49 23153.92

Not sure what i will do quite yet, but I have a job that started on 5/27/2018 @ 9PM that is now 39 hours and 29 minutes into the Synthetic Full backup sitting at 99% job progress, but in the details says "Creating synthetic full backup (67% done) [fast clone]

I suspect it might have something to do with the processor patch for Meltdown / Spectre, so I may investigate disabling that option through a registry change.

[NightBird]

Same problem here... I just upgraded a small setup to test.

From 1mn30s to 9mn10s... for synthetic full

ReFS driver version
10.0.14393.2273 04/28/2018

[JimmyO]

OMG - never ending story? I´m just migrating my entire environmet from NTFS to ReFS for the second time. A 3-6-months job and now this!

I really hope Veeam can put some pressure on MS to fix this asap..

[EzE]

I've uninstalled KB4103720 and have confirmed ReFS.sys is back to version 2248. I'm awaiting a Merge operation to see speeds, but scis and NightBird appear to confirm the issue starts when ReFS is updated to 10.0.14393.2273. I have a ticket open with Veeam, but someone with a test environment might be in a better position to go back and forth. I could open a ticket with Microsoft, but something tells me I won't get anywhere.

[Gostev]

We have the test environment, no worries! And already confirmed there's indeed a performance drop of over 3x after installing May updates in our lab.

And after looking at the ReFS drivers included in May updates to the previous (stable) version from February, it is apparent that Microsoft included the new (and actually different) versions of the driver in both packages without noting anything in the release notes of either update...

KB4093120 from April 17, 2018 (OS Build 14393.2214)
Refs.sys size 1,709,400 version 10.0.14393.2097 from 12-Feb-18 22:37

KB4103723 from May 8, 2018 (OS Build 14393.2248)
Refs.sys size 1,709,400 version 10.0.14393.2248 from 28-Apr-18 5:25

KB4103720 from May 17, 2018 (OS Build 14393.2273)
Refs.sys size 1,709,912 version 10.0.14393.2273 from 28-Apr-18 6:44

I am translating all of our findings in real-time directly into the ReFS dev team, they are online.

The quick fix for now is obviously to simply uninstall May updates in order to revert to the previous version of the driver.

[emapola]

I confirm, w2016 build 1803, local REFS ~50TB, same performance issues. Also GFS to tape hang on error (before build upgrade works fine).

[Gostev]

Another test we did today is dropping ReFS.sys from February (10.0.14393.2097) into the lab with May updates installed. This made block cloning performance come back to almost normal (performance drop from May updates was over 3x, and using the old driver improved performance back by 2.5x).

So, it does look like the issue is caused by some "under the carpet" changes to ReFS.sys which were not even documented in the release notes...

[SteveBLGS]

Removing KB4103720 resolved our performance issues.

refs.sys version
-before removing patch: 10.0.14393.2273
-after removing patch: 10.0.14393.2248

We have a 14 TB VAL job that typically takes around 1 hour to perform a synthetic full using fast clone. After this patch was applied it took nearly 100 hours. We also have a 2 TB VAL job that typically performs a synthetic full backup using fast clone in < 10 minutes. After the path, it was taking over 1 hour.

Steve

[mkretzer]

Can we rename this thread to the original title please?

Serious: Has someone already talked to MS about the reason why they did this? We were just re-considering REFS and were doing some small tests...

Markus

[Gostev]

Yes, as noted above I've been in touch with them. They're investigating the root cause with the servicing team now.

[scis]

I'd posted last week about regretfully applying KB103720, KB4132216, & KB 4103723 to our disk backup servers, and then experiencing major performance issues with Veeam Fast Cloning ( Veeam Synthetic Fulls ) and looking at options for disabling processor patches for Meltdown / Spectre. I confirmed that the processor patches were not actually enabled with Powershell script. Then I removed a single patch, the Microsoft windows update KB4103720 on 5/29/2018.

So looking at the Veeam backup logs over the last few days after removing the Microsoft windows update KB4103720 and performance appears to be doing REALLY GOOD again. Most Synthetic Fulls are running in less than half an hour, some are a little over half an hour, and a couple larger ones are running in around an hour and a half. SO MUCH BETTER now!

Thanks to everyone else posting their experience after this ReFS debaucle due to an updated ReFS driver included in the May 2018 KB4103720 windows update.

[scis]

Correction to my original Post, beginning with

I regretfully applied KB103720, KB4132216, & KB 4103723 to our disk backup servers...

Should read...
I regretfully applied KB4103720, KB4132216, & KB 4103723 to our disk backup servers. I left that "4" out, sorry about that.

[mweissen13]

Unfortunately, removing Updates comes with it's own problems. On one of our Veeam servers, the removal of KB4103720 resulted in the server not booting anymore. After the removal it was hanging during startup doing the "circling dots" thing and not doing anything else (no CPU and no disk activity). After an about hour of waiting i had to give up and restore the last good backup - which of course resulted in a "chicken and egg" problem, because this was the Veeam VM. Sigh.

Is there an easy way to just downgrade refs.sys but leave the cumulative update installed?

[Gostev]

Is there an easy way to just downgrade refs.sys but leave the cumulative update installed?

As I mentioned just a few posts above, you can simply drop an older version ReFS.sys into the fully patched system. This is exactly what our QC did as a part of their testing. Thanks!

[oscaru]

Yes, removing May 2018 KB4103720 windows update fixes the full clone performance problem for us.
As a small service provider it's time consuming to replace the refs.sys 2248 driver on all Veeam servers we manage, but I could confirm it fixes the problem.

[mweissen13]

As I mentioned just a few posts above, you can simply drop an older version ReFS.sys into the fully patched system. This is exactly what our QC did as a part of their testing. Thanks!

Yes, but you can't just copy it over while Windows is running - the file is locked. So you would have to start from some recovery media first, which is not exactly and always "easy". Also i would expect that Windows's sfc (or the next update) would put the original / broken file back.

[Gostev]

Actually, we have already received a private ReFS driver from Microsoft last night, and even had a chance to test it in our lab today. Happy to report that it does fix the issue, bringing block cloning performance back to normal. Let's hope that it's going to make it into the immediate Windows update!

[elerium]

Yes, but you can't just copy it over while Windows is running - the file is locked. So you would have to start from some recovery media first, which is not exactly and always "easy". Also i would expect that Windows's sfc (or the next update) would put the original / broken file back.

You can copy over it by taking ownership of the file and granting administrator Full access. Then grant ownership back to NT Service\TrustedInstaller. This works from Windows explorer without booting into recovery media. I would still expect the next security rollup will update refs.sys again but is a shorterm workaround that keeps all existing security updates.

[KFM]

Actually, we have already received a private ReFS driver from Microsoft last night, and even had a chance to test it in our lab today. Happy to report that it does fix the issue, bringing block cloning performance back to normal. Let's hope that it's going to make it into the immediate Windows update!

So until this is released to the masses via WU or the plain old "drop-the-refs.sys-file-into-the-directory-and-reboot" option, can I confirm that the desired refs.sys driver and version is (as per Gostev's post https://forums.veeam.com/posting.php?mo ... 9#pr282607):

KB4103723 from May 8, 2018 (OS Build 14393.2248)
Refs.sys size 1,709,400 version 10.0.14393.2248 from 28-Apr-18 5:25

On our repos, I can see that KB4103723 is ready to be installed but I didn't want pull the trigger until confirmed. I am currently on refs.sys driver version 10.0.14393.2097.

[Gostev]

2097 is the one you want to be using at this time.

[nezzer]

Any updates on the REFS upgrade from Microsoft @gostev? This slowdown is driving our backupwindows off the charts... ?

[Gostev]

I assume they will be including the fixed driver in one of immediate updates... probably in July, as I think it is too late to make June update.

[mkaec]

I think some would prefer if they were to issue it as an individual out of band update. There's no law that is forcing them to hold it for a CU.

[Gostev]

This is indeed something Veeam would do. I am not aware of Windows servicing process, but it can be very different from ours - so I would not confidently state that there's no such law.

[mkaec]

Well, I have seen out-of-band Windows patches get released within the last 12 months through Windows Update. So, I know it's possible. But, you're right. We're not aware of the bureaucracy that may be going on behind the scenes.

[Augustronic]

2097 is the one you want to be using at this time.

How do I get this old ReFS build?

[Henrik.Grevelund]

Hi Achim,

It's in MS patch : KB4093120

you can download the patch from here:
https://www.catalog.update.microsoft.co ... =KB4093120

[mvalpreda]

Are you saying to install that patch again? Or extract files from that patch?