Comprehensive data protection for all workloads
Locked
rverhe
Influencer
Posts: 11
Liked: never
Joined: Dec 01, 2014 12:50 pm
Full Name: Rob Verhees
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by rverhe »

Have the same issues with update: KB4103720 , REFS Fast clone merge time doubled...
Someone already have an solution for this?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

KB4103720 does not list any changes which could affect ReFS performance even remotely. Can you confirm it is indeed caused by this update by temporarily uninstalling one?
scis
Novice
Posts: 3
Liked: 1 time
Joined: May 29, 2018 6:30 pm
Full Name: Clint Gayle
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by scis » 1 person likes this post

I regretfully applied KB103720, KB4132216, & KB 4103723 to our disk backup servers, and I too am having issues with Veeam Fast Clone after applying these windows updates on 5/21/2018. I'm using ReFS on our primary disk backup server with 208 TB in our storage pool. The other disk backup servers are using NTFS in a SOBR configuration, and i only have a single small backup job with 4 objects and 1.4 TB of data to these servers. I'm not really noticing an issue with the other disk servers using NTFS, but they likely are using the processor as much either since they're setup with NTFS.

Exchange backup jobs for Synthetics backups.
Backup TypeStart TimeDurationProcessing Rate (MB/Sec)Data Size (GB)Transferred (GB)Total Backup Size (GB)
Synthetic Full 5/5/2018 4:00 PM 00:18:53 402.28 5131.00 56.12 6172.65
Synthetic Full 5/12/2018 4:00 PM 00:19:14 348.61 5131.00 51.18 6228.79
Synthetic Full 5/19/2018 4:00 PM 00:18:41 409.74 5131.00 62.69 6296.45
Synthetic Full 5/26/2018 4:00 PM 04:13:03 338.37 5131.00 50.33 6322.44

Another example with 61 objects and 17TB of data....
Synthetic Full 5/14/2018 10:00 PM 00:41:17 375.08 17069.82 211.66 28112.28
Synthetic Full 5/21/2018 10:00 PM 06:22:42 335.23 17085.82 210.25 28413.62

Laserfiche backup jobs
Synthetic Full 5/4/2018 7:00 PM 00:39:35 318.60 12667.40 34.92 25655.09
Synthetic Full 5/11/2018 7:00 PM 00:43:48 369.57 10772.00 45.05 25763.46
Synthetic Full 5/18/2018 7:00 PM 00:47:23 222.24 10772.00 25.36 25485.99
Synthetic Full 5/25/2018 7:00 PM 02 day(s) 15:59:21 330.44 10822.00 32.88 0.00

One large file server job....
Synthetic Full 5/5/2018 4:45 PM 00:27:50 123.38 7787.80 5.23 23235.91
Synthetic Full 5/12/2018 4:45 PM 00:26:45 128.98 7787.80 5.05 23250.56
Synthetic Full 5/19/2018 4:45 PM 00:25:40 132.72 7787.80 6.77 23421.17
Synthetic Full 5/26/2018 4:45 PM 01 day(s) 06:41:08 136.91 7787.80 5.49 23153.92

Not sure what i will do quite yet, but I have a job that started on 5/27/2018 @ 9PM that is now 39 hours and 29 minutes into the Synthetic Full backup sitting at 99% job progress, but in the details says "Creating synthetic full backup (67% done) [fast clone]

I suspect it might have something to do with the processor patch for Meltdown / Spectre, so I may investigate disabling that option through a registry change.
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by NightBird » 1 person likes this post

Same problem here... I just upgraded a small setup to test.

From 1mn30s to 9mn10s... for synthetic full

ReFS driver version
10.0.14393.2273 04/28/2018
JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by JimmyO »

OMG - never ending story? I´m just migrating my entire environmet from NTFS to ReFS for the second time. A 3-6-months job and now this!

I really hope Veeam can put some pressure on MS to fix this asap..
EzE
Influencer
Posts: 19
Liked: never
Joined: Feb 06, 2015 3:48 pm
Full Name: Eric H
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by EzE »

I've uninstalled KB4103720 and have confirmed ReFS.sys is back to version 2248. I'm awaiting a Merge operation to see speeds, but scis and NightBird appear to confirm the issue starts when ReFS is updated to 10.0.14393.2273. I have a ticket open with Veeam, but someone with a test environment might be in a better position to go back and forth. I could open a ticket with Microsoft, but something tells me I won't get anywhere.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

We have the test environment, no worries! And already confirmed there's indeed a performance drop of over 3x after installing May updates in our lab.

And after looking at the ReFS drivers included in May updates to the previous (stable) version from February, it is apparent that Microsoft included the new (and actually different) versions of the driver in both packages without noting anything in the release notes of either update...

KB4093120 from April 17, 2018 (OS Build 14393.2214)
Refs.sys size 1,709,400 version 10.0.14393.2097 from 12-Feb-18 22:37

KB4103723 from May 8, 2018 (OS Build 14393.2248)
Refs.sys size 1,709,400 version 10.0.14393.2248 from 28-Apr-18 5:25

KB4103720 from May 17, 2018 (OS Build 14393.2273)
Refs.sys size 1,709,912 version 10.0.14393.2273 from 28-Apr-18 6:44

I am translating all of our findings in real-time directly into the ReFS dev team, they are online.

The quick fix for now is obviously to simply uninstall May updates in order to revert to the previous version of the driver.
emapola
Lurker
Posts: 1
Liked: never
Joined: May 31, 2018 11:07 am
Full Name: Emanuele Pola
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by emapola »

I confirm, w2016 build 1803, local REFS ~50TB, same performance issues. Also GFS to tape hang on error (before build upgrade works fine).
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

Another test we did today is dropping ReFS.sys from February (10.0.14393.2097) into the lab with May updates installed. This made block cloning performance come back to almost normal (performance drop from May updates was over 3x, and using the old driver improved performance back by 2.5x).

So, it does look like the issue is caused by some "under the carpet" changes to ReFS.sys which were not even documented in the release notes...
SteveBLGS
Novice
Posts: 7
Liked: 1 time
Joined: Jan 03, 2018 9:52 pm
Full Name: Steve
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by SteveBLGS » 1 person likes this post

Removing KB4103720 resolved our performance issues.

refs.sys version
-before removing patch: 10.0.14393.2273
-after removing patch: 10.0.14393.2248

We have a 14 TB VAL job that typically takes around 1 hour to perform a synthetic full using fast clone. After this patch was applied it took nearly 100 hours. We also have a 2 TB VAL job that typically performs a synthetic full backup using fast clone in < 10 minutes. After the path, it was taking over 1 hour.

Steve
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mkretzer »

Can we rename this thread to the original title please? ;-)

Serious: Has someone already talked to MS about the reason why they did this? We were just re-considering REFS and were doing some small tests...

Markus
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

Yes, as noted above I've been in touch with them. They're investigating the root cause with the servicing team now.
scis
Novice
Posts: 3
Liked: 1 time
Joined: May 29, 2018 6:30 pm
Full Name: Clint Gayle
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by scis »

I'd posted last week about regretfully applying KB103720, KB4132216, & KB 4103723 to our disk backup servers, and then experiencing major performance issues with Veeam Fast Cloning ( Veeam Synthetic Fulls ) and looking at options for disabling processor patches for Meltdown / Spectre. I confirmed that the processor patches were not actually enabled with Powershell script. Then I removed a single patch, the Microsoft windows update KB4103720 on 5/29/2018.

So looking at the Veeam backup logs over the last few days after removing the Microsoft windows update KB4103720 and performance appears to be doing REALLY GOOD again. Most Synthetic Fulls are running in less than half an hour, some are a little over half an hour, and a couple larger ones are running in around an hour and a half. SO MUCH BETTER now! :D

Thanks to everyone else posting their experience after this ReFS debaucle due to an updated ReFS driver included in the May 2018 KB4103720 windows update.
scis
Novice
Posts: 3
Liked: 1 time
Joined: May 29, 2018 6:30 pm
Full Name: Clint Gayle
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by scis »

Correction to my original Post, beginning with
scis wrote:I regretfully applied KB103720, KB4132216, & KB 4103723 to our disk backup servers...
Should read...
I regretfully applied KB4103720, KB4132216, & KB 4103723 to our disk backup servers. I left that "4" out, sorry about that.
mweissen13
Enthusiast
Posts: 93
Liked: 54 times
Joined: Dec 28, 2017 3:22 pm
Full Name: Michael Weissenbacher
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mweissen13 »

Unfortunately, removing Updates comes with it's own problems. On one of our Veeam servers, the removal of KB4103720 resulted in the server not booting anymore. After the removal it was hanging during startup doing the "circling dots" thing and not doing anything else (no CPU and no disk activity). After an about hour of waiting i had to give up and restore the last good backup - which of course resulted in a "chicken and egg" problem, because this was the Veeam VM. Sigh.

Is there an easy way to just downgrade refs.sys but leave the cumulative update installed?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » 1 person likes this post

mweissen13 wrote:Is there an easy way to just downgrade refs.sys but leave the cumulative update installed?
As I mentioned just a few posts above, you can simply drop an older version ReFS.sys into the fully patched system. This is exactly what our QC did as a part of their testing. Thanks!
oscaru
Service Provider
Posts: 27
Liked: 11 times
Joined: Jul 26, 2016 6:49 pm
Full Name: Oscar Suarez
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by oscaru »

Yes, removing May 2018 KB4103720 windows update fixes the full clone performance problem for us.
As a small service provider it's time consuming to replace the refs.sys 2248 driver on all Veeam servers we manage, but I could confirm it fixes the problem.
mweissen13
Enthusiast
Posts: 93
Liked: 54 times
Joined: Dec 28, 2017 3:22 pm
Full Name: Michael Weissenbacher
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mweissen13 »

Gostev wrote:As I mentioned just a few posts above, you can simply drop an older version ReFS.sys into the fully patched system. This is exactly what our QC did as a part of their testing. Thanks!
Yes, but you can't just copy it over while Windows is running - the file is locked. So you would have to start from some recovery media first, which is not exactly and always "easy". Also i would expect that Windows's sfc (or the next update) would put the original / broken file back.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » 1 person likes this post

Actually, we have already received a private ReFS driver from Microsoft last night, and even had a chance to test it in our lab today. Happy to report that it does fix the issue, bringing block cloning performance back to normal. Let's hope that it's going to make it into the immediate Windows update!
elerium
Lurker
Posts: 1
Liked: never
Joined: Nov 01, 2017 9:21 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by elerium »

mweissen13 wrote: Yes, but you can't just copy it over while Windows is running - the file is locked. So you would have to start from some recovery media first, which is not exactly and always "easy". Also i would expect that Windows's sfc (or the next update) would put the original / broken file back.
You can copy over it by taking ownership of the file and granting administrator Full access. Then grant ownership back to NT Service\TrustedInstaller. This works from Windows explorer without booting into recovery media. I would still expect the next security rollup will update refs.sys again but is a shorterm workaround that keeps all existing security updates.
KFM
Service Provider
Posts: 13
Liked: 2 times
Joined: May 14, 2013 1:46 am
Full Name: KFM
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by KFM »

Gostev wrote:Actually, we have already received a private ReFS driver from Microsoft last night, and even had a chance to test it in our lab today. Happy to report that it does fix the issue, bringing block cloning performance back to normal. Let's hope that it's going to make it into the immediate Windows update!
So until this is released to the masses via WU or the plain old "drop-the-refs.sys-file-into-the-directory-and-reboot" option, can I confirm that the desired refs.sys driver and version is (as per Gostev's post https://forums.veeam.com/posting.php?mo ... 9#pr282607):

KB4103723 from May 8, 2018 (OS Build 14393.2248)
Refs.sys size 1,709,400 version 10.0.14393.2248 from 28-Apr-18 5:25

On our repos, I can see that KB4103723 is ready to be installed but I didn't want pull the trigger until confirmed. I am currently on refs.sys driver version 10.0.14393.2097.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

2097 is the one you want to be using at this time.
nezzer
Influencer
Posts: 22
Liked: 5 times
Joined: Apr 27, 2018 11:40 am
Full Name: Andreas Svensson
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by nezzer »

Any updates on the REFS upgrade from Microsoft @gostev? This slowdown is driving our backupwindows off the charts... ?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

I assume they will be including the fixed driver in one of immediate updates... probably in July, as I think it is too late to make June update.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mkaec »

I think some would prefer if they were to issue it as an individual out of band update. There's no law that is forcing them to hold it for a CU.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

This is indeed something Veeam would do. I am not aware of Windows servicing process, but it can be very different from ours - so I would not confidently state that there's no such law.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mkaec »

Well, I have seen out-of-band Windows patches get released within the last 12 months through Windows Update. So, I know it's possible. But, you're right. We're not aware of the bureaucracy that may be going on behind the scenes.
Augustronic
Lurker
Posts: 1
Liked: never
Joined: Jun 08, 2018 11:19 am
Full Name: Achim August Tietz
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Augustronic »

Gostev wrote:2097 is the one you want to be using at this time.
How do I get this old ReFS build?
Henrik.Grevelund
Service Provider
Posts: 160
Liked: 18 times
Joined: Feb 13, 2017 2:56 pm
Full Name: Henrik Grevelund
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Henrik.Grevelund »

Hi Achim,

It's in MS patch : KB4093120

you can download the patch from here:
https://www.catalog.update.microsoft.co ... =KB4093120
Have nice day,
Henrik
mvalpreda
Enthusiast
Posts: 77
Liked: 3 times
Joined: May 06, 2015 10:57 pm
Full Name: Mark Valpreda
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mvalpreda »

Are you saying to install that patch again? Or extract files from that patch?
Locked

Who is online

Users browsing this forum: Google [Bot] and 159 guests