-
- Influencer
- Posts: 11
- Liked: never
- Joined: Dec 01, 2014 12:50 pm
- Full Name: Rob Verhees
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Have the same issues with update: KB4103720 , REFS Fast clone merge time doubled...
Someone already have an solution for this?
Someone already have an solution for this?
-
- Chief Product Officer
- Posts: 31783
- Liked: 7283 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
KB4103720 does not list any changes which could affect ReFS performance even remotely. Can you confirm it is indeed caused by this update by temporarily uninstalling one?
-
- Novice
- Posts: 3
- Liked: 1 time
- Joined: May 29, 2018 6:30 pm
- Full Name: Clint Gayle
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I regretfully applied KB103720, KB4132216, & KB 4103723 to our disk backup servers, and I too am having issues with Veeam Fast Clone after applying these windows updates on 5/21/2018. I'm using ReFS on our primary disk backup server with 208 TB in our storage pool. The other disk backup servers are using NTFS in a SOBR configuration, and i only have a single small backup job with 4 objects and 1.4 TB of data to these servers. I'm not really noticing an issue with the other disk servers using NTFS, but they likely are using the processor as much either since they're setup with NTFS.
Exchange backup jobs for Synthetics backups.
Backup TypeStart TimeDurationProcessing Rate (MB/Sec)Data Size (GB)Transferred (GB)Total Backup Size (GB)
Synthetic Full 5/5/2018 4:00 PM 00:18:53 402.28 5131.00 56.12 6172.65
Synthetic Full 5/12/2018 4:00 PM 00:19:14 348.61 5131.00 51.18 6228.79
Synthetic Full 5/19/2018 4:00 PM 00:18:41 409.74 5131.00 62.69 6296.45
Synthetic Full 5/26/2018 4:00 PM 04:13:03 338.37 5131.00 50.33 6322.44
Another example with 61 objects and 17TB of data....
Synthetic Full 5/14/2018 10:00 PM 00:41:17 375.08 17069.82 211.66 28112.28
Synthetic Full 5/21/2018 10:00 PM 06:22:42 335.23 17085.82 210.25 28413.62
Laserfiche backup jobs
Synthetic Full 5/4/2018 7:00 PM 00:39:35 318.60 12667.40 34.92 25655.09
Synthetic Full 5/11/2018 7:00 PM 00:43:48 369.57 10772.00 45.05 25763.46
Synthetic Full 5/18/2018 7:00 PM 00:47:23 222.24 10772.00 25.36 25485.99
Synthetic Full 5/25/2018 7:00 PM 02 day(s) 15:59:21 330.44 10822.00 32.88 0.00
One large file server job....
Synthetic Full 5/5/2018 4:45 PM 00:27:50 123.38 7787.80 5.23 23235.91
Synthetic Full 5/12/2018 4:45 PM 00:26:45 128.98 7787.80 5.05 23250.56
Synthetic Full 5/19/2018 4:45 PM 00:25:40 132.72 7787.80 6.77 23421.17
Synthetic Full 5/26/2018 4:45 PM 01 day(s) 06:41:08 136.91 7787.80 5.49 23153.92
Not sure what i will do quite yet, but I have a job that started on 5/27/2018 @ 9PM that is now 39 hours and 29 minutes into the Synthetic Full backup sitting at 99% job progress, but in the details says "Creating synthetic full backup (67% done) [fast clone]
I suspect it might have something to do with the processor patch for Meltdown / Spectre, so I may investigate disabling that option through a registry change.
Exchange backup jobs for Synthetics backups.
Backup TypeStart TimeDurationProcessing Rate (MB/Sec)Data Size (GB)Transferred (GB)Total Backup Size (GB)
Synthetic Full 5/5/2018 4:00 PM 00:18:53 402.28 5131.00 56.12 6172.65
Synthetic Full 5/12/2018 4:00 PM 00:19:14 348.61 5131.00 51.18 6228.79
Synthetic Full 5/19/2018 4:00 PM 00:18:41 409.74 5131.00 62.69 6296.45
Synthetic Full 5/26/2018 4:00 PM 04:13:03 338.37 5131.00 50.33 6322.44
Another example with 61 objects and 17TB of data....
Synthetic Full 5/14/2018 10:00 PM 00:41:17 375.08 17069.82 211.66 28112.28
Synthetic Full 5/21/2018 10:00 PM 06:22:42 335.23 17085.82 210.25 28413.62
Laserfiche backup jobs
Synthetic Full 5/4/2018 7:00 PM 00:39:35 318.60 12667.40 34.92 25655.09
Synthetic Full 5/11/2018 7:00 PM 00:43:48 369.57 10772.00 45.05 25763.46
Synthetic Full 5/18/2018 7:00 PM 00:47:23 222.24 10772.00 25.36 25485.99
Synthetic Full 5/25/2018 7:00 PM 02 day(s) 15:59:21 330.44 10822.00 32.88 0.00
One large file server job....
Synthetic Full 5/5/2018 4:45 PM 00:27:50 123.38 7787.80 5.23 23235.91
Synthetic Full 5/12/2018 4:45 PM 00:26:45 128.98 7787.80 5.05 23250.56
Synthetic Full 5/19/2018 4:45 PM 00:25:40 132.72 7787.80 6.77 23421.17
Synthetic Full 5/26/2018 4:45 PM 01 day(s) 06:41:08 136.91 7787.80 5.49 23153.92
Not sure what i will do quite yet, but I have a job that started on 5/27/2018 @ 9PM that is now 39 hours and 29 minutes into the Synthetic Full backup sitting at 99% job progress, but in the details says "Creating synthetic full backup (67% done) [fast clone]
I suspect it might have something to do with the processor patch for Meltdown / Spectre, so I may investigate disabling that option through a registry change.
-
- Expert
- Posts: 245
- Liked: 58 times
- Joined: Apr 28, 2009 8:33 am
- Location: Strasbourg, FRANCE
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Same problem here... I just upgraded a small setup to test.
From 1mn30s to 9mn10s... for synthetic full
ReFS driver version
10.0.14393.2273 04/28/2018
From 1mn30s to 9mn10s... for synthetic full
ReFS driver version
10.0.14393.2273 04/28/2018
-
- Enthusiast
- Posts: 55
- Liked: 9 times
- Joined: Apr 27, 2014 8:19 pm
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
OMG - never ending story? I´m just migrating my entire environmet from NTFS to ReFS for the second time. A 3-6-months job and now this!
I really hope Veeam can put some pressure on MS to fix this asap..
I really hope Veeam can put some pressure on MS to fix this asap..
-
- Influencer
- Posts: 19
- Liked: never
- Joined: Feb 06, 2015 3:48 pm
- Full Name: Eric H
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I've uninstalled KB4103720 and have confirmed ReFS.sys is back to version 2248. I'm awaiting a Merge operation to see speeds, but scis and NightBird appear to confirm the issue starts when ReFS is updated to 10.0.14393.2273. I have a ticket open with Veeam, but someone with a test environment might be in a better position to go back and forth. I could open a ticket with Microsoft, but something tells me I won't get anywhere.
-
- Chief Product Officer
- Posts: 31783
- Liked: 7283 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
We have the test environment, no worries! And already confirmed there's indeed a performance drop of over 3x after installing May updates in our lab.
And after looking at the ReFS drivers included in May updates to the previous (stable) version from February, it is apparent that Microsoft included the new (and actually different) versions of the driver in both packages without noting anything in the release notes of either update...
KB4093120 from April 17, 2018 (OS Build 14393.2214)
Refs.sys size 1,709,400 version 10.0.14393.2097 from 12-Feb-18 22:37
KB4103723 from May 8, 2018 (OS Build 14393.2248)
Refs.sys size 1,709,400 version 10.0.14393.2248 from 28-Apr-18 5:25
KB4103720 from May 17, 2018 (OS Build 14393.2273)
Refs.sys size 1,709,912 version 10.0.14393.2273 from 28-Apr-18 6:44
I am translating all of our findings in real-time directly into the ReFS dev team, they are online.
The quick fix for now is obviously to simply uninstall May updates in order to revert to the previous version of the driver.
And after looking at the ReFS drivers included in May updates to the previous (stable) version from February, it is apparent that Microsoft included the new (and actually different) versions of the driver in both packages without noting anything in the release notes of either update...
KB4093120 from April 17, 2018 (OS Build 14393.2214)
Refs.sys size 1,709,400 version 10.0.14393.2097 from 12-Feb-18 22:37
KB4103723 from May 8, 2018 (OS Build 14393.2248)
Refs.sys size 1,709,400 version 10.0.14393.2248 from 28-Apr-18 5:25
KB4103720 from May 17, 2018 (OS Build 14393.2273)
Refs.sys size 1,709,912 version 10.0.14393.2273 from 28-Apr-18 6:44
I am translating all of our findings in real-time directly into the ReFS dev team, they are online.
The quick fix for now is obviously to simply uninstall May updates in order to revert to the previous version of the driver.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: May 31, 2018 11:07 am
- Full Name: Emanuele Pola
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I confirm, w2016 build 1803, local REFS ~50TB, same performance issues. Also GFS to tape hang on error (before build upgrade works fine).
-
- Chief Product Officer
- Posts: 31783
- Liked: 7283 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Another test we did today is dropping ReFS.sys from February (10.0.14393.2097) into the lab with May updates installed. This made block cloning performance come back to almost normal (performance drop from May updates was over 3x, and using the old driver improved performance back by 2.5x).
So, it does look like the issue is caused by some "under the carpet" changes to ReFS.sys which were not even documented in the release notes...
So, it does look like the issue is caused by some "under the carpet" changes to ReFS.sys which were not even documented in the release notes...
-
- Novice
- Posts: 7
- Liked: 1 time
- Joined: Jan 03, 2018 9:52 pm
- Full Name: Steve
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Removing KB4103720 resolved our performance issues.
refs.sys version
-before removing patch: 10.0.14393.2273
-after removing patch: 10.0.14393.2248
We have a 14 TB VAL job that typically takes around 1 hour to perform a synthetic full using fast clone. After this patch was applied it took nearly 100 hours. We also have a 2 TB VAL job that typically performs a synthetic full backup using fast clone in < 10 minutes. After the path, it was taking over 1 hour.
Steve
refs.sys version
-before removing patch: 10.0.14393.2273
-after removing patch: 10.0.14393.2248
We have a 14 TB VAL job that typically takes around 1 hour to perform a synthetic full using fast clone. After this patch was applied it took nearly 100 hours. We also have a 2 TB VAL job that typically performs a synthetic full backup using fast clone in < 10 minutes. After the path, it was taking over 1 hour.
Steve
-
- Veeam Legend
- Posts: 1202
- Liked: 416 times
- Joined: Dec 17, 2015 7:17 am
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Can we rename this thread to the original title please?
Serious: Has someone already talked to MS about the reason why they did this? We were just re-considering REFS and were doing some small tests...
Markus
Serious: Has someone already talked to MS about the reason why they did this? We were just re-considering REFS and were doing some small tests...
Markus
-
- Chief Product Officer
- Posts: 31783
- Liked: 7283 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Yes, as noted above I've been in touch with them. They're investigating the root cause with the servicing team now.
-
- Novice
- Posts: 3
- Liked: 1 time
- Joined: May 29, 2018 6:30 pm
- Full Name: Clint Gayle
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I'd posted last week about regretfully applying KB103720, KB4132216, & KB 4103723 to our disk backup servers, and then experiencing major performance issues with Veeam Fast Cloning ( Veeam Synthetic Fulls ) and looking at options for disabling processor patches for Meltdown / Spectre. I confirmed that the processor patches were not actually enabled with Powershell script. Then I removed a single patch, the Microsoft windows update KB4103720 on 5/29/2018.
So looking at the Veeam backup logs over the last few days after removing the Microsoft windows update KB4103720 and performance appears to be doing REALLY GOOD again. Most Synthetic Fulls are running in less than half an hour, some are a little over half an hour, and a couple larger ones are running in around an hour and a half. SO MUCH BETTER now!
Thanks to everyone else posting their experience after this ReFS debaucle due to an updated ReFS driver included in the May 2018 KB4103720 windows update.
So looking at the Veeam backup logs over the last few days after removing the Microsoft windows update KB4103720 and performance appears to be doing REALLY GOOD again. Most Synthetic Fulls are running in less than half an hour, some are a little over half an hour, and a couple larger ones are running in around an hour and a half. SO MUCH BETTER now!
Thanks to everyone else posting their experience after this ReFS debaucle due to an updated ReFS driver included in the May 2018 KB4103720 windows update.
-
- Novice
- Posts: 3
- Liked: 1 time
- Joined: May 29, 2018 6:30 pm
- Full Name: Clint Gayle
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Correction to my original Post, beginning with
I regretfully applied KB4103720, KB4132216, & KB 4103723 to our disk backup servers. I left that "4" out, sorry about that.
Should read...scis wrote:I regretfully applied KB103720, KB4132216, & KB 4103723 to our disk backup servers...
I regretfully applied KB4103720, KB4132216, & KB 4103723 to our disk backup servers. I left that "4" out, sorry about that.
-
- Enthusiast
- Posts: 93
- Liked: 54 times
- Joined: Dec 28, 2017 3:22 pm
- Full Name: Michael Weissenbacher
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Unfortunately, removing Updates comes with it's own problems. On one of our Veeam servers, the removal of KB4103720 resulted in the server not booting anymore. After the removal it was hanging during startup doing the "circling dots" thing and not doing anything else (no CPU and no disk activity). After an about hour of waiting i had to give up and restore the last good backup - which of course resulted in a "chicken and egg" problem, because this was the Veeam VM. Sigh.
Is there an easy way to just downgrade refs.sys but leave the cumulative update installed?
Is there an easy way to just downgrade refs.sys but leave the cumulative update installed?
-
- Chief Product Officer
- Posts: 31783
- Liked: 7283 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
As I mentioned just a few posts above, you can simply drop an older version ReFS.sys into the fully patched system. This is exactly what our QC did as a part of their testing. Thanks!mweissen13 wrote:Is there an easy way to just downgrade refs.sys but leave the cumulative update installed?
-
- Service Provider
- Posts: 27
- Liked: 11 times
- Joined: Jul 26, 2016 6:49 pm
- Full Name: Oscar Suarez
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Yes, removing May 2018 KB4103720 windows update fixes the full clone performance problem for us.
As a small service provider it's time consuming to replace the refs.sys 2248 driver on all Veeam servers we manage, but I could confirm it fixes the problem.
As a small service provider it's time consuming to replace the refs.sys 2248 driver on all Veeam servers we manage, but I could confirm it fixes the problem.
-
- Enthusiast
- Posts: 93
- Liked: 54 times
- Joined: Dec 28, 2017 3:22 pm
- Full Name: Michael Weissenbacher
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Yes, but you can't just copy it over while Windows is running - the file is locked. So you would have to start from some recovery media first, which is not exactly and always "easy". Also i would expect that Windows's sfc (or the next update) would put the original / broken file back.Gostev wrote:As I mentioned just a few posts above, you can simply drop an older version ReFS.sys into the fully patched system. This is exactly what our QC did as a part of their testing. Thanks!
-
- Chief Product Officer
- Posts: 31783
- Liked: 7283 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Actually, we have already received a private ReFS driver from Microsoft last night, and even had a chance to test it in our lab today. Happy to report that it does fix the issue, bringing block cloning performance back to normal. Let's hope that it's going to make it into the immediate Windows update!
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Nov 01, 2017 9:21 pm
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
You can copy over it by taking ownership of the file and granting administrator Full access. Then grant ownership back to NT Service\TrustedInstaller. This works from Windows explorer without booting into recovery media. I would still expect the next security rollup will update refs.sys again but is a shorterm workaround that keeps all existing security updates.mweissen13 wrote: Yes, but you can't just copy it over while Windows is running - the file is locked. So you would have to start from some recovery media first, which is not exactly and always "easy". Also i would expect that Windows's sfc (or the next update) would put the original / broken file back.
-
- Service Provider
- Posts: 13
- Liked: 2 times
- Joined: May 14, 2013 1:46 am
- Full Name: KFM
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
So until this is released to the masses via WU or the plain old "drop-the-refs.sys-file-into-the-directory-and-reboot" option, can I confirm that the desired refs.sys driver and version is (as per Gostev's post https://forums.veeam.com/posting.php?mo ... 9#pr282607):Gostev wrote:Actually, we have already received a private ReFS driver from Microsoft last night, and even had a chance to test it in our lab today. Happy to report that it does fix the issue, bringing block cloning performance back to normal. Let's hope that it's going to make it into the immediate Windows update!
KB4103723 from May 8, 2018 (OS Build 14393.2248)
Refs.sys size 1,709,400 version 10.0.14393.2248 from 28-Apr-18 5:25
On our repos, I can see that KB4103723 is ready to be installed but I didn't want pull the trigger until confirmed. I am currently on refs.sys driver version 10.0.14393.2097.
-
- Chief Product Officer
- Posts: 31783
- Liked: 7283 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
2097 is the one you want to be using at this time.
-
- Influencer
- Posts: 22
- Liked: 5 times
- Joined: Apr 27, 2018 11:40 am
- Full Name: Andreas Svensson
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Any updates on the REFS upgrade from Microsoft @gostev? This slowdown is driving our backupwindows off the charts... ?
-
- Chief Product Officer
- Posts: 31783
- Liked: 7283 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I assume they will be including the fixed driver in one of immediate updates... probably in July, as I think it is too late to make June update.
-
- Veteran
- Posts: 465
- Liked: 136 times
- Joined: Jul 16, 2015 1:31 pm
- Full Name: Marc K
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I think some would prefer if they were to issue it as an individual out of band update. There's no law that is forcing them to hold it for a CU.
-
- Chief Product Officer
- Posts: 31783
- Liked: 7283 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
This is indeed something Veeam would do. I am not aware of Windows servicing process, but it can be very different from ours - so I would not confidently state that there's no such law.
-
- Veteran
- Posts: 465
- Liked: 136 times
- Joined: Jul 16, 2015 1:31 pm
- Full Name: Marc K
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Well, I have seen out-of-band Windows patches get released within the last 12 months through Windows Update. So, I know it's possible. But, you're right. We're not aware of the bureaucracy that may be going on behind the scenes.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Jun 08, 2018 11:19 am
- Full Name: Achim August Tietz
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
How do I get this old ReFS build?Gostev wrote:2097 is the one you want to be using at this time.
-
- Service Provider
- Posts: 171
- Liked: 26 times
- Joined: Feb 13, 2017 2:56 pm
- Full Name: Henrik Grevelund
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Hi Achim,
It's in MS patch : KB4093120
you can download the patch from here:
https://www.catalog.update.microsoft.co ... =KB4093120
It's in MS patch : KB4093120
you can download the patch from here:
https://www.catalog.update.microsoft.co ... =KB4093120
Have nice day,
Henrik
Henrik
-
- Enthusiast
- Posts: 77
- Liked: 3 times
- Joined: May 06, 2015 10:57 pm
- Full Name: Mark Valpreda
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Are you saying to install that patch again? Or extract files from that patch?
Who is online
Users browsing this forum: Bing [Bot], Marijn, mschwaermer, oscarm, vcharlie and 205 guests