Windows Server 2019 Hyper-V VM I/O Performance Problem

[johan.h]

Just wanted to say thank you again @stephc_msft for all the hard work and follow-through on this Feb 11th fix for Server 2022 and the change in the Server 2025 functionality.

[stephc_msft]

To avoid the need for two reboots (reboot to apply CU, 2nd reboot after enabling the fix) the enablement procedure can be set up in advance before the CU is applied.

[Rmachado]

Latest update.
I can confirm the fix that we are confident will help overcome this issue will be in the Windows Server 2022 February 2025 update (on 11th Feb).
HOWEVER, it will be a 'disabled by default' fix, that will need to be explicitly enabled
There will be a msi file available to enable it
Generic info about 'disabled by default' and how to use the msi/admx and to set up a group policy to enable is at:
https://learn.microsoft.com/en-us/troub ... by-default
(The enablement actually just sets up up a registry key and the details of that could theoretically be determined from the msi ...)

The fix is also in WS2025, but is permanently enabled in that.
Anyone here have any feedback on WS2025? ie to confirm if no longer having the issue after moving their Hyper-V hosts to WS2025 ?

Hello

Thanks for the update.

It will only available for 2022 and forward? nothing for Windows 2019?

Thank you again.

[gman42]

I think it was mentioned previously that 2019 won't get the patch because it's already out of mainstream support. Which sucks because the discovery of the bug obviously predates that by years, but it is what it is.

[Nick-SAC]

... 2019 won't get the patch because it's already out of mainstream support. Which sucks because the discovery of the bug obviously predates that by years,...

Exactly!

[stephc_msft]

Re WS2019 fix. Yeap, sorry I did push for one but it was rejected.
In the past there was no incentive to move to WS2022 as it was 'known' that WS2022 still had the issue.
However now, once it is confirmed that the WS2022 fix works, that gives an incentive to update (and will need to update in due course anyway due to other end-of-life/supportability factors).

[mkeating44]

To avoid the need for two reboots (reboot to apply CU, 2nd reboot after enabling the fix) the enablement procedure can be set up in advance before the CU is applied.

Thanks Stephen! Really appreciate it! Also thanks to Johan and the Veeam team for their help as well.

Will there be documentation released for this fix as part of the February update, or available somewhere? Or is the KB number for the February CU enough?

[SodaPop87]

Hoping for some more clarification on WS2022 update. Does this update need to be applied to the VEEAM server or the Hyper-V hosts, or both, and by disabled by default do I need to enable something before the update to able to download the update or do something after the update is deployed?

[steendp]

The fix is also in WS2025, but is permanently enabled in that.
Anyone here have any feedback on WS2025? ie to confirm if no longer having the issue after moving their Hyper-V hosts to WS2025 ?

We have moved part of our workload to WS2025 and have yet to experience the problem on these hosts. Keeping my fingers crossed .

[stephc_msft]

The fix (February Cumulative Update) need to be applied to the Hyper-V host servers / nodes.

The enablement policy (msi) file is available from
https://download.microsoft.com/download ... review.msi

The technique to run it and then apply the policy (local policy or can be done by GPO) is described in the generic article
https://learn.microsoft.com/en-us/troub ... by-default

As I mentioned, applying the enablement and policy just before the CU update avoids having to do two reboots.

[johan.h]

We've gone ahead and published a KB for this topic as well https://www.veeam.com/kb4717

[yoda-ict]

Ok, as there is a solution for this problem available we need to upgrade our 3-node 2019 Hyper-V cluster to 2022 Hyper-V. As there are a lot of users here watching this topic and having Hyper-V cluster experience; i cannot determine if we can use In-place upgrade of each 2019 Hyper-V node to 2022 Hyper-V using Cluster rolling upgrade. So can we drain, evict each node separately, and upgrade windows 2019 to windows 2022 using the ISO, upgrade HPE drivers, and join the existing cluster again? A lot of publications suggest reinstall the nodes from scratch but as inplace upgrades are sort of nobrainer these days i want your opinions on this to speed up upgrades significanty..

[SnakeSK]

We've gone ahead and published a KB for this topic as well https://www.veeam.com/kb4717

Wrong path - HKLM\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides is correct

[johan.h]

Good catch. Will fix as soon as the team comes back online

[RBeismann]

Ok, as there is a solution for this problem available we need to upgrade our 3-node 2019 Hyper-V cluster to 2022 Hyper-V. As there are a lot of users here watching this topic and having Hyper-V cluster experience; i cannot determine if we can use In-place upgrade of each 2019 Hyper-V node to 2022 Hyper-V using Cluster rolling upgrade. So can we drain, evict each node separately, and upgrade windows 2019 to windows 2022 using the ISO, upgrade HPE drivers, and join the existing cluster again? A lot of publications suggest reinstall the nodes from scratch but as inplace upgrades are sort of nobrainer these days i want your opinions on this to speed up upgrades significanty..

I can just tell from experience, we're always upgrading our Hyper-V hosts using in-place upgrade and don't evict them from the cluster beforehand. All we do is putting them into maintenance mode (via SCVMM), push our IPU Task Sequence via SCCM and re-enable them afterwards. We did this procedure for at least 5 clusters in our environment without issues.

[stephc_msft]

If its an S2D cluster, then also need to be careful to ensure no storage jobs are running, and to put the node about to be updated into Storage Maintenance mode.
May want to update the cluster functional level (from v10 in ws2019 to v11 in ws2022) later on, but not necessary for running existing VM's

If using ReFS (which S2D will usually be using on its virtualdisks) then the ReFS version may also update from v3.4 to v3.7
it should be clever enough to not update till all nodes are on ws2022
as ws2019 can NOT read v3.7
Else if worried there is a registry key that can be set IN ADVANCE to avoid the update
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem] "RefsDisableVolumeUpgrade" dword:00000001

[yoda-ict]

I can just tell from experience, we're always upgrading our Hyper-V hosts using in-place upgrade and don't evict them from the cluster beforehand. All we do is putting them into maintenance mode (via SCVMM), push our IPU Task Sequence via SCCM and re-enable them afterwards. We did this procedure for at least 5 clusters in our environment without issues.

OK that is a lot easier than what i read globally on the web! We maintain the cluster within SCVMM2022 so putting into maintentance mode and perform an inplace-upgrade together with latest drivers should suffice. Afterwards we can update the functional level of the cluster. We only use CSV volumes so no S2D/ReFS issues.
Due to the workarounds mentioned in this topic we disabled Intel Hyperthreading on all hosts. As the issues are solved is it recommended to re-enable Hyperthreading again?

[stephc_msft]

side note for the few customers (several from here) who were testing the private back in Oct/Nov 2024
You need to uninstall the private before updating to the Feb update
Ping me if this applies to you and if you have any concerns

[mkeating44]

I can just tell from experience, we're always upgrading our Hyper-V hosts using in-place upgrade and don't evict them from the cluster beforehand. All we do is putting them into maintenance mode (via SCVMM), push our IPU Task Sequence via SCCM and re-enable them afterwards. We did this procedure for at least 5 clusters in our environment without issues.

If not running SCVMM, evicting the node from the cluster, removing from domain, clean install of Windows Server 2022, domain join and re-join cluster works remarkably well. We used the information at https://learn.microsoft.com/en-us/windo ... de-process and it worked a treat. No downtime, no impact to VMs. Given the history of Windows OS and cluster upgrades I was very happy this was so good.

S2D guidance is at https://learn.microsoft.com/en-us/windo ... re-running and we will be testing this in the coming weeks. I have no reason to think there will be issues as the non-S2D cluster upgrade process was flawless, but @stephc_msft 's point and potential fix is useful!

[rold]

Thanks Microsoft for these wonderful 8 years

[stephc_msft]

WS2022 February update now available
https://support.microsoft.com/en-us/top ... 12fe80b2ee
Has link to 'Catalog' to download from
https://www.catalog.update.microsoft.co ... =KB5051979
21H2 is WS2022 (can check with winver)
22H2 is Azure Stack HCI 22H2
(although the fix package is identical)

See earlier notes about the extra .msi and policy to enable the fix that is also needed

[marksf]

Glad this is finally fixed, is it confirmed that it would be fixed if we were to do a in place upgrade from 2019 to 2025?

[JKSAAHS]

Finally it comes to an end a 5 year long struggle with slow servers - There is stuff enough for at least a podcast about it.

[SodaPop87]

Applied, enabled and rebooted yesterday the Enable policy. Just pulled, installed and rebooted the CU update for Microsoft on all 3 HV hosts. Will report back next week if we have any needs to Live Migrate our SQL server still after 4-5 days. Thank you everyone who has been working on this annoying longstanding issue and not giving up.

[NewVM]

Hi,

I don't understand. The Veeam article mention 'HKLM\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides', but I can't find any?

Sander.

[david.domask]

Hi Sander,

The KB mentions to create this value, so go ahead and create the value if it doesn't exist already, or install the fix enablement policy as described in the KB article.

[NewVM]

Okay, somehow I rembered the locatio should already exists, but mine only goes till [Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Policies\ -> only key Hardware exist]
But do I still have to create the register key after installing the production CU release? (just installed, rebooted, but no key)

Sander.

[NewVM]

Ok, created the full path and dword. Updated five failover clusternodes for a selected customer. Will see in 48 hours.

[andy.dedeckker]

Sander,

I you extract the msi with 7-zip, you can see that it installs a local group policy.
If you look into the admx file, you can see that enabling that policy would create the registry key.
So installing the msi is not enough. After installing the msi, you should also need to enable the local policy through gpedit.msc. After you enabled that policy, you could see the registry key created.
But, manually creating the registry key should work fine...

[NewVM]

I understand now, the article (https://www.veeam.com/kb4717) is preparing administrators to stage the fix enablement so that after the installation ánd reboot to finalize of the Feb CU (MS kb5051979) the fix is activated.
(I did the assumption that there was a preview fix, but there is none)

Code: Select all

  <policies>
    <policy name="KB5051979_250131_0845_7_FeaturePreview" class="Machine" displayName="$(string.KB5051979_250131_0845_7_FeaturePreview)" explainText="$(string.WUKnownIssue_Help)" key="SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides">
      <parentCategory ref="KnownIssueRollback_Server_2022" />
      <supportedOn ref="SUPPORTED_Windows_10_0_Server_2022_Only" />
      <enabledList defaultKey="SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides"><item valueName="636159629"><value><decimal value="1" /></value></item></enabledList>
      <disabledList defaultKey="SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides"><item valueName="636159629"><value><decimal value="0" /></value></item></disabledList>
    </policy>
  </policies>