Windows Server 2019 Hyper-V VM I/O Performance Problem

[mkaec]

I you extract the msi with 7-zip, you can see that it installs a local group policy.
If you look into the admx file, you can see that enabling that policy would create the registry key.
So installing the msi is not enough. After installing the msi, you should also need to enable the local policy through gpedit.msc. After you enabled that policy, you could see the registry key created.
But, manually creating the registry key should work fine...

It's odd that they made it so complicated. Group Policy Preferences has been standard in Windows since Windows Server 2008. The Microsoft instructions could just show how to create the registry entry directly through GPP.

[Techie4Life]

Hi,

I just applied the patch. Anyone here have any good way to determine if the patch is successful? Running 2022 here.

I see a few instances since I applied it yesterday of Hyper-V-StorageVSP EvtID 9 in my logs still. Still doing testing to see if performance is better for users.

[NewVM]

Hi all,

can confirm that the updates are working for us (server2022 failoverclusters with S2D and Veeam as backup). No high latency any more after backups on big disks.

Sander.

[Techie4Life]

Hmm, we are running iSCSI and not S2D. What is the difference between running CSVFS over iSCSI and S2D?

[mkeating44]

Hmm, we are running iSCSI and not S2D. What is the difference between running CSVFS over iSCSI and S2D?

None - the issue is any storage used by Hyper-V to run VMs.

S2D just has the built in telemetry to see the issue, with non-S2D it's harder to identify at the storage subsystem.

[bardhivie]

Hello,
Is this registry key provided in the article also for Windows Server 2019 Hyper-V or just for Windows Server 2022?
Or is an upgrade to Windows Server 2022 Hyper-V absolutely necessary?

Thank you in advance for your reply

[johan.h]

The fix is only available in Server 2022 (and equivalent AZSHCI OS versions) via Cumulative Update and enablement, or in Server 2025.

[SodaPop87]

Applied, enabled and rebooted yesterday the Enable policy. Just pulled, installed and rebooted the CU update for Microsoft on all 3 HV hosts. Will report back next week if we have any needs to Live Migrate our SQL server still after 4-5 days. Thank you everyone who has been working on this annoying longstanding issue and not giving up.

It has now been 13 days going on 14 and I have not had to perform a Live Migration on our large SQL database since performing the initial group policy setup and then installing the WU this month. Thank you everyone who continued to work endlessly on this annoying issue and being persistent with Microsoft in getting them to find a permanent solution. It has been years in the making!

[mortendb]

Amazing, i was i part in this issue in the beginning but we ended up switching from Hyper-V to VMware. It might have been a good call.

From and including: Monday, 30 September 2019
To, but not including Wednesday, 12 February 2025

Result: 1962 days
It is 1962 days from the start date to the end date, but not including the end date.

Or 5 years, 4 months, 13 days excluding the end date.

Or 64 months, 13 days excluding the end date.

But its great to see it seems to finally have been resolved.

[Nick-SAC]

5 years, 4 months, 13 days...

Times the countless number of hours IT Techs wasted troubleshooting & trying to rectify the problem... and the incalculable lost production time of those affected by it!

But hey, at least we got Bing for free

[mkaec]

Not having to deal with Broadcom is a big plus in my book. I'm quite surprised that @mortendb considers the switch to VMware a good call.

[m.novelli]

Broadcom issue is only pricing , if you use the basic features of VMware (as you should with every software) it's rock solid since 15 years

Sorry, VMware screwed vCenter even as appliance so many times... I'm wrong

Marco

[mkeating44]

Quick thread bump - we upgraded all our S2D hosts last month to Windows Server 2022. Had a mix of in-place upgrades and fresh builds. No issues with ReFS upgrades prior to cluster version upgrade, once the cluster version was upgraded we saw an increase of disk latency across the board for a week (old hardware plus what I am assuming is the ReFS upgrade). After that we've only seen disk latency lower than what we had on Server 2019, less spikes and the issue with the high usage VM disk latency is completely gone. Also noticing VM's are "snappier" and far more responsive in general.

Took a fair bit of effort for us but it was worth it, both to resolve the issue and get the benefit of Server 2022 hypervisor. Sleeping a bit easier at night now. A massive thanks for the Veeam support team for their work on this and with me, being a conduit to Microsoft to get it resolved.