Windows Server 2019 Hyper-V VM I/O Performance Problem

[joelg]

No particular details. Just that it is a private fix that potentially resolved the RCT I/O issues. It can be installed/uninstalled on Server 2019.

I haven't reproduced the error yet on my test environment. We ran a backup yesterday and the MRT/RCT files were created, now trying to get that error to pop up before testing the fix.

Joel

[pterpumpkin]

My MS case was updated to say KB5022286 resolves the issue too. We're yet to test.

[karsten123]

Can anyone else confirm that this update will fix the issue?

[BigBadBarry]

I'll be installing KB5022286 and KB900379 from MSFT on a production server tonight so should hopefully be able to say at some point tomorrow.
No test environment so please say a prayer for me

[karsten123]

good luck

[alexandre.rogoski]

Not fixed yet for me.

[BigBadBarry]

KB5022286 is installing now though I believe Steph has already confirmed this does not contain the fix, I'm unable to install KB900379 this evening so will re-try in person tomorrow evening as host requires some BIOS changes (If anyone has this update but has ran out of time you can set build date to later in privatepkgdata)

[BigBadBarry]

Initial testing looks better with KB900379 so far but will only know for sure tomorrow morning after backups have been completed

[steendp]

Is KB900379 the private hf?

[peno@edgemo.com]

Hi there
we have been struggling with a similar problem on Hyper-V 2019 - and bad performance on VMs running on that. (precisely by a factor 10 degradation on I/O towards storage)
For the moment we only see the problem on especially windows server 2012 R2 as Guest OS on VMs - and them running SQL 2016 - and ONLY on the drive where the Databases are stored.

Could some of you please specify which Guest OS and server types you are running, experiencing this particular I/O performance degradation
(Guest OS, application etc and any other relation towards this issue)

Thanks

[BigBadBarry]

I have a few DL360 Gen10 8SFF with SRV 2019 installed, the servers with DB's are SQL 2019 (CSV on a HPE MSA 2062)
KB900379 is the private hf, seems to resolve the issue for me but need to test further before rolling out to the other hosts.

[peno@edgemo.com]

Hi Barry
So the problem you had was also related to or around SQL servers in general, giving you bad performance on I/O - storage ??

You have Hyper-V 2019 installed on Host - and then running SQL server VMs/Guests on hyper-V - right ?
- Was there any particular drives that had this issue on the VMs / SQL server - or just all drives ?
- Where did you experience this performance degradation ?
- Have you run a Image or application aware Backup after the KB900379 was applied - and tested if the problem is there or not?
- what tool do you test with ? (we are using DiskSpd)

Please share as much info you can, as we are working on the same issue
Thanks

[nexusdis]

Hi,
where can I get KB900379 ?
Only from MS support?

[stephc_msft]

KB900379 is the test fix (and that is just a temporary KB number, it will not called that if/when gets rolled up into an update)
Ping me if you want to try it (ws2019/ws2022), but note this does not guarantee it will become an official fix.

The above is for the RCT issue
Also note, there seems to be a similar issue where significant io degradation can occur even without RCT being used.
And again the only way to 'clear' the host problem state is to reboot the host or live migrate the affected VM to a 'fresh' host.
This is also being investigated.

[steendp]

We seem to be experiencing the issue you describe. RCT has been disabled, we have cleaned up the rct/mrt files but am still experiencing disk latency issues that is resolved by live migration.

[stephc_msft]

FYI if you PM me for the test fix, please supply your full email and/or ensure you accept private messages
else I see
"Some users couldn’t be added as they have disabled private message receipt"

[karsten123]

do server 2022 also suffer from this bug?

[stephc_msft]

RE WS2022. My understanding is some people are also seeing the issue on WS2022 host (IO degradation after Veeam backup, either due to RCT or due to some other similar issue that is NOT RCT related). But hopefully others can confirm here.
Also is anyone seeing it with WS2016 host?
WS2012R2 host does not have any issue according to a few people on here.

[rold]

Also note, there seems to be a similar issue where significant io degradation can occur even without RCT being used.
And again the only way to 'clear' the host problem state is to reboot the host or live migrate the affected VM to a 'fresh' host.
This is also being investigated.

This issue looks like a bug in the code. If the backup runs longer than some time, disk IO becomes "locked" to queue depth = 1.
We use Scale-Out File Server and this can be reproduced without Veeam - just make a snapshot and simply copy the .vhdx disk file anywhere. Then remove the snapshot. If the disk file is large enough, then this problem will appear.

[exstyle]

We have been encountering this issue for sometime on 2019 as well, same as everyone else, a live migration of the VM immediately fixes the issue.

Can you share the hotfix with me? Thank you!

[nielsengelen]

Microsoft is aware of the issue and should have a fix.

Please open a support case at Microsoft for assistance and to receive the hotfix if it applies to your problem.

Thanks!

[peno@edgemo.com]

We have tried ms hotfix and it doesn´t resolve the issue

[stephc_msft]

Just a reminder, there seem to be two separate, distinct issues, although seem very similar and somewhat related.
1. Issue with RCT leading to some serialization of io
2. Issue with IO slowdown after veeam backup even without RCT [and also possibly triggered by other things involving checkpoints][and only on ceratin disks with certain workloads]

In fact most people are probably hitting issue2, and this is the one live migrating to a fresh host works around

A characteristic of Issue 2 seems to be the Queue Length in the problem state shows as very low.
[as per the user 'rold' 's finding above]

If anyone can perfmon their HV host and look at physicaldisk counter for the relevant disk (or csv) and check on the Avg. Disk Write Queue Length, it would be useful.
This might confirm a highish number (20+) in good state, and bad a number (0 or 1) in bad state.
Exactly why queue depth usage is getting affected in the problem state is being investigated.

[stephc_msft]

Correction to the above re queue length
The queue length effect will only be noticeable if the disk is mainly used by VM's that are affected by the issue.
If there are other VM's that are not affected by the issue using the same host disk or host csv, then they might keep the queue length figure up
ie may not see so big a drop in queue length.

[soncscy]

Hi Stephen,

To preface this, I'm happy to see (who I presume) is a Microsoft representative finally addressing this, I'm afraid I've got a ton of questions about Microsoft Support and these RCT issues.

I manage quite a few clients that are all-in on HyperV, and for the last couple of years it's been a struggle to even get the Microsoft Support to acknowledge this issue, despite countless reference cases posted in this topic, demonstrable proof across multiple backup vendors, etc.

This ended up eating weeks (and in some cases months) of chasing Microsoft Support both for my clients and for my team. I understand you might not be directly related to Microsoft Support, but to see you suddenly active in this topic is a good surprise, I have a few practical questions I'm hoping you can address.

1. If one of my clients looks to be affected by this issue, what is the absolute best way to get Microsoft Support to acknowledge and research the issue? As soon as a backup solution (Veeam, Commvault, etc) is mentioned, the cases my clients and my team open get closed or indefinitely delayed citing "contact the backup vendor". Absolutely useless suggestion which only convinces my clients that Microsoft Support didn't even read the case.

Your steps above are fine for me and my team, but my team's and my time is billable and I cannot in good faith ask my clients to spend money on what I expect Microsoft Support to be handling, so I really would like to know how can we get Microsoft Support to take ownership of these two issues you mention.

2. Do you have any ETA on when a patch for the two issues you mentioned will appear? The second issue I see the symptoms of even in non-Veeam-protected environments, so I find the phrasing you use very "interesting". (Please do not understand this as offensive, but your answer leads to more questions, not understanding)

3. Why is this discussion happening on a backup application forum? I understand that backup applications might exacerbate the issue, but as I understand it from your descriptions and the testing done in this thread, this is a HyperV issue. Backup Applications might trigger the issue, but they aren't inherently the root cause. Searching on google for variations on HyperV 2019 VM performance, and this topic is usually in the top 3 results; why has Microsoft stayed so silent on this?

Again, please don't take this as a personal attack; I'm happy that _anyone_ from Microsoft is finally discussing the issue, but I've got clients that refuse to open Microsoft cases because of this issue they've been dealing with for years, only to get their support case closed out and to be told "ask Veeam/Symantec/Dell/whoever". I can say with full confidence that at least a few clients moved away from Microsoft virtual_anything because of this issue and the lack of even acknowledgment over several years from Microsoft.

I understand you maybe cannot answer all of these questions, but if you can find a way to get some statement on these items, it would be a nice starting point for many of my clients that got slammed by these issues and are only just now getting some closure.

[kevin.boddy]

RE WS2022. My understanding is some people are also seeing the issue on WS2022 host (IO degradation after Veeam backup, either due to RCT or due to some other similar issue that is NOT RCT related). But hopefully others can confirm here.
Also is anyone seeing it with WS2016 host?
WS2012R2 host does not have any issue according to a few people on here.

Hi,

I have seen similar issues with customers who have large and very busy SQL servers on WS2016.
When the backup has been running for a long time e.g. 8 hours and the checkpoint file has grown to over a TB, then the VMs disk IO performance drops so much that the VM starts becoming unresponsive.

The moment the backup is stopped, and the checkpoint removal process starts, then the VMs disk IO performance issue is resolved.
Even while the host is still merging the 1TB+ checkpoint file, which is obviously consuming a lot of host disk IO, the VM disk IO performance is normal.

What I would dearly wish for is a way for Veeam to do storage snapshot backups where the checkpoint is immediately removed after the storage snapshot is taken. That would solve the problem for me.

Thanks
Kevin

[bhead]

Hello everyone,

in our case KB5022840 (OS Build 17763.4010) did not resolve the issue.
We just rolled out the update to one of our nodes and ran some tests. After creating a backup, the IOs inside the VM drop to a crawling speed!

I still have faith though. The MS technician we talked to in our support-case did say that the fix will most likely be published in march 2023.
Let's hope that this issue will be gone in march.

Has anybody taken a look at Veeam V.12 yet? Maybe, despite all other facts, the new vesion does not cause the issue anymore?
I will wait for a couple of weeks before I roll out the new version in our datacenter!

[RexfordHaugen_COLT]

RE WS2022. My understanding is some people are also seeing the issue on WS2022 host (IO degradation after Veeam backup, either due to RCT or due to some other similar issue that is NOT RCT related). But hopefully others can confirm here.

Hello Stephen,

We are running WS2022DC in two Hyper-V Failover Clusters and we are seeing the performance issues. Happy to roll out KB900379 to one of the clusters for testing since we can move high priority VMs to the other stack.

Rexford

[heiko.wundram]

Please open a support case for assistance and to receive the hotfix if it applies to your problem. Thanks!

We're hitting this problem with nested Hyper-Vs and Server 2019, where the hosts running in the nested Hyper-V are the ones backed up by Veeam and migrating those to another nested Hyper-V resolves the I/O problems, so that it seems that this is the RCT issue. We'd like to test the hotfix in the nested Hyper-V. How do we go about getting access to it?

[nielsengelen]

Hi Heiko,

Seems I wasn’t 100% clear for this, apologies.

Microsoft is aware of the issue and should have a fix.

Please open a support case at Microsoft for assistance and to receive the hotfix if it applies to your problem.

Thanks!