Windows Server 2019 Hyper-V VM I/O Performance Problem

[rold]

RCT problem will celebrate its 5th anniversary this year)

[WesleyUC]

Wow, 5th already? Time flies.
But still no patch from Microsoft?
Did veeam get an update already from MS about this problem?

[wishr]

Hello,

I will confirm with the responsible person, but to the best of my knowledge, no updates so far. The recommendation remains the same: more users open up cases with Microsoft - higher changes the issue will get fixed (actually, this applies to any issue when the root cause lies on the Microsoft side).

Thanks

[pterpumpkin]

Some (meaningless) updates from my MS case.

16/02/2021

A private fix was tested in one of the customer’s environment, however that did not help and hence Internal Escalations were involved again.

Currently additional log collection is in progress and it may take some more time for next update.

We will keep you posted as soon as we get a final resolution/bug fix.

17/03/2021

Currently the PG team is working on performance testing by increasing the IOPS and trace collections, and it may take some more time.

I will email you once we have a final resolution/workaround for this bug fix.

14/04/2021

We are still waiting to hear from PG team.

There is no update on the final fix yet, and unfortunately no ETA as well.

We will keep you posted as soon as we get an update on the fix/resolution.

It seems like he's just blowing me off...

In the meantime we continue to do a daily live migration of the affected VM which stops the issue from occurring.

[johan.h]

We had several customers test the fix, and it did not work. In my lab environment there was some improvement with the fix. We're waiting to hear back around what next steps are.

[rold]

Veeam guys please port your own Veeam CBT driver to Server 2016 and later
It's the best fix you can do

[wishr]

Hi Rold,

Currently, we have no such plans, and this post pretty much explains why.

Thanks

[johan.h]

I was told they are still working on further testing. Will keep you posted as we learn more.

[johan.h]

Since I had a few questions come up outside of the forums I figured I'd post an update. No changes yet, although it seems some progress has been made. They are waiting on further testing of changes to take place. At this point I don't know how long that will take. Until that happens I'd continue to recommend that if you're affected by this issue you open a ticket with Microsoft support.

[soncscy]

Has Microsoft at least acknowledged this publicly somewhere? I came across a client this last week that was bit by this as best we can tell (with Veeam and another product they're using for their HyperV environment), and I'd say my team has spent most of the week convincing them that they need to talk to Microsoft, but the client refuses because they just see that it happens because of backups.

Is there maybe a bug posting from Microsoft I missed?

[wishr]

Hi Harvey,

I do not think Microsoft has publicly available information about the RCT issue.

Thanks

[Gostev]

@johan.h this seems like a reasonable ask. Can we ask our Microsoft contacts on this issue to create a support KB article?

[johan.h]

Yes, that seems very reasonable. I'll make that request.

[johan.h]

A short update, I've asked for this to be documented. But also I've heard that a new private package may be available upon request for customers to try in the near term.

[dasfliege]

I can confirm that there is a new private fix in the pipeline to be released in the next few weeks.

„The dev has created a fix that includes all the changes and in collaboration with their team for final approval. Once it is ready, will be working with WSD for backport and the private will be ready for final validation. Unfortunately, all of this will take some time and also I am out of office for next three weeks and will share an update once I return.“

[botti2]

Hello everyone,

I would like to get involved here because we also have Event ID 9 problems.

Here is my course:

- comming from Hyper-V 2016, Dell VRTX, 4 Nodes (M630), an SSD CSV and an HDD CSV. There were no problems
- Update to Hyper-V 2019, no problems at the beginning
- Event ID 9 events occurred after the migration of some VMs
- If I change the VM allocation on the hosts through live migration, the VHDX / VHD reported in Event ID 9 also changes

For me it is the case that these events occur between 0:00 and 0:15 at night. There are no backup jobs during this period, not even any special highloads that is produced by the VM itself.

There are no events during business hours or backup runs.

Now I have a second piece of identical hardware. It only differs from the BIOS version (2.11 / 2.12) and the Windows patch level.
The cluster with the older patch level works without any problems without any events.

Now I am in a bind. I have a cluster with inexplicable storage latency and a cluster with no problems, but it needs to be patched and expanded.
Now I can only use both Hyper-V clusters to a very limited.

There are already Dell and Microsoft cases, but so far only logs have been collected or the problem has been pushed onto the hardware.

From my point of view, the behavior of my system fits some of the contributions that are described here.

What is your meaning? And are there any solution already in place I don´t know?

[johan.h]

Hi Andy, welcome to the forums! Event ID 9 is a symptom of I/O issues, which can be caused by various things. If the latency is entirely inexplicable then RCT may be a factor. Normally I would recommend that you have the same patch level on both boxes - it makes it easier to maintain. But you could check if there is a difference in the version of vhdmp.sys between systems.

Once you create a backup of virtual machines with RCT, at that point 3 bitmaps of CBT data data are created, in-memory, RCT and MRT. Maintaining these is where the performance overhead is. These bitmaps continue to exist, whether backup runs or not. How noticeable this effect is depends on the amount of ongoing I/O, what your VMs are doing and how many disks they have in ratio to the volumes they are stored on for example.

In the interim I've been told a KB article is being worked on and that the new private package is still in the works.

[johan.h]

So an update today. I've been given a new private fix to try. I'll provide an update here as soon as I have more.

[WesleyUC]

Hi @johan.h

Anymore info about the private fix you got from Microsoft?

[Eluich]

So an update today. I've been given a new private fix to try. I'll provide an update here as soon as I have more.

Hi johan

I have also received the private fix but have not yet installed it. Have you done ? What is the result?

Best Regards

[Nick-SAC]

Well we found one way to “fix” this. For reasons unrelated we shut down all the VMs on the box over a month ago and the HV-Host hasn’t generated a single I/O Warning ever since... Imagine that! </s>

Almost 2 full years since I started this thread and reported this to Microsoft and they still haven’t fixed it! What a disheartening disgrace! I’ve got other clients who desperately need new Servers and I’ve been holding them off because I can’t in good conscience put them on Server 2019

[johan.h]

Just reset my cluster, deploying VMFleet, running some baselines and then comparing with the new fix. I hope to have an update end of this week or early next.

[johan.h]

My testing came up inconclusive unfortunately. I'd love to see if this works for someone else in a non-production environment. I used a Windows Server 2019, two node cluster, with the August Cumulative Updates installed. I was having some environmental stability issues, and didn't see improvements after installing the fix - however this could be localized to my environment. Feel free to DM me if you want to have access to the private fix.

[Eluich]

Hi johan

Which release of windows server 2019 do you have ? version 20H2 and release 19042.508.200927-1902 ?
Actually I can't install the patch but my version is 1809

Best Regards

[johan.h]

I have been using RS5 - 1809 17763.2114 with KB5005030 (2021-08 CU). The patch targets RS5. The hotfix didn't like the build number until the latest CU was installed.

[Eluich]

Hi

I just installed the patch on the first node of my cluster and it crashed my cluster. I can't stop the virtual machines on the patched node or do a live migrate.
The node is looping bluecreen so I had to uninstall the patch. It's a production farm, I can't risk having sudden downtime.

Best Regards

[johan.h]

I'm sorry to hear that. At this point I think we can conclude that the patch is to blame. I've asked Microsoft for an update on this one.

[MarkAlbacore]

Now that server 2022 is out, does anyone know if the issue still occurs with that OS?

[Jrsac]

Should I not install Veeam? My server setup is similar to the original, with 2019 Windows Server running 2 Hyper V virtual machines (one VM is 2019 and one is 2016). I can install Veeam on another HyperV server running on the same 2019 physical server; or, I could install it on a separate physical machine running 2008 R2. However, from what I have read if I'm correct, the issue is that Resilient Changed Tracking (RCT) gets enabled anyway on the 2019 server in order to track block changes in VMs and would potentially impact the performance on the 2019 server.

Is the performance noted in the forums an issue for everyone in this configuration?

I am somewhat tentative about just installing and running it not knowing if my server’s performance will be impacted

My questions are (1) should I install Veeam and (2) if so, where should I install and run it? I appreciate any guidance.

[johan.h]

I'm quite sure the issue still occurs in the latest version - due to the approach around the private hotfix, but I'm happy to test that and come back on this.