Discussions specific to the Microsoft Hyper-V hypervisor
rold
Service Provider
Posts: 5
Liked: 3 times
Joined: Sep 14, 2016 12:04 pm

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by rold » 3 people like this post

RCT problem will celebrate its 5th anniversary this year)

WesleyUC
Influencer
Posts: 11
Liked: 3 times
Joined: Aug 27, 2019 8:55 am
Full Name: LeslieUC
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by WesleyUC »

Wow, 5th already? Time flies.
But still no patch from Microsoft?
Did veeam get an update already from MS about this problem?

wishr
Veeam Software
Posts: 2544
Liked: 334 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

Hello,

I will confirm with the responsible person, but to the best of my knowledge, no updates so far. The recommendation remains the same: more users open up cases with Microsoft - higher changes the issue will get fixed (actually, this applies to any issue when the root cause lies on the Microsoft side).

Thanks

pterpumpkin
Enthusiast
Posts: 35
Liked: 4 times
Joined: Jun 14, 2016 9:36 am
Full Name: Pter Pumpkin
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by pterpumpkin »

Some (meaningless) updates from my MS case.

16/02/2021
A private fix was tested in one of the customer’s environment, however that did not help and hence Internal Escalations were involved again.

Currently additional log collection is in progress and it may take some more time for next update.

We will keep you posted as soon as we get a final resolution/bug fix.

17/03/2021
Currently the PG team is working on performance testing by increasing the IOPS and trace collections, and it may take some more time.

I will email you once we have a final resolution/workaround for this bug fix.


14/04/2021
We are still waiting to hear from PG team.

There is no update on the final fix yet, and unfortunately no ETA as well.

We will keep you posted as soon as we get an update on the fix/resolution.

It seems like he's just blowing me off...

In the meantime we continue to do a daily live migration of the affected VM which stops the issue from occurring.

johan.h
Veeam Software
Posts: 626
Liked: 139 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h » 2 people like this post

We had several customers test the fix, and it did not work. In my lab environment there was some improvement with the fix. We're waiting to hear back around what next steps are.

rold
Service Provider
Posts: 5
Liked: 3 times
Joined: Sep 14, 2016 12:04 pm

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by rold »

Veeam guys please port your own Veeam CBT driver to Server 2016 and later
It's the best fix you can do :)

wishr
Veeam Software
Posts: 2544
Liked: 334 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

Hi Rold,

Currently, we have no such plans, and this post pretty much explains why.

Thanks

johan.h
Veeam Software
Posts: 626
Liked: 139 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h » 1 person likes this post

I was told they are still working on further testing. Will keep you posted as we learn more.

johan.h
Veeam Software
Posts: 626
Liked: 139 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h » 1 person likes this post

Since I had a few questions come up outside of the forums I figured I'd post an update. No changes yet, although it seems some progress has been made. They are waiting on further testing of changes to take place. At this point I don't know how long that will take. Until that happens I'd continue to recommend that if you're affected by this issue you open a ticket with Microsoft support.

soncscy
Veeam Legend
Posts: 436
Liked: 209 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey Carel
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by soncscy »

Has Microsoft at least acknowledged this publicly somewhere? I came across a client this last week that was bit by this as best we can tell (with Veeam and another product they're using for their HyperV environment), and I'd say my team has spent most of the week convincing them that they need to talk to Microsoft, but the client refuses because they just see that it happens because of backups.

Is there maybe a bug posting from Microsoft I missed?

wishr
Veeam Software
Posts: 2544
Liked: 334 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

Hi Harvey,

I do not think Microsoft has publicly available information about the RCT issue.

Thanks

Gostev
SVP, Product Management
Posts: 28676
Liked: 5188 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Gostev » 1 person likes this post

@johan.h this seems like a reasonable ask. Can we ask our Microsoft contacts on this issue to create a support KB article?

johan.h
Veeam Software
Posts: 626
Liked: 139 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

Yes, that seems very reasonable. I'll make that request.

johan.h
Veeam Software
Posts: 626
Liked: 139 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

A short update, I've asked for this to be documented. But also I've heard that a new private package may be available upon request for customers to try in the near term.

dasfliege
Service Provider
Posts: 140
Liked: 38 times
Joined: Nov 17, 2014 1:48 pm
Location: Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by dasfliege » 1 person likes this post

I can confirm that there is a new private fix in the pipeline to be released in the next few weeks.
„The dev has created a fix that includes all the changes and in collaboration with their team for final approval. Once it is ready, will be working with WSD for backport and the private will be ready for final validation. Unfortunately, all of this will take some time and also I am out of office for next three weeks and will share an update once I return.“

botti2
Lurker
Posts: 1
Liked: never
Joined: Jul 14, 2021 9:12 am
Full Name: Andy
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by botti2 »

Hello everyone,

I would like to get involved here because we also have Event ID 9 problems.

Here is my course:

- comming from Hyper-V 2016, Dell VRTX, 4 Nodes (M630), an SSD CSV and an HDD CSV. There were no problems
- Update to Hyper-V 2019, no problems at the beginning
- Event ID 9 events occurred after the migration of some VMs
- If I change the VM allocation on the hosts through live migration, the VHDX / VHD reported in Event ID 9 also changes

For me it is the case that these events occur between 0:00 and 0:15 at night. There are no backup jobs during this period, not even any special highloads that is produced by the VM itself.

There are no events during business hours or backup runs.

Now I have a second piece of identical hardware. It only differs from the BIOS version (2.11 / 2.12) and the Windows patch level.
The cluster with the older patch level works without any problems without any events.

Now I am in a bind. I have a cluster with inexplicable storage latency and a cluster with no problems, but it needs to be patched and expanded.
Now I can only use both Hyper-V clusters to a very limited.

There are already Dell and Microsoft cases, but so far only logs have been collected or the problem has been pushed onto the hardware.

From my point of view, the behavior of my system fits some of the contributions that are described here.

What is your meaning? And are there any solution already in place I don´t know?

johan.h
Veeam Software
Posts: 626
Liked: 139 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h » 1 person likes this post

Hi Andy, welcome to the forums! Event ID 9 is a symptom of I/O issues, which can be caused by various things. If the latency is entirely inexplicable then RCT may be a factor. Normally I would recommend that you have the same patch level on both boxes - it makes it easier to maintain. But you could check if there is a difference in the version of vhdmp.sys between systems.

Once you create a backup of virtual machines with RCT, at that point 3 bitmaps of CBT data data are created, in-memory, RCT and MRT. Maintaining these is where the performance overhead is. These bitmaps continue to exist, whether backup runs or not. How noticeable this effect is depends on the amount of ongoing I/O, what your VMs are doing and how many disks they have in ratio to the volumes they are stored on for example.

In the interim I've been told a KB article is being worked on and that the new private package is still in the works.

Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests