Host-based backup of Microsoft Hyper-V VMs.
Post Reply
rold
Service Provider
Posts: 11
Liked: 7 times
Joined: Sep 14, 2016 12:04 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by rold » 3 people like this post

RCT problem will celebrate its 5th anniversary this year)
WesleyUC
Influencer
Posts: 12
Liked: 3 times
Joined: Aug 27, 2019 8:55 am
Full Name: LeslieUC
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by WesleyUC »

Wow, 5th already? Time flies.
But still no patch from Microsoft?
Did veeam get an update already from MS about this problem?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

Hello,

I will confirm with the responsible person, but to the best of my knowledge, no updates so far. The recommendation remains the same: more users open up cases with Microsoft - higher changes the issue will get fixed (actually, this applies to any issue when the root cause lies on the Microsoft side).

Thanks
pterpumpkin
Enthusiast
Posts: 36
Liked: 4 times
Joined: Jun 14, 2016 9:36 am
Full Name: Pter Pumpkin
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by pterpumpkin »

Some (meaningless) updates from my MS case.

16/02/2021
A private fix was tested in one of the customer’s environment, however that did not help and hence Internal Escalations were involved again.

Currently additional log collection is in progress and it may take some more time for next update.

We will keep you posted as soon as we get a final resolution/bug fix.

17/03/2021
Currently the PG team is working on performance testing by increasing the IOPS and trace collections, and it may take some more time.

I will email you once we have a final resolution/workaround for this bug fix.


14/04/2021
We are still waiting to hear from PG team.

There is no update on the final fix yet, and unfortunately no ETA as well.

We will keep you posted as soon as we get an update on the fix/resolution.

It seems like he's just blowing me off...

In the meantime we continue to do a daily live migration of the affected VM which stops the issue from occurring.
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h » 2 people like this post

We had several customers test the fix, and it did not work. In my lab environment there was some improvement with the fix. We're waiting to hear back around what next steps are.
rold
Service Provider
Posts: 11
Liked: 7 times
Joined: Sep 14, 2016 12:04 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by rold »

Veeam guys please port your own Veeam CBT driver to Server 2016 and later
It's the best fix you can do :)
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

Hi Rold,

Currently, we have no such plans, and this post pretty much explains why.

Thanks
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h » 1 person likes this post

I was told they are still working on further testing. Will keep you posted as we learn more.
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h » 1 person likes this post

Since I had a few questions come up outside of the forums I figured I'd post an update. No changes yet, although it seems some progress has been made. They are waiting on further testing of changes to take place. At this point I don't know how long that will take. Until that happens I'd continue to recommend that if you're affected by this issue you open a ticket with Microsoft support.
soncscy
Veteran
Posts: 643
Liked: 312 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by soncscy »

Has Microsoft at least acknowledged this publicly somewhere? I came across a client this last week that was bit by this as best we can tell (with Veeam and another product they're using for their HyperV environment), and I'd say my team has spent most of the week convincing them that they need to talk to Microsoft, but the client refuses because they just see that it happens because of backups.

Is there maybe a bug posting from Microsoft I missed?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

Hi Harvey,

I do not think Microsoft has publicly available information about the RCT issue.

Thanks
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Gostev » 1 person likes this post

@johan.h this seems like a reasonable ask. Can we ask our Microsoft contacts on this issue to create a support KB article?
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

Yes, that seems very reasonable. I'll make that request.
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

A short update, I've asked for this to be documented. But also I've heard that a new private package may be available upon request for customers to try in the near term.
dasfliege
Service Provider
Posts: 233
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by dasfliege » 1 person likes this post

I can confirm that there is a new private fix in the pipeline to be released in the next few weeks.
„The dev has created a fix that includes all the changes and in collaboration with their team for final approval. Once it is ready, will be working with WSD for backport and the private will be ready for final validation. Unfortunately, all of this will take some time and also I am out of office for next three weeks and will share an update once I return.“
botti2
Lurker
Posts: 1
Liked: never
Joined: Jul 14, 2021 9:12 am
Full Name: Andy
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by botti2 »

Hello everyone,

I would like to get involved here because we also have Event ID 9 problems.

Here is my course:

- comming from Hyper-V 2016, Dell VRTX, 4 Nodes (M630), an SSD CSV and an HDD CSV. There were no problems
- Update to Hyper-V 2019, no problems at the beginning
- Event ID 9 events occurred after the migration of some VMs
- If I change the VM allocation on the hosts through live migration, the VHDX / VHD reported in Event ID 9 also changes

For me it is the case that these events occur between 0:00 and 0:15 at night. There are no backup jobs during this period, not even any special highloads that is produced by the VM itself.

There are no events during business hours or backup runs.

Now I have a second piece of identical hardware. It only differs from the BIOS version (2.11 / 2.12) and the Windows patch level.
The cluster with the older patch level works without any problems without any events.

Now I am in a bind. I have a cluster with inexplicable storage latency and a cluster with no problems, but it needs to be patched and expanded.
Now I can only use both Hyper-V clusters to a very limited.

There are already Dell and Microsoft cases, but so far only logs have been collected or the problem has been pushed onto the hardware.

From my point of view, the behavior of my system fits some of the contributions that are described here.

What is your meaning? And are there any solution already in place I don´t know?
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h » 1 person likes this post

Hi Andy, welcome to the forums! Event ID 9 is a symptom of I/O issues, which can be caused by various things. If the latency is entirely inexplicable then RCT may be a factor. Normally I would recommend that you have the same patch level on both boxes - it makes it easier to maintain. But you could check if there is a difference in the version of vhdmp.sys between systems.

Once you create a backup of virtual machines with RCT, at that point 3 bitmaps of CBT data data are created, in-memory, RCT and MRT. Maintaining these is where the performance overhead is. These bitmaps continue to exist, whether backup runs or not. How noticeable this effect is depends on the amount of ongoing I/O, what your VMs are doing and how many disks they have in ratio to the volumes they are stored on for example.

In the interim I've been told a KB article is being worked on and that the new private package is still in the works.
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

So an update today. I've been given a new private fix to try. I'll provide an update here as soon as I have more.
WesleyUC
Influencer
Posts: 12
Liked: 3 times
Joined: Aug 27, 2019 8:55 am
Full Name: LeslieUC
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by WesleyUC »

Hi @johan.h

Anymore info about the private fix you got from Microsoft?
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 1 person likes this post

johan.h wrote: Aug 04, 2021 5:05 am So an update today. I've been given a new private fix to try. I'll provide an update here as soon as I have more.
Hi johan

I have also received the private fix but have not yet installed it. Have you done ? What is the result?

Best Regards
Nick-SAC
Enthusiast
Posts: 73
Liked: 15 times
Joined: Oct 27, 2017 5:42 pm
Full Name: Nick
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Nick-SAC » 4 people like this post

Well we found one way to “fix” this. For reasons unrelated we shut down all the VMs on the box over a month ago and the HV-Host hasn’t generated a single I/O Warning ever since... Imagine that! </s>

Almost 2 full years since I started this thread and reported this to Microsoft and they still haven’t fixed it! What a disheartening disgrace! I’ve got other clients who desperately need new Servers and I’ve been holding them off because I can’t in good conscience put them on Server 2019 :x
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

Just reset my cluster, deploying VMFleet, running some baselines and then comparing with the new fix. I hope to have an update end of this week or early next.
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

My testing came up inconclusive unfortunately. I'd love to see if this works for someone else in a non-production environment. I used a Windows Server 2019, two node cluster, with the August Cumulative Updates installed. I was having some environmental stability issues, and didn't see improvements after installing the fix - however this could be localized to my environment. Feel free to DM me if you want to have access to the private fix.
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich »

Hi johan

Which release of windows server 2019 do you have ? version 20H2 and release 19042.508.200927-1902 ?
Actually I can't install the patch but my version is 1809

Best Regards
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

I have been using RS5 - 1809 17763.2114 with KB5005030 (2021-08 CU). The patch targets RS5. The hotfix didn't like the build number until the latest CU was installed.
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 1 person likes this post

Hi

I just installed the patch on the first node of my cluster and it crashed my cluster. I can't stop the virtual machines on the patched node or do a live migrate.
The node is looping bluecreen so I had to uninstall the patch. It's a production farm, I can't risk having sudden downtime.

Best Regards
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

I'm sorry to hear that. At this point I think we can conclude that the patch is to blame. I've asked Microsoft for an update on this one.
MarkAlbacore
Lurker
Posts: 1
Liked: never
Joined: Oct 26, 2018 2:30 pm
Full Name: Mark Seaton-Fry
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by MarkAlbacore »

Now that server 2022 is out, does anyone know if the issue still occurs with that OS?
Jrsac
Lurker
Posts: 2
Liked: never
Joined: Aug 26, 2021 12:49 am
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Jrsac »

Should I not install Veeam? My server setup is similar to the original, with 2019 Windows Server running 2 Hyper V virtual machines (one VM is 2019 and one is 2016). I can install Veeam on another HyperV server running on the same 2019 physical server; or, I could install it on a separate physical machine running 2008 R2. However, from what I have read if I'm correct, the issue is that Resilient Changed Tracking (RCT) gets enabled anyway on the 2019 server in order to track block changes in VMs and would potentially impact the performance on the 2019 server.

Is the performance noted in the forums an issue for everyone in this configuration?

I am somewhat tentative about just installing and running it not knowing if my server’s performance will be impacted

My questions are (1) should I install Veeam and (2) if so, where should I install and run it? I appreciate any guidance.
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

I'm quite sure the issue still occurs in the latest version - due to the approach around the private hotfix, but I'm happy to test that and come back on this.
Post Reply

Who is online

Users browsing this forum: No registered users and 13 guests