-
- Service Provider
- Posts: 12
- Liked: 8 times
- Joined: Sep 14, 2016 12:04 pm
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
RCT problem will celebrate its 5th anniversary this year)
-
- Influencer
- Posts: 12
- Liked: 3 times
- Joined: Aug 27, 2019 8:55 am
- Full Name: LeslieUC
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Wow, 5th already? Time flies.
But still no patch from Microsoft?
Did veeam get an update already from MS about this problem?
But still no patch from Microsoft?
Did veeam get an update already from MS about this problem?
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hello,
I will confirm with the responsible person, but to the best of my knowledge, no updates so far. The recommendation remains the same: more users open up cases with Microsoft - higher changes the issue will get fixed (actually, this applies to any issue when the root cause lies on the Microsoft side).
Thanks
I will confirm with the responsible person, but to the best of my knowledge, no updates so far. The recommendation remains the same: more users open up cases with Microsoft - higher changes the issue will get fixed (actually, this applies to any issue when the root cause lies on the Microsoft side).
Thanks
-
- Enthusiast
- Posts: 36
- Liked: 4 times
- Joined: Jun 14, 2016 9:36 am
- Full Name: Pter Pumpkin
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Some (meaningless) updates from my MS case.
16/02/2021
17/03/2021
14/04/2021
It seems like he's just blowing me off...
In the meantime we continue to do a daily live migration of the affected VM which stops the issue from occurring.
16/02/2021
A private fix was tested in one of the customer’s environment, however that did not help and hence Internal Escalations were involved again.
Currently additional log collection is in progress and it may take some more time for next update.
We will keep you posted as soon as we get a final resolution/bug fix.
17/03/2021
Currently the PG team is working on performance testing by increasing the IOPS and trace collections, and it may take some more time.
I will email you once we have a final resolution/workaround for this bug fix.
14/04/2021
We are still waiting to hear from PG team.
There is no update on the final fix yet, and unfortunately no ETA as well.
We will keep you posted as soon as we get an update on the fix/resolution.
It seems like he's just blowing me off...
In the meantime we continue to do a daily live migration of the affected VM which stops the issue from occurring.
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
We had several customers test the fix, and it did not work. In my lab environment there was some improvement with the fix. We're waiting to hear back around what next steps are.
-
- Service Provider
- Posts: 12
- Liked: 8 times
- Joined: Sep 14, 2016 12:04 pm
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Veeam guys please port your own Veeam CBT driver to Server 2016 and later
It's the best fix you can do
It's the best fix you can do
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
I was told they are still working on further testing. Will keep you posted as we learn more.
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Since I had a few questions come up outside of the forums I figured I'd post an update. No changes yet, although it seems some progress has been made. They are waiting on further testing of changes to take place. At this point I don't know how long that will take. Until that happens I'd continue to recommend that if you're affected by this issue you open a ticket with Microsoft support.
-
- Veteran
- Posts: 643
- Liked: 312 times
- Joined: Aug 04, 2019 2:57 pm
- Full Name: Harvey
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Has Microsoft at least acknowledged this publicly somewhere? I came across a client this last week that was bit by this as best we can tell (with Veeam and another product they're using for their HyperV environment), and I'd say my team has spent most of the week convincing them that they need to talk to Microsoft, but the client refuses because they just see that it happens because of backups.
Is there maybe a bug posting from Microsoft I missed?
Is there maybe a bug posting from Microsoft I missed?
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi Harvey,
I do not think Microsoft has publicly available information about the RCT issue.
Thanks
I do not think Microsoft has publicly available information about the RCT issue.
Thanks
-
- Chief Product Officer
- Posts: 31806
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
@johan.h this seems like a reasonable ask. Can we ask our Microsoft contacts on this issue to create a support KB article?
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Yes, that seems very reasonable. I'll make that request.
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
A short update, I've asked for this to be documented. But also I've heard that a new private package may be available upon request for customers to try in the near term.
-
- Service Provider
- Posts: 276
- Liked: 61 times
- Joined: Nov 17, 2014 1:48 pm
- Full Name: Florin
- Location: Switzerland
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
I can confirm that there is a new private fix in the pipeline to be released in the next few weeks.
„The dev has created a fix that includes all the changes and in collaboration with their team for final approval. Once it is ready, will be working with WSD for backport and the private will be ready for final validation. Unfortunately, all of this will take some time and also I am out of office for next three weeks and will share an update once I return.“
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Jul 14, 2021 9:12 am
- Full Name: Andy
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hello everyone,
I would like to get involved here because we also have Event ID 9 problems.
Here is my course:
- comming from Hyper-V 2016, Dell VRTX, 4 Nodes (M630), an SSD CSV and an HDD CSV. There were no problems
- Update to Hyper-V 2019, no problems at the beginning
- Event ID 9 events occurred after the migration of some VMs
- If I change the VM allocation on the hosts through live migration, the VHDX / VHD reported in Event ID 9 also changes
For me it is the case that these events occur between 0:00 and 0:15 at night. There are no backup jobs during this period, not even any special highloads that is produced by the VM itself.
There are no events during business hours or backup runs.
Now I have a second piece of identical hardware. It only differs from the BIOS version (2.11 / 2.12) and the Windows patch level.
The cluster with the older patch level works without any problems without any events.
Now I am in a bind. I have a cluster with inexplicable storage latency and a cluster with no problems, but it needs to be patched and expanded.
Now I can only use both Hyper-V clusters to a very limited.
There are already Dell and Microsoft cases, but so far only logs have been collected or the problem has been pushed onto the hardware.
From my point of view, the behavior of my system fits some of the contributions that are described here.
What is your meaning? And are there any solution already in place I don´t know?
I would like to get involved here because we also have Event ID 9 problems.
Here is my course:
- comming from Hyper-V 2016, Dell VRTX, 4 Nodes (M630), an SSD CSV and an HDD CSV. There were no problems
- Update to Hyper-V 2019, no problems at the beginning
- Event ID 9 events occurred after the migration of some VMs
- If I change the VM allocation on the hosts through live migration, the VHDX / VHD reported in Event ID 9 also changes
For me it is the case that these events occur between 0:00 and 0:15 at night. There are no backup jobs during this period, not even any special highloads that is produced by the VM itself.
There are no events during business hours or backup runs.
Now I have a second piece of identical hardware. It only differs from the BIOS version (2.11 / 2.12) and the Windows patch level.
The cluster with the older patch level works without any problems without any events.
Now I am in a bind. I have a cluster with inexplicable storage latency and a cluster with no problems, but it needs to be patched and expanded.
Now I can only use both Hyper-V clusters to a very limited.
There are already Dell and Microsoft cases, but so far only logs have been collected or the problem has been pushed onto the hardware.
From my point of view, the behavior of my system fits some of the contributions that are described here.
What is your meaning? And are there any solution already in place I don´t know?
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi Andy, welcome to the forums! Event ID 9 is a symptom of I/O issues, which can be caused by various things. If the latency is entirely inexplicable then RCT may be a factor. Normally I would recommend that you have the same patch level on both boxes - it makes it easier to maintain. But you could check if there is a difference in the version of vhdmp.sys between systems.
Once you create a backup of virtual machines with RCT, at that point 3 bitmaps of CBT data data are created, in-memory, RCT and MRT. Maintaining these is where the performance overhead is. These bitmaps continue to exist, whether backup runs or not. How noticeable this effect is depends on the amount of ongoing I/O, what your VMs are doing and how many disks they have in ratio to the volumes they are stored on for example.
In the interim I've been told a KB article is being worked on and that the new private package is still in the works.
Once you create a backup of virtual machines with RCT, at that point 3 bitmaps of CBT data data are created, in-memory, RCT and MRT. Maintaining these is where the performance overhead is. These bitmaps continue to exist, whether backup runs or not. How noticeable this effect is depends on the amount of ongoing I/O, what your VMs are doing and how many disks they have in ratio to the volumes they are stored on for example.
In the interim I've been told a KB article is being worked on and that the new private package is still in the works.
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
So an update today. I've been given a new private fix to try. I'll provide an update here as soon as I have more.
-
- Influencer
- Posts: 12
- Liked: 3 times
- Joined: Aug 27, 2019 8:55 am
- Full Name: LeslieUC
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi @johan.h
Anymore info about the private fix you got from Microsoft?
Anymore info about the private fix you got from Microsoft?
-
- Influencer
- Posts: 19
- Liked: 18 times
- Joined: Jul 06, 2020 2:31 pm
- Contact:
-
- Enthusiast
- Posts: 76
- Liked: 16 times
- Joined: Oct 27, 2017 5:42 pm
- Full Name: Nick
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Well we found one way to “fix” this. For reasons unrelated we shut down all the VMs on the box over a month ago and the HV-Host hasn’t generated a single I/O Warning ever since... Imagine that! </s>
Almost 2 full years since I started this thread and reported this to Microsoft and they still haven’t fixed it! What a disheartening disgrace! I’ve got other clients who desperately need new Servers and I’ve been holding them off because I can’t in good conscience put them on Server 2019
Almost 2 full years since I started this thread and reported this to Microsoft and they still haven’t fixed it! What a disheartening disgrace! I’ve got other clients who desperately need new Servers and I’ve been holding them off because I can’t in good conscience put them on Server 2019
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Just reset my cluster, deploying VMFleet, running some baselines and then comparing with the new fix. I hope to have an update end of this week or early next.
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
My testing came up inconclusive unfortunately. I'd love to see if this works for someone else in a non-production environment. I used a Windows Server 2019, two node cluster, with the August Cumulative Updates installed. I was having some environmental stability issues, and didn't see improvements after installing the fix - however this could be localized to my environment. Feel free to DM me if you want to have access to the private fix.
-
- Influencer
- Posts: 19
- Liked: 18 times
- Joined: Jul 06, 2020 2:31 pm
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi johan
Which release of windows server 2019 do you have ? version 20H2 and release 19042.508.200927-1902 ?
Actually I can't install the patch but my version is 1809
Best Regards
Which release of windows server 2019 do you have ? version 20H2 and release 19042.508.200927-1902 ?
Actually I can't install the patch but my version is 1809
Best Regards
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
I have been using RS5 - 1809 17763.2114 with KB5005030 (2021-08 CU). The patch targets RS5. The hotfix didn't like the build number until the latest CU was installed.
-
- Influencer
- Posts: 19
- Liked: 18 times
- Joined: Jul 06, 2020 2:31 pm
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi
I just installed the patch on the first node of my cluster and it crashed my cluster. I can't stop the virtual machines on the patched node or do a live migrate.
The node is looping bluecreen so I had to uninstall the patch. It's a production farm, I can't risk having sudden downtime.
Best Regards
I just installed the patch on the first node of my cluster and it crashed my cluster. I can't stop the virtual machines on the patched node or do a live migrate.
The node is looping bluecreen so I had to uninstall the patch. It's a production farm, I can't risk having sudden downtime.
Best Regards
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
I'm sorry to hear that. At this point I think we can conclude that the patch is to blame. I've asked Microsoft for an update on this one.
-
- Service Provider
- Posts: 1
- Liked: never
- Joined: Oct 26, 2018 2:30 pm
- Full Name: Mark Seaton-Fry
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Now that server 2022 is out, does anyone know if the issue still occurs with that OS?
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Aug 26, 2021 12:49 am
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Should I not install Veeam? My server setup is similar to the original, with 2019 Windows Server running 2 Hyper V virtual machines (one VM is 2019 and one is 2016). I can install Veeam on another HyperV server running on the same 2019 physical server; or, I could install it on a separate physical machine running 2008 R2. However, from what I have read if I'm correct, the issue is that Resilient Changed Tracking (RCT) gets enabled anyway on the 2019 server in order to track block changes in VMs and would potentially impact the performance on the 2019 server.
Is the performance noted in the forums an issue for everyone in this configuration?
I am somewhat tentative about just installing and running it not knowing if my server’s performance will be impacted
My questions are (1) should I install Veeam and (2) if so, where should I install and run it? I appreciate any guidance.
Is the performance noted in the forums an issue for everyone in this configuration?
I am somewhat tentative about just installing and running it not knowing if my server’s performance will be impacted
My questions are (1) should I install Veeam and (2) if so, where should I install and run it? I appreciate any guidance.
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
I'm quite sure the issue still occurs in the latest version - due to the approach around the private hotfix, but I'm happy to test that and come back on this.
Who is online
Users browsing this forum: No registered users and 28 guests