Discussions specific to the Microsoft Hyper-V hypervisor
ChristineAlexa
Influencer
Posts: 24
Liked: 5 times
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

WeasleyUC
There are two issues going on here, for some hardware, See my testing results above.
And agreed, Neither of the issues are due to Veeam's code. Also agreed, the RCT one affects all vendors' products (that I'm aware of) using that technology.

Christine.

bhead
Lurker
Posts: 1
Liked: never
Joined: Sep 30, 2020 9:18 am
Full Name: Bjoern Goerlich
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by bhead »

Hello everybody,

we're currently dealing with the exact same issue at one of our datacenter-sites:

Hyper-V 2019 Failover-Cluster with a total of 9 nodes.
The nodes are connected to the storage (All-Flash) via ISCSI on a 10 GBit/s network.
IO Performance randomly drops to nearly a complete stall.
Doing some test on any node, writing and reading data from the CSV, everything looks fine. Our network and our storage are as fast it gets.
Doing the same tests inside a random VM, writing data to and from a virtual hard disk (dynamic vhdx / static vhdx with a fixed size) will result in about 10% of the IO that we see on the parent partition.

We're currently dealing with the Microsoft Support. Right now they don't seem to have an idea on what's causing the latency and the low IO performance.

@pterpumkin: We would like to submit your MS-Ticket ID to our support contact. Maybe they can make the connection here to make some progress on both our cases? Maybe you could contact me via PM?

ChristineAlexa
Influencer
Posts: 24
Liked: 5 times
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

bhead:
Please see my reply earlier in this post, post383997.html#p383997

But this sounds more like the performance issue inside a HyperV VM that I documented with solutions for some hardware there with Hyperthreading vs the CBT/RCT bug that we are discussing as well. Note solving the performance in a VM does NOT solve the CBT/RCT bug, but on some hardware (all that I have personally tested), makes it happen far less frequently.

Just for a data points
- What vendor and model are the servers?
- What disk controllers and bus type are you using?
- What brand and model are the flash devices.

Thanks,
Christine

CasperGN
Novice
Posts: 5
Liked: never
Joined: Sep 29, 2020 11:59 am
Full Name: Casper Glasius-Nyborg
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by CasperGN »

I got ticket numbers from Pterpumkin.

0. 120060926001170 - Pter's own ticket.
1. 120063025001053
2. 120081024004251
3. 120040625000059
4. 119050122000550 - Veeam ticket number
5. 119100924004286
6. 120052425000074 – other backup vendor then Veeam, but same issue with performance.
7. 120081221000284
8. 120040625000059 - My own ticket which im pushing heavy on.

/Casper

CasperGN
Novice
Posts: 5
Liked: never
Joined: Sep 29, 2020 11:59 am
Full Name: Casper Glasius-Nyborg
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by CasperGN »

Hi All

Im still pushing onward with MS, only got them to say that they have escalated this internally and are awaiting updates.
Has any of you gotten any positiv outcome from your MS cases ?

/Casper

mkaec
Expert
Posts: 351
Liked: 81 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by mkaec »

Casper,

Are you dealing with actual MS support or outsourced support? I've had a few incidents with outsourced support and they tried really hard to bury the case each time. With my last case, MS (appears to have) accidentally released the fix I was asking for. I asked the support rep to confirm the internal case he was referring to confirmed the fix and he ghosted me.

scootrz32
Novice
Posts: 4
Liked: never
Joined: Nov 17, 2009 2:25 am
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by scootrz32 »

Have you found any correlation. Like we see the issue and have Dell 14g servers with H740p controllers

ChristineAlexa
Influencer
Posts: 24
Liked: 5 times
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

scootrz32:
Have you looked at my research earlier in the post? Though all of my testing is with 11g, 12g, and 13g Dell servers (and the appropriate PERC controllers for the generations)

scootrz32
Novice
Posts: 4
Liked: never
Joined: Nov 17, 2009 2:25 am
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by scootrz32 »

I think I resolved ours today. I am going to test for a couple days. It was SentinelOne AV causing our issue. Once I removed it, it appears to be working.

Anyone else running S1?

CasperGN
Novice
Posts: 5
Liked: never
Joined: Sep 29, 2020 11:59 am
Full Name: Casper Glasius-Nyborg
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by CasperGN »

Hi

@mkaec Im dealing with MS directly.
@Scootrz32 We have SentinelOne AV, but we havent installed them on our HyperV 2019 hosts or this VM. Yesterday we moved one of the most affected vm's to another environment we have which is running Hyper V 2016, at the moment that setup is not used so much, so the vm have plenty of power and at the moment everything looks very good.
I noticed another thing yesterday during our storage migration for the move to Hyper V 2016, the VM's Disk Queue and latency dropped to almost nothing just when the storage migration finished. The VM where still on the same host and same storage, just a new storage volume. Im currently talking with our Storage vendor Infinidat on this issue. Because it looked very strange.

/Casper

scootrz32
Novice
Posts: 4
Liked: never
Joined: Nov 17, 2009 2:25 am
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by scootrz32 »

Once I removed it from the virtual machine. Mine seems to take snapshots and work OK.

Eluich
Novice
Posts: 6
Liked: 4 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich »

Hi

Do you have news on your Microsoft cases ?
On my side, no significant progress

Best Regards

pterpumpkin
Enthusiast
Posts: 34
Liked: 4 times
Joined: Jun 14, 2016 9:36 am
Full Name: Pter Pumpkin
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by pterpumpkin »

Nothing yet :(

CasperGN
Novice
Posts: 5
Liked: never
Joined: Sep 29, 2020 11:59 am
Full Name: Casper Glasius-Nyborg
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by CasperGN »

My MS ticket was incorrectly closed with info that they are working on a fix, but the issue is different from what we are experiencing. But it might help others.

MS Comment
-------------------------
The instructions that my colleague Jose Ramon sent to you were to rollback and uninstall the updates since August 2020 on the Hypervisor (Not the virtual machine):

In this link you can see a discussion about this same issue that affects to backup and replication tasks.
https://docs.microsoft.com/en-us/answer ... er-au.html

The reason for that was that the team is working on a fix that will be released in the future. Currently we cannot provide a time when this will be available so you will have to wait for it. For that, and to try to mitigate the issue, my colleague suggested to delete the cumulative updates after KB4559003 (https://support.microsoft.com/en-in/help/4559003) (Do not uninstall this KB).
--------------------------

But we dont have that patch and doesnt see performance degradation doing image backup, but afterwards. But MS didnt seem to understand that, the guy who worked on our case, kept on troubleshooting as it is a VM issue.

Hope it help some of you.

yoda-ict
Lurker
Posts: 1
Liked: never
Joined: Dec 28, 2017 1:58 pm
Full Name: Frans Rampen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by yoda-ict »

Any news about this issue? Disabling Hyper-V helps but after a while we get a lot of StorageVSP Events nr 9 with description:

An I/O request for device 'C:\ClusterStorage\Volume7\ServerName\Virtual Hard Disks\Name.vhdx' took 18150 milliseconds to complete. Operation code = WRITE, Data transfer length = 524288, Status = SRB_STATUS_SUCCESS.

pterpumpkin
Enthusiast
Posts: 34
Liked: 4 times
Joined: Jun 14, 2016 9:36 am
Full Name: Pter Pumpkin
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by pterpumpkin »

MS closed my case and left us without a fix. They did this as they said it has been identified as a bug and there are no further troubleshooting steps, so there is no need to have a case open *sigh*.

I am emailing them weekly to ask if there is any update on the bug fix. This seems to be going nowhere and i have a feeling they don't actually check, they just say "no" and that they will let me know when a fix becomes available.

I have just tried deleting all of the RCT points for our affected VM and disabled our automatic daily live migration pre-emptive fix we had in place. The issue normally takes a few days to a few weeks to occur, so i will let you know how it goes.

Eluich
Novice
Posts: 6
Liked: 4 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 1 person likes this post

On my side, the case is still open, normally a patch is in production but the person responsible for my case told me it could take several weeks

Eluich
Novice
Posts: 6
Liked: 4 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 1 person likes this post

Small update, I just got Microsoft. The patch has been created. They are finishing testing it on their lab before releasing it.

wishr
Veeam Software
Posts: 1928
Liked: 230 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

Hi Eluich,

Thank you for the update.

Does the patch have a reference number or any other ID?

Eluich
Novice
Posts: 6
Liked: 4 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 1 person likes this post

For the moment I don't have any other information
I just know that the patch concerns the VNext.

Post Reply

Who is online

Users browsing this forum: Majestic-12 [Bot] and 8 guests