Windows Server 2019 Hyper-V VM I/O Performance Problem

[ChristineAlexa]

WeasleyUC
There are two issues going on here, for some hardware, See my testing results above.
And agreed, Neither of the issues are due to Veeam's code. Also agreed, the RCT one affects all vendors' products (that I'm aware of) using that technology.

Christine.

[bhead]

Hello everybody,

we're currently dealing with the exact same issue at one of our datacenter-sites:

Hyper-V 2019 Failover-Cluster with a total of 9 nodes.
The nodes are connected to the storage (All-Flash) via ISCSI on a 10 GBit/s network.
IO Performance randomly drops to nearly a complete stall.
Doing some test on any node, writing and reading data from the CSV, everything looks fine. Our network and our storage are as fast it gets.
Doing the same tests inside a random VM, writing data to and from a virtual hard disk (dynamic vhdx / static vhdx with a fixed size) will result in about 10% of the IO that we see on the parent partition.

We're currently dealing with the Microsoft Support. Right now they don't seem to have an idea on what's causing the latency and the low IO performance.

@pterpumkin: We would like to submit your MS-Ticket ID to our support contact. Maybe they can make the connection here to make some progress on both our cases? Maybe you could contact me via PM?

[ChristineAlexa]

bhead:
Please see my reply earlier in this post, post383997.html#p383997

But this sounds more like the performance issue inside a HyperV VM that I documented with solutions for some hardware there with Hyperthreading vs the CBT/RCT bug that we are discussing as well. Note solving the performance in a VM does NOT solve the CBT/RCT bug, but on some hardware (all that I have personally tested), makes it happen far less frequently.

Just for a data points
- What vendor and model are the servers?
- What disk controllers and bus type are you using?
- What brand and model are the flash devices.

Thanks,
Christine

[CasperGN]

I got ticket numbers from Pterpumkin.

0. 120060926001170 - Pter's own ticket.
1. 120063025001053
2. 120081024004251
3. 120040625000059
4. 119050122000550 - Veeam ticket number
5. 119100924004286
6. 120052425000074 – other backup vendor then Veeam, but same issue with performance.
7. 120081221000284
8. 120040625000059 - My own ticket which im pushing heavy on.

/Casper

[CasperGN]

Hi All

Im still pushing onward with MS, only got them to say that they have escalated this internally and are awaiting updates.
Has any of you gotten any positiv outcome from your MS cases ?

/Casper

[mkaec]

Casper,

Are you dealing with actual MS support or outsourced support? I've had a few incidents with outsourced support and they tried really hard to bury the case each time. With my last case, MS (appears to have) accidentally released the fix I was asking for. I asked the support rep to confirm the internal case he was referring to confirmed the fix and he ghosted me.

[scootrz32]

Have you found any correlation. Like we see the issue and have Dell 14g servers with H740p controllers

[ChristineAlexa]

scootrz32:
Have you looked at my research earlier in the post? Though all of my testing is with 11g, 12g, and 13g Dell servers (and the appropriate PERC controllers for the generations)

[scootrz32]

I think I resolved ours today. I am going to test for a couple days. It was SentinelOne AV causing our issue. Once I removed it, it appears to be working.

Anyone else running S1?

[CasperGN]

Hi

@mkaec Im dealing with MS directly.
@Scootrz32 We have SentinelOne AV, but we havent installed them on our HyperV 2019 hosts or this VM. Yesterday we moved one of the most affected vm's to another environment we have which is running Hyper V 2016, at the moment that setup is not used so much, so the vm have plenty of power and at the moment everything looks very good.
I noticed another thing yesterday during our storage migration for the move to Hyper V 2016, the VM's Disk Queue and latency dropped to almost nothing just when the storage migration finished. The VM where still on the same host and same storage, just a new storage volume. Im currently talking with our Storage vendor Infinidat on this issue. Because it looked very strange.

/Casper

[scootrz32]

Once I removed it from the virtual machine. Mine seems to take snapshots and work OK.

[Eluich]

Hi

Do you have news on your Microsoft cases ?
On my side, no significant progress

Best Regards

[pterpumpkin]

Nothing yet

[CasperGN]

My MS ticket was incorrectly closed with info that they are working on a fix, but the issue is different from what we are experiencing. But it might help others.

MS Comment
-------------------------
The instructions that my colleague Jose Ramon sent to you were to rollback and uninstall the updates since August 2020 on the Hypervisor (Not the virtual machine):

In this link you can see a discussion about this same issue that affects to backup and replication tasks.
https://docs.microsoft.com/en-us/answer ... er-au.html

The reason for that was that the team is working on a fix that will be released in the future. Currently we cannot provide a time when this will be available so you will have to wait for it. For that, and to try to mitigate the issue, my colleague suggested to delete the cumulative updates after KB4559003 (https://support.microsoft.com/en-in/help/4559003) (Do not uninstall this KB).
--------------------------

But we dont have that patch and doesnt see performance degradation doing image backup, but afterwards. But MS didnt seem to understand that, the guy who worked on our case, kept on troubleshooting as it is a VM issue.

Hope it help some of you.

[yoda-ict]

Any news about this issue? Disabling Hyper-V helps but after a while we get a lot of StorageVSP Events nr 9 with description:

An I/O request for device 'C:\ClusterStorage\Volume7\ServerName\Virtual Hard Disks\Name.vhdx' took 18150 milliseconds to complete. Operation code = WRITE, Data transfer length = 524288, Status = SRB_STATUS_SUCCESS.

[pterpumpkin]

MS closed my case and left us without a fix. They did this as they said it has been identified as a bug and there are no further troubleshooting steps, so there is no need to have a case open *sigh*.

I am emailing them weekly to ask if there is any update on the bug fix. This seems to be going nowhere and i have a feeling they don't actually check, they just say "no" and that they will let me know when a fix becomes available.

I have just tried deleting all of the RCT points for our affected VM and disabled our automatic daily live migration pre-emptive fix we had in place. The issue normally takes a few days to a few weeks to occur, so i will let you know how it goes.

[Eluich]

On my side, the case is still open, normally a patch is in production but the person responsible for my case told me it could take several weeks

[Eluich]

Small update, I just got Microsoft. The patch has been created. They are finishing testing it on their lab before releasing it.

[wishr]

Hi Eluich,

Thank you for the update.

Does the patch have a reference number or any other ID?

[Eluich]

For the moment I don't have any other information
I just know that the patch concerns the VNext.

[rold]

Last update from MS on RCT performance issue ticket:

Code: Select all

1. This is not a code defect that we can fix straightaway. It is a design level change that needs approval from two CVPs before the work can officially start.
2. The 1C check-in deadline is over and given that 2C check-in deadline is on 1/8, I don’t think this will be ready by then. So my best guess is 3C (or later) for this and not earlier.

So .. maybe it will be fixed in vNext, maybe in vNextNext, maybe in some "Azure OS" only , maybe never.

[wishr]

This is indeed sad, but at least Microsoft is aware of the demand. Thank you for the update, Rold.

[Gostev]

This just shows how much exactly Microsoft cares about on-prem Hyper-V I mean, it's not like they did not have an understanding of the issue over 1 year ago (see the first post of this topic). So normally, they should have secured all required approvals and planned the fix back then. But instead, it sounds like this process is starting only now? I don't get it.

[rold]

Maybe you can port Veeam CBT driver to Server 2016 and later?

[plinfra]

Same issue here:

Windows Server 2019 Datacenter (64 bit) with Hyper-V role assigned
ProLiant DL380 Gen9
5.5TB RAID6 (Created in Smart Array)

It was hosting 6VMs and currently is hosting 4 and still showing issues:
An I/O request for device 'device.vhdx' took 37186 milliseconds to complete. Operation code = WRITE16, Data transfer length = 4096, Status = SRB_STATUS_SUCCESS.
3x 2012r2
1x 2019

[Andreas Neufert]

Please allow me to add here that not all performance issues come from the RCT. Specifically if you have latency in the seconds area.

I had some customers that had real trouble with their ODX integration on the storage side. Create a test VM on the local disks of an Hyper-V host and compare the backup with the same VM on the storage (CSV).

[ChristineAlexa]

Same issue here:

Windows Server 2019 Datacenter (64 bit) with Hyper-V role assigned
ProLiant DL380 Gen9
5.5TB RAID6 (Created in Smart Array)

It was hosting 6VMs and currently is hosting 4 and still showing issues:
An I/O request for device 'device.vhdx' took 37186 milliseconds to complete. Operation code = WRITE16, Data transfer length = 4096, Status = SRB_STATUS_SUCCESS.
3x 2012r2
1x 2019

Did you try any of the suggestions I listed above in this post? They help dramatically on some hardware for the Non-CBT performance issues.

[DonZoomik]

I have noticed this issue occasionally on some VMs that run on Veeam repository's ReFS file system for reasons, but only overnight when there are a lot of backup, copy and replication jobs running at the same time. Queue depth and IO latency stay very reasonable in monitoring but Hyper-V has up to 2 minutes of IO latency according to Event Log and VMs become nearly frozen until issue clears by itself some time later.
Sidechannel attack mitigations are already disabled and classic scheduler is enabled in BCD, all VMs have HwThreadCountPerCore at 0. These VM's shouldn't have RCT enabled as these VMs have never been backed up through hypervisor (in-VM agent only).

SuperMicro SSG-6029P-E1CR24H
Broadcom MegaRAID 9361 (SuperMicro OEM) with 24*16TB SAS RAID60 (2 spans of 12) + 2*1.6TB SAS RAID1 CacheCade (SSD Caching).

[mortendb]

Does anyone have any updates on this issue, i am seeing the same with my setup.

[WesleyUC]

Please allow me to add here that not all performance issues come from the RCT. Specifically if you have latency in the seconds area.

I had some customers that had real trouble with their ODX integration on the storage side. Create a test VM on the local disks of an Hyper-V host and compare the backup with the same VM on the storage (CSV).

Correct, a lot of reactions on this forum item does not apply with the RCT and/or CBT problems that the original poster is referring to.