Windows Server 2019 Hyper-V VM I/O Performance Problem

[Andreas Neufert]

I have collected together some tips and tricks that are helpful outside of the RCT related performance things:

- IO freezes or high latency (>500ms) try to disable TRIM and ODX
TRIM: Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "DisableDeleteNotification" -Value 1
ODX: Set-ItemProperty hklm:systemcurrentcontrolsetcontrolfilesystem -Name "FilterSupportedFeaturesMode" -Value 1
Specifically if you have issues during SnapShot times
Note: Veeam Support can give you a "Snapshot Helper" tool that allows you to create VM Backup Snapshots without our Software involved. It uses native Windows commands. That way you can analyse if the issue is related to our software or a general issue.
- Check Antivirus exclusion list of Hyper-V and Veeam
http://social.technet.microsoft.com/wik ... hosts.aspx
https://www.veeam.com/kb1999
- Check for SnapShot redirections => very bad idea => Setting of the VM - Management - Checkpoint File Location
- Don´t touch the pagefile settings. Microsoft defaults are working fine.
- Hardware VSS provider: Check firmware, driver and provider compatibility
- Don´t use checkpoints (Backup may fail) => use Veeam Quick Backups => Remove checkpoints
- Check for Firmware Update that support Hyper-V and use correct driver.
- Windows Update is NOT enough. Check Microsoft Lists for Hyper-V and Cluster
- Don´t mix AMD/Intel in the same cluster. Keep „instruction set“ the same.
- Keep OS and VM data on seperate disks
- Keep the Hosts equaly (NIC & Storage Controllers !)
- No other roles on the hyper-v host
- No Inguest iSCSI/Path trough disks
- Only ReFS and NTFS are supported.
- If you run Win2003 or SQL2000 disable backup guest interaction + KB940349
- Disk Manager – „System reserved“ need at min 42MB free
- VM – Integration Services – Backup enabled ?
- Hyper-V Volume Shadow Copy Requestor Service started ?
- Use Jumbo frames (on Hyper-V NIC, Switches and Storage) and use logo tested NICs
- 64kb disk allocation size for VHD(x)
- Consider running Hyper-V with „high performance“ not default „balanced“
- Min 300MB free disk space per disk (local/CSV/any)
- Latest Integration components

[dasfliege]

We have the same problems in our Hyper-V 2016 environement. Two datacenter sites with 5 hosts each.
We especially observe the issues on terminal server sessions (citrix) where profile data is located on a windows fileserver on the hyper-v cluster. We have reports from useres who weren't able to logon or have freezes while opening files etc.

As proposed by veeam support, we've switched to veeam agent backup for one of the affected fileservers and didn't had any problems since then. IO and latency measured with diskIO is far better in the VM (x10). This is only due to the fact, that agent backup won't trigger any RCT operation.

We've also opened a case with Microsoft premier support and all we have been told, is that the problem is acknowledged as an issue and that they are working on a fix for hyper-V 2019. There will be no chance for getting a fix for 2016.

[Eluich]

Hi

This morning, Microsoft sent me a test patch
If you want to test on your platform, please send me a private message so that I can provide you the information.

Best Regards

[Eluich]

Hi

In my case the patch doesn't seem to work

Best Regards

[wishr]

Hi Eluich,

Thanks for keeping us aware of the current state. Please, keep it coming.

[Eluich]

Hi

On my first test I had installed the patch on only one node of the cluster.
I have installed on all the nodes, I will keep you informed.

Best Regards

[stuart.s]

Thank you for all the PM's with case numbers! Please keep them coming. I have a total of 6 now (including my own).

Hi Peter. I initially logged a VEEAM support case "04609525" (27-01-2021) for exact same symptoms. In our case, there were half a dozen VMs with corrupt VMCX files. I wasn't sure if these were due to Event ID 9 in the Hyper-V-StorageVSP's Admin log.

"An I/O request for device 'C:\ClusterStorage\Vol06-PoolB-R5-STD\VMs\SVRARCHIVE2\Virtual Hard Disks\SVRARCHIVE2-Disk2.vhdx' took 57994 milliseconds to complete. Operation code = WRITE16, Data transfer length = 8388608, Status = SRB_STATUS_SUCCESS."

It would appear the random spates of event ID 9 would also cause the cluster shared volume(s) to enter a paused state - not always, but sometimes. This would then sometimes cause a Hyper-V VM to enter a 'pause-critical' state, and eventually power off the VM.

Cluster Event: "Cluster Shared Volume 'Vol06-PoolB-R5-STD' ('Vol06-PoolB-R5-STD') has entered a paused state because of 'STATUS_IO_TIMEOUT(c00000b5)'. All I/O will temporarily be queued until a path to the volume is reestablished."

I had the VEEAM case closed, since fixing those corrupt VMCX files resulted in a reduction of CSV pause issues. In hindsight this was perhaps just coincidental, because we continue to see Event ID 9 - although it now doesn't seem to be bad enough to cause the CSVs to enter a pause state as often as before.
I had a Microsoft case opened in conjunction - "121012523000015", if you are still collecting MS case numbers....

[ChristineAlexa]

Stuart_s:
From what I have observed on my clusters,

If you are seeing random CSV I/O paused, EventID 9 is a symptom, not the problem.
- ensure your I/O path isn't dropping, solving depends on the storage type
- For S2D, ensure you are on lossless network and have RDMA configured correctly.
- For DAS, ensure the driver isn't timing out and/or disk corruption/failing and/or you don't have a failing SCSI/FC cable or EMM on the DAS itself.
- When doing failover, EventID 9 may happen, especially if you didn't put storage in "StorageMaintenanceMode" before stopping the node with S2D, but sometimes even when you put it in storagemaintmode, you'll get a couple of them as it decides to put the I/O to a different device/path.

Hope that helps, and YMMV
Christine

[dasfliege]

We' also received a private patch, but as the fix is for Server 2019, we're unable to install it on our 2016 farm.
Its name is KB900235. Is that the same that you received, @Eluich?

[Eluich]

@Dasfliege

Yep, the private fix name's is KB900235, it is only applicable to Windows Server 2019 Hyper-V environments

Best Regards

[Nick-SAC]

This thread has taken a few meandering trails with apparently similar symptoms but evidently different root causes and has gotten somewhat hard to follow...

Does the KB900235 Fix purportedly address the specific problem I reported in my original post (the 1st post in this thread) where a Single Stand-Alone Non-Clustered Server with the VM’s on Local Direct Attached Storage is getting 10,000-20,000ms I/O Delays that appears to be a Server 2019 RCT related bug?

Thanks,
Nick

[dasfliege]

@Nick-SAC

Yes.
It actually doesn't matter if it is a cluster or not or how your storage configuration looks like. It's also the same problem on Server 2016 and 2019. As soon as RCT is in use, you will encounter serious drops in throughput and latency.

[dasfliege]

@Eluich

Have you been able to deploy the patch to all cluster nodes? How is it behaving?

[Eluich]

@dasfliege

I didn't have any problems installing the patch, but in my case the patch doesn't solve my problem.

Best Regards

[JacoJ]

We have been experiencing the same issue in our environment (Dell R640, Hyper-V Windows Server 2019, direct fibre connected Pure Storage). We formatted the volume using ReFs and started experiencing random huge delays in response time etc. The host worked perfectly fine but all VM's could not be logged in etc. We now formatted the volume to NTFS and all seems to be working at the moment. Shoul it fail again we would like to try the MS patch you mentioned (KB900235) @Eluich. Where can we download it?

[Eluich]

@JacoJ

If you want test the private fix send me a PM

Best Regards

[ChristineAlexa]

+1 on wanting a copy

[CasperGN]

Hi all

@Eluich and @dasfliege can you send me the patch also, if you have any additional info or readme file i would like to have that also.
As we are heavy effected with this issue on our Hyper V 2019 setup.

Kind Regards
Casper

[ThomasFraedrich]

Hi all

@Eluich and @dasfliege can you send me the patch also.

Kind regards
Thomas

[stuart.s]

I asked Microsoft Support about the private KB mentioned in the thread - i.e. KB900235
The support person said he was not able to see it listed on the official site. Does anyone have further information about this patch?

[WesleyUC]

This thread has taken a few meandering trails with apparently similar symptoms but evidently different root causes and has gotten somewhat hard to follow...

Does the KB900235 Fix purportedly address the specific problem I reported in my original post (the 1st post in this thread) where a Single Stand-Alone Non-Clustered Server with the VM’s on Local Direct Attached Storage is getting 10,000-20,000ms I/O Delays that appears to be a Server 2019 RCT related bug?

Thanks,
Nick

Hi @Nick-SAC

Did you patch your environment?
Did it solve the original problem from the first post?

Can someone share more information about this private patch. Release notes, patch information etc.

Can someone from Veeam look in to this patch and confirm if this patch solves the original problem?

Thanks.

[johan.h]

So far it seems from feedback here that the patch did not work. Please let me know if anyone finds otherwise.

I have not yet reviewed it, but it was on the short list for me to look at this week. I'll let you know as soon as I have an update.

[DonZoomik]

I asked Microsoft Support about the private KB mentioned in the thread - i.e. KB900235
The support person said he was not able to see it listed on the official site. Does anyone have further information about this patch?

Private hotfixes often have no documentation (only internal to product team) and fictitious KB numbers (KB99999, KB98765...). I can't speak for this one but they have occasionally required test-signing (not installable on default Windows systems).

[dasfliege]

Any news here? Is someone who installed the patch working actively with MS?
We are in touch with MS, but can't test it as we're on Server 2016.

[WesleyUC]

So far it seems from feedback here that the patch did not work. Please let me know if anyone finds otherwise.

I have not yet reviewed it, but it was on the short list for me to look at this week. I'll let you know as soon as I have an update.

Hi, @johan.h

Did you had time to review the patch?

[johan.h]

Not quite yet, but I will let you know as soon as I'm finished testing.

[johan.h]

I have confirmation the above mentioned fix won't address this issue. I do have it on good authority the RCT issue is being worked on though.

[WesleyUC]

Hi, is there any more info about this?

[dasfliege]

Not yet. Still waiting for the next gen fix.

[DarrenD]

Just tagging this so I can keep track. We're seeing what appears to be the exact same issue across a small 2019 cluster with 2019 guests on HPE servers and MSA2052 storage.