-
- VP, Product Management
- Posts: 7081
- Liked: 1511 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
I have collected together some tips and tricks that are helpful outside of the RCT related performance things:
- IO freezes or high latency (>500ms) try to disable TRIM and ODX
TRIM: Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "DisableDeleteNotification" -Value 1
ODX: Set-ItemProperty hklm:systemcurrentcontrolsetcontrolfilesystem -Name "FilterSupportedFeaturesMode" -Value 1
Specifically if you have issues during SnapShot times
Note: Veeam Support can give you a "Snapshot Helper" tool that allows you to create VM Backup Snapshots without our Software involved. It uses native Windows commands. That way you can analyse if the issue is related to our software or a general issue.
- Check Antivirus exclusion list of Hyper-V and Veeam
http://social.technet.microsoft.com/wik ... hosts.aspx
https://www.veeam.com/kb1999
- Check for SnapShot redirections => very bad idea => Setting of the VM - Management - Checkpoint File Location
- Don´t touch the pagefile settings. Microsoft defaults are working fine.
- Hardware VSS provider: Check firmware, driver and provider compatibility
- Don´t use checkpoints (Backup may fail) => use Veeam Quick Backups => Remove checkpoints
- Check for Firmware Update that support Hyper-V and use correct driver.
- Windows Update is NOT enough. Check Microsoft Lists for Hyper-V and Cluster
- Don´t mix AMD/Intel in the same cluster. Keep „instruction set“ the same.
- Keep OS and VM data on seperate disks
- Keep the Hosts equaly (NIC & Storage Controllers !)
- No other roles on the hyper-v host
- No Inguest iSCSI/Path trough disks
- Only ReFS and NTFS are supported.
- If you run Win2003 or SQL2000 disable backup guest interaction + KB940349
- Disk Manager – „System reserved“ need at min 42MB free
- VM – Integration Services – Backup enabled ?
- Hyper-V Volume Shadow Copy Requestor Service started ?
- Use Jumbo frames (on Hyper-V NIC, Switches and Storage) and use logo tested NICs
- 64kb disk allocation size for VHD(x)
- Consider running Hyper-V with „high performance“ not default „balanced“
- Min 300MB free disk space per disk (local/CSV/any)
- Latest Integration components
- IO freezes or high latency (>500ms) try to disable TRIM and ODX
TRIM: Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "DisableDeleteNotification" -Value 1
ODX: Set-ItemProperty hklm:systemcurrentcontrolsetcontrolfilesystem -Name "FilterSupportedFeaturesMode" -Value 1
Specifically if you have issues during SnapShot times
Note: Veeam Support can give you a "Snapshot Helper" tool that allows you to create VM Backup Snapshots without our Software involved. It uses native Windows commands. That way you can analyse if the issue is related to our software or a general issue.
- Check Antivirus exclusion list of Hyper-V and Veeam
http://social.technet.microsoft.com/wik ... hosts.aspx
https://www.veeam.com/kb1999
- Check for SnapShot redirections => very bad idea => Setting of the VM - Management - Checkpoint File Location
- Don´t touch the pagefile settings. Microsoft defaults are working fine.
- Hardware VSS provider: Check firmware, driver and provider compatibility
- Don´t use checkpoints (Backup may fail) => use Veeam Quick Backups => Remove checkpoints
- Check for Firmware Update that support Hyper-V and use correct driver.
- Windows Update is NOT enough. Check Microsoft Lists for Hyper-V and Cluster
- Don´t mix AMD/Intel in the same cluster. Keep „instruction set“ the same.
- Keep OS and VM data on seperate disks
- Keep the Hosts equaly (NIC & Storage Controllers !)
- No other roles on the hyper-v host
- No Inguest iSCSI/Path trough disks
- Only ReFS and NTFS are supported.
- If you run Win2003 or SQL2000 disable backup guest interaction + KB940349
- Disk Manager – „System reserved“ need at min 42MB free
- VM – Integration Services – Backup enabled ?
- Hyper-V Volume Shadow Copy Requestor Service started ?
- Use Jumbo frames (on Hyper-V NIC, Switches and Storage) and use logo tested NICs
- 64kb disk allocation size for VHD(x)
- Consider running Hyper-V with „high performance“ not default „balanced“
- Min 300MB free disk space per disk (local/CSV/any)
- Latest Integration components
-
- Service Provider
- Posts: 277
- Liked: 61 times
- Joined: Nov 17, 2014 1:48 pm
- Full Name: Florin
- Location: Switzerland
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
We have the same problems in our Hyper-V 2016 environement. Two datacenter sites with 5 hosts each.
We especially observe the issues on terminal server sessions (citrix) where profile data is located on a windows fileserver on the hyper-v cluster. We have reports from useres who weren't able to logon or have freezes while opening files etc.
As proposed by veeam support, we've switched to veeam agent backup for one of the affected fileservers and didn't had any problems since then. IO and latency measured with diskIO is far better in the VM (x10). This is only due to the fact, that agent backup won't trigger any RCT operation.
We've also opened a case with Microsoft premier support and all we have been told, is that the problem is acknowledged as an issue and that they are working on a fix for hyper-V 2019. There will be no chance for getting a fix for 2016.
We especially observe the issues on terminal server sessions (citrix) where profile data is located on a windows fileserver on the hyper-v cluster. We have reports from useres who weren't able to logon or have freezes while opening files etc.
As proposed by veeam support, we've switched to veeam agent backup for one of the affected fileservers and didn't had any problems since then. IO and latency measured with diskIO is far better in the VM (x10). This is only due to the fact, that agent backup won't trigger any RCT operation.
We've also opened a case with Microsoft premier support and all we have been told, is that the problem is acknowledged as an issue and that they are working on a fix for hyper-V 2019. There will be no chance for getting a fix for 2016.
-
- Influencer
- Posts: 19
- Liked: 18 times
- Joined: Jul 06, 2020 2:31 pm
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi
This morning, Microsoft sent me a test patch
If you want to test on your platform, please send me a private message so that I can provide you the information.
Best Regards
This morning, Microsoft sent me a test patch
If you want to test on your platform, please send me a private message so that I can provide you the information.
Best Regards
-
- Influencer
- Posts: 19
- Liked: 18 times
- Joined: Jul 06, 2020 2:31 pm
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi
In my case the patch doesn't seem to work
Best Regards
In my case the patch doesn't seem to work
Best Regards
-
- Veteran
- Posts: 3077
- Liked: 455 times
- Joined: Aug 07, 2018 3:11 pm
- Full Name: Fedor Maslov
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi Eluich,
Thanks for keeping us aware of the current state. Please, keep it coming.
Thanks for keeping us aware of the current state. Please, keep it coming.
-
- Influencer
- Posts: 19
- Liked: 18 times
- Joined: Jul 06, 2020 2:31 pm
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi
On my first test I had installed the patch on only one node of the cluster.
I have installed on all the nodes, I will keep you informed.
Best Regards
On my first test I had installed the patch on only one node of the cluster.
I have installed on all the nodes, I will keep you informed.
Best Regards
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Feb 03, 2021 1:10 pm
- Full Name: Stuart Szeto
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi Peter. I initially logged a VEEAM support case "04609525" (27-01-2021) for exact same symptoms. In our case, there were half a dozen VMs with corrupt VMCX files. I wasn't sure if these were due to Event ID 9 in the Hyper-V-StorageVSP's Admin log.pterpumpkin wrote: ↑Sep 02, 2020 8:20 pm Thank you for all the PM's with case numbers! Please keep them coming. I have a total of 6 now (including my own).
"An I/O request for device 'C:\ClusterStorage\Vol06-PoolB-R5-STD\VMs\SVRARCHIVE2\Virtual Hard Disks\SVRARCHIVE2-Disk2.vhdx' took 57994 milliseconds to complete. Operation code = WRITE16, Data transfer length = 8388608, Status = SRB_STATUS_SUCCESS."
It would appear the random spates of event ID 9 would also cause the cluster shared volume(s) to enter a paused state - not always, but sometimes. This would then sometimes cause a Hyper-V VM to enter a 'pause-critical' state, and eventually power off the VM.
Cluster Event: "Cluster Shared Volume 'Vol06-PoolB-R5-STD' ('Vol06-PoolB-R5-STD') has entered a paused state because of 'STATUS_IO_TIMEOUT(c00000b5)'. All I/O will temporarily be queued until a path to the volume is reestablished."
I had the VEEAM case closed, since fixing those corrupt VMCX files resulted in a reduction of CSV pause issues. In hindsight this was perhaps just coincidental, because we continue to see Event ID 9 - although it now doesn't seem to be bad enough to cause the CSVs to enter a pause state as often as before.
I had a Microsoft case opened in conjunction - "121012523000015", if you are still collecting MS case numbers....
-
- Enthusiast
- Posts: 47
- Liked: 10 times
- Joined: Aug 26, 2019 7:04 am
- Full Name: Christine Boersen
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Stuart_s:
From what I have observed on my clusters,
If you are seeing random CSV I/O paused, EventID 9 is a symptom, not the problem.
- ensure your I/O path isn't dropping, solving depends on the storage type
- For S2D, ensure you are on lossless network and have RDMA configured correctly.
- For DAS, ensure the driver isn't timing out and/or disk corruption/failing and/or you don't have a failing SCSI/FC cable or EMM on the DAS itself.
- When doing failover, EventID 9 may happen, especially if you didn't put storage in "StorageMaintenanceMode" before stopping the node with S2D, but sometimes even when you put it in storagemaintmode, you'll get a couple of them as it decides to put the I/O to a different device/path.
Hope that helps, and YMMV
Christine
From what I have observed on my clusters,
If you are seeing random CSV I/O paused, EventID 9 is a symptom, not the problem.
- ensure your I/O path isn't dropping, solving depends on the storage type
- For S2D, ensure you are on lossless network and have RDMA configured correctly.
- For DAS, ensure the driver isn't timing out and/or disk corruption/failing and/or you don't have a failing SCSI/FC cable or EMM on the DAS itself.
- When doing failover, EventID 9 may happen, especially if you didn't put storage in "StorageMaintenanceMode" before stopping the node with S2D, but sometimes even when you put it in storagemaintmode, you'll get a couple of them as it decides to put the I/O to a different device/path.
Hope that helps, and YMMV
Christine
-
- Service Provider
- Posts: 277
- Liked: 61 times
- Joined: Nov 17, 2014 1:48 pm
- Full Name: Florin
- Location: Switzerland
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
We' also received a private patch, but as the fix is for Server 2019, we're unable to install it on our 2016 farm.
Its name is KB900235. Is that the same that you received, @Eluich?
Its name is KB900235. Is that the same that you received, @Eluich?
-
- Influencer
- Posts: 19
- Liked: 18 times
- Joined: Jul 06, 2020 2:31 pm
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
@Dasfliege
Yep, the private fix name's is KB900235, it is only applicable to Windows Server 2019 Hyper-V environments
Best Regards
Yep, the private fix name's is KB900235, it is only applicable to Windows Server 2019 Hyper-V environments
Best Regards
-
- Enthusiast
- Posts: 76
- Liked: 16 times
- Joined: Oct 27, 2017 5:42 pm
- Full Name: Nick
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
This thread has taken a few meandering trails with apparently similar symptoms but evidently different root causes and has gotten somewhat hard to follow...
Does the KB900235 Fix purportedly address the specific problem I reported in my original post (the 1st post in this thread) where a Single Stand-Alone Non-Clustered Server with the VM’s on Local Direct Attached Storage is getting 10,000-20,000ms I/O Delays that appears to be a Server 2019 RCT related bug?
Thanks,
Nick
Does the KB900235 Fix purportedly address the specific problem I reported in my original post (the 1st post in this thread) where a Single Stand-Alone Non-Clustered Server with the VM’s on Local Direct Attached Storage is getting 10,000-20,000ms I/O Delays that appears to be a Server 2019 RCT related bug?
Thanks,
Nick
-
- Service Provider
- Posts: 277
- Liked: 61 times
- Joined: Nov 17, 2014 1:48 pm
- Full Name: Florin
- Location: Switzerland
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
@Nick-SAC
Yes.
It actually doesn't matter if it is a cluster or not or how your storage configuration looks like. It's also the same problem on Server 2016 and 2019. As soon as RCT is in use, you will encounter serious drops in throughput and latency.
Yes.
It actually doesn't matter if it is a cluster or not or how your storage configuration looks like. It's also the same problem on Server 2016 and 2019. As soon as RCT is in use, you will encounter serious drops in throughput and latency.
-
- Service Provider
- Posts: 277
- Liked: 61 times
- Joined: Nov 17, 2014 1:48 pm
- Full Name: Florin
- Location: Switzerland
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
@Eluich
Have you been able to deploy the patch to all cluster nodes? How is it behaving?
Have you been able to deploy the patch to all cluster nodes? How is it behaving?
-
- Influencer
- Posts: 19
- Liked: 18 times
- Joined: Jul 06, 2020 2:31 pm
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
@dasfliege
I didn't have any problems installing the patch, but in my case the patch doesn't solve my problem.
Best Regards
I didn't have any problems installing the patch, but in my case the patch doesn't solve my problem.
Best Regards
-
- Lurker
- Posts: 1
- Liked: 1 time
- Joined: Feb 16, 2021 2:18 pm
- Full Name: Jaco Joubert
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
We have been experiencing the same issue in our environment (Dell R640, Hyper-V Windows Server 2019, direct fibre connected Pure Storage). We formatted the volume using ReFs and started experiencing random huge delays in response time etc. The host worked perfectly fine but all VM's could not be logged in etc. We now formatted the volume to NTFS and all seems to be working at the moment. Shoul it fail again we would like to try the MS patch you mentioned (KB900235) @Eluich. Where can we download it?
-
- Influencer
- Posts: 19
- Liked: 18 times
- Joined: Jul 06, 2020 2:31 pm
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
@JacoJ
If you want test the private fix send me a PM
Best Regards
If you want test the private fix send me a PM
Best Regards
-
- Enthusiast
- Posts: 47
- Liked: 10 times
- Joined: Aug 26, 2019 7:04 am
- Full Name: Christine Boersen
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
+1 on wanting a copy
-
- Novice
- Posts: 6
- Liked: never
- Joined: Sep 29, 2020 11:59 am
- Full Name: Casper Glasius-Nyborg
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi all
@Eluich and @dasfliege can you send me the patch also, if you have any additional info or readme file i would like to have that also.
As we are heavy effected with this issue on our Hyper V 2019 setup.
Kind Regards
Casper
@Eluich and @dasfliege can you send me the patch also, if you have any additional info or readme file i would like to have that also.
As we are heavy effected with this issue on our Hyper V 2019 setup.
Kind Regards
Casper
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Jan 28, 2021 11:43 am
- Full Name: Thomas Fraedrich
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi all
@Eluich and @dasfliege can you send me the patch also.
Kind regards
Thomas
@Eluich and @dasfliege can you send me the patch also.
Kind regards
Thomas
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Feb 03, 2021 1:10 pm
- Full Name: Stuart Szeto
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
I asked Microsoft Support about the private KB mentioned in the thread - i.e. KB900235
The support person said he was not able to see it listed on the official site. Does anyone have further information about this patch?
The support person said he was not able to see it listed on the official site. Does anyone have further information about this patch?
-
- Influencer
- Posts: 12
- Liked: 3 times
- Joined: Aug 27, 2019 8:55 am
- Full Name: LeslieUC
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi @Nick-SACNick-SAC wrote: ↑Feb 11, 2021 8:11 am This thread has taken a few meandering trails with apparently similar symptoms but evidently different root causes and has gotten somewhat hard to follow...
Does the KB900235 Fix purportedly address the specific problem I reported in my original post (the 1st post in this thread) where a Single Stand-Alone Non-Clustered Server with the VM’s on Local Direct Attached Storage is getting 10,000-20,000ms I/O Delays that appears to be a Server 2019 RCT related bug?
Thanks,
Nick
Did you patch your environment?
Did it solve the original problem from the first post?
Can someone share more information about this private patch. Release notes, patch information etc.
Can someone from Veeam look in to this patch and confirm if this patch solves the original problem?
Thanks.
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
So far it seems from feedback here that the patch did not work. Please let me know if anyone finds otherwise.
I have not yet reviewed it, but it was on the short list for me to look at this week. I'll let you know as soon as I have an update.
I have not yet reviewed it, but it was on the short list for me to look at this week. I'll let you know as soon as I have an update.
-
- Service Provider
- Posts: 372
- Liked: 120 times
- Joined: Nov 25, 2016 1:56 pm
- Full Name: Mihkel Soomere
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Private hotfixes often have no documentation (only internal to product team) and fictitious KB numbers (KB99999, KB98765...). I can't speak for this one but they have occasionally required test-signing (not installable on default Windows systems).
-
- Service Provider
- Posts: 277
- Liked: 61 times
- Joined: Nov 17, 2014 1:48 pm
- Full Name: Florin
- Location: Switzerland
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Any news here? Is someone who installed the patch working actively with MS?
We are in touch with MS, but can't test it as we're on Server 2016.
We are in touch with MS, but can't test it as we're on Server 2016.
-
- Influencer
- Posts: 12
- Liked: 3 times
- Joined: Aug 27, 2019 8:55 am
- Full Name: LeslieUC
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi, @johan.h
Did you had time to review the patch?
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Not quite yet, but I will let you know as soon as I'm finished testing.
-
- Veeam Software
- Posts: 723
- Liked: 185 times
- Joined: Jun 05, 2013 9:45 am
- Full Name: Johan Huttenga
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
I have confirmation the above mentioned fix won't address this issue. I do have it on good authority the RCT issue is being worked on though.
-
- Influencer
- Posts: 12
- Liked: 3 times
- Joined: Aug 27, 2019 8:55 am
- Full Name: LeslieUC
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Hi, is there any more info about this?
-
- Service Provider
- Posts: 277
- Liked: 61 times
- Joined: Nov 17, 2014 1:48 pm
- Full Name: Florin
- Location: Switzerland
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Not yet. Still waiting for the next gen fix.
-
- Service Provider
- Posts: 16
- Liked: 3 times
- Joined: Feb 19, 2015 5:10 pm
- Full Name: Darren Durbin
- Location: Hampshire, UK
- Contact:
Re: Windows Server 2019 Hyper-V VM I/O Performance Problem
Just tagging this so I can keep track. We're seeing what appears to be the exact same issue across a small 2019 cluster with 2019 guests on HPE servers and MSA2052 storage.
Who is online
Users browsing this forum: No registered users and 6 guests