Host-based backup of Microsoft Hyper-V VMs.
Post Reply
Andreas Neufert
VP, Product Management
Posts: 6702
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Andreas Neufert » 2 people like this post

I have collected together some tips and tricks that are helpful outside of the RCT related performance things:

- IO freezes or high latency (>500ms) try to disable TRIM and ODX
TRIM: Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem -Name "DisableDeleteNotification" -Value 1
ODX: Set-ItemProperty hklm:systemcurrentcontrolsetcontrolfilesystem -Name "FilterSupportedFeaturesMode" -Value 1
Specifically if you have issues during SnapShot times
Note: Veeam Support can give you a "Snapshot Helper" tool that allows you to create VM Backup Snapshots without our Software involved. It uses native Windows commands. That way you can analyse if the issue is related to our software or a general issue.
- Check Antivirus exclusion list of Hyper-V and Veeam
http://social.technet.microsoft.com/wik ... hosts.aspx
https://www.veeam.com/kb1999
- Check for SnapShot redirections => very bad idea => Setting of the VM - Management - Checkpoint File Location
- Don´t touch the pagefile settings. Microsoft defaults are working fine.
- Hardware VSS provider: Check firmware, driver and provider compatibility
- Don´t use checkpoints (Backup may fail) => use Veeam Quick Backups => Remove checkpoints
- Check for Firmware Update that support Hyper-V and use correct driver.
- Windows Update is NOT enough. Check Microsoft Lists for Hyper-V and Cluster
- Don´t mix AMD/Intel in the same cluster. Keep „instruction set“ the same.
- Keep OS and VM data on seperate disks
- Keep the Hosts equaly (NIC & Storage Controllers !)
- No other roles on the hyper-v host
- No Inguest iSCSI/Path trough disks
- Only ReFS and NTFS are supported.
- If you run Win2003 or SQL2000 disable backup guest interaction + KB940349
- Disk Manager – „System reserved“ need at min 42MB free
- VM – Integration Services – Backup enabled ?
- Hyper-V Volume Shadow Copy Requestor Service started ?
- Use Jumbo frames (on Hyper-V NIC, Switches and Storage) and use logo tested NICs
- 64kb disk allocation size for VHD(x)
- Consider running Hyper-V with „high performance“ not default „balanced“
- Min 300MB free disk space per disk (local/CSV/any)
- Latest Integration components
dasfliege
Service Provider
Posts: 233
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by dasfliege » 1 person likes this post

We have the same problems in our Hyper-V 2016 environement. Two datacenter sites with 5 hosts each.
We especially observe the issues on terminal server sessions (citrix) where profile data is located on a windows fileserver on the hyper-v cluster. We have reports from useres who weren't able to logon or have freezes while opening files etc.

As proposed by veeam support, we've switched to veeam agent backup for one of the affected fileservers and didn't had any problems since then. IO and latency measured with diskIO is far better in the VM (x10). This is only due to the fact, that agent backup won't trigger any RCT operation.

We've also opened a case with Microsoft premier support and all we have been told, is that the problem is acknowledged as an issue and that they are working on a fix for hyper-V 2019. There will be no chance for getting a fix for 2016.
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 6 people like this post

Hi

This morning, Microsoft sent me a test patch
If you want to test on your platform, please send me a private message so that I can provide you the information.

Best Regards
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich »

Hi

In my case the patch doesn't seem to work

Best Regards
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

Hi Eluich,

Thanks for keeping us aware of the current state. Please, keep it coming.
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 2 people like this post

Hi

On my first test I had installed the patch on only one node of the cluster.
I have installed on all the nodes, I will keep you informed.

Best Regards
stuart.s
Lurker
Posts: 2
Liked: never
Joined: Feb 03, 2021 1:10 pm
Full Name: Stuart Szeto
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by stuart.s »

pterpumpkin wrote: Sep 02, 2020 8:20 pm Thank you for all the PM's with case numbers! Please keep them coming. I have a total of 6 now (including my own).
Hi Peter. I initially logged a VEEAM support case "04609525" (27-01-2021) for exact same symptoms. In our case, there were half a dozen VMs with corrupt VMCX files. I wasn't sure if these were due to Event ID 9 in the Hyper-V-StorageVSP's Admin log.

"An I/O request for device 'C:\ClusterStorage\Vol06-PoolB-R5-STD\VMs\SVRARCHIVE2\Virtual Hard Disks\SVRARCHIVE2-Disk2.vhdx' took 57994 milliseconds to complete. Operation code = WRITE16, Data transfer length = 8388608, Status = SRB_STATUS_SUCCESS."

It would appear the random spates of event ID 9 would also cause the cluster shared volume(s) to enter a paused state - not always, but sometimes. This would then sometimes cause a Hyper-V VM to enter a 'pause-critical' state, and eventually power off the VM.

Cluster Event: "Cluster Shared Volume 'Vol06-PoolB-R5-STD' ('Vol06-PoolB-R5-STD') has entered a paused state because of 'STATUS_IO_TIMEOUT(c00000b5)'. All I/O will temporarily be queued until a path to the volume is reestablished."

I had the VEEAM case closed, since fixing those corrupt VMCX files resulted in a reduction of CSV pause issues. In hindsight this was perhaps just coincidental, because we continue to see Event ID 9 - although it now doesn't seem to be bad enough to cause the CSVs to enter a pause state as often as before.
I had a Microsoft case opened in conjunction - "121012523000015", if you are still collecting MS case numbers....
ChristineAlexa
Enthusiast
Posts: 44
Liked: 6 times
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

Stuart_s:
From what I have observed on my clusters,

If you are seeing random CSV I/O paused, EventID 9 is a symptom, not the problem.
- ensure your I/O path isn't dropping, solving depends on the storage type
- For S2D, ensure you are on lossless network and have RDMA configured correctly.
- For DAS, ensure the driver isn't timing out and/or disk corruption/failing and/or you don't have a failing SCSI/FC cable or EMM on the DAS itself.
- When doing failover, EventID 9 may happen, especially if you didn't put storage in "StorageMaintenanceMode" before stopping the node with S2D, but sometimes even when you put it in storagemaintmode, you'll get a couple of them as it decides to put the I/O to a different device/path.

Hope that helps, and YMMV :)
Christine
dasfliege
Service Provider
Posts: 233
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by dasfliege »

We' also received a private patch, but as the fix is for Server 2019, we're unable to install it on our 2016 farm.
Its name is KB900235. Is that the same that you received, @Eluich?
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 1 person likes this post

@Dasfliege

Yep, the private fix name's is KB900235, it is only applicable to Windows Server 2019 Hyper-V environments

Best Regards
Nick-SAC
Enthusiast
Posts: 73
Liked: 15 times
Joined: Oct 27, 2017 5:42 pm
Full Name: Nick
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Nick-SAC »

This thread has taken a few meandering trails with apparently similar symptoms but evidently different root causes and has gotten somewhat hard to follow...

Does the KB900235 Fix purportedly address the specific problem I reported in my original post (the 1st post in this thread) where a Single Stand-Alone Non-Clustered Server with the VM’s on Local Direct Attached Storage is getting 10,000-20,000ms I/O Delays that appears to be a Server 2019 RCT related bug?

Thanks,
Nick
dasfliege
Service Provider
Posts: 233
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by dasfliege » 1 person likes this post

@Nick-SAC

Yes.
It actually doesn't matter if it is a cluster or not or how your storage configuration looks like. It's also the same problem on Server 2016 and 2019. As soon as RCT is in use, you will encounter serious drops in throughput and latency.
dasfliege
Service Provider
Posts: 233
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by dasfliege »

@Eluich

Have you been able to deploy the patch to all cluster nodes? How is it behaving?
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich »

@dasfliege

I didn't have any problems installing the patch, but in my case the patch doesn't solve my problem.

Best Regards
JacoJ
Lurker
Posts: 1
Liked: 1 time
Joined: Feb 16, 2021 2:18 pm
Full Name: Jaco Joubert
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by JacoJ » 1 person likes this post

We have been experiencing the same issue in our environment (Dell R640, Hyper-V Windows Server 2019, direct fibre connected Pure Storage). We formatted the volume using ReFs and started experiencing random huge delays in response time etc. The host worked perfectly fine but all VM's could not be logged in etc. We now formatted the volume to NTFS and all seems to be working at the moment. Shoul it fail again we would like to try the MS patch you mentioned (KB900235) @Eluich. Where can we download it?
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 2 people like this post

@JacoJ

If you want test the private fix send me a PM

Best Regards
ChristineAlexa
Enthusiast
Posts: 44
Liked: 6 times
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

+1 on wanting a copy :)
CasperGN
Novice
Posts: 6
Liked: never
Joined: Sep 29, 2020 11:59 am
Full Name: Casper Glasius-Nyborg
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by CasperGN »

Hi all

@Eluich and @dasfliege can you send me the patch also, if you have any additional info or readme file i would like to have that also.
As we are heavy effected with this issue on our Hyper V 2019 setup.

Kind Regards
Casper
ThomasFraedrich
Lurker
Posts: 1
Liked: never
Joined: Jan 28, 2021 11:43 am
Full Name: Thomas Fraedrich
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ThomasFraedrich »

Hi all

@Eluich and @dasfliege can you send me the patch also.

Kind regards
Thomas
stuart.s
Lurker
Posts: 2
Liked: never
Joined: Feb 03, 2021 1:10 pm
Full Name: Stuart Szeto
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by stuart.s »

I asked Microsoft Support about the private KB mentioned in the thread - i.e. KB900235
The support person said he was not able to see it listed on the official site. Does anyone have further information about this patch?
WesleyUC
Influencer
Posts: 12
Liked: 3 times
Joined: Aug 27, 2019 8:55 am
Full Name: LeslieUC
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by WesleyUC »

Nick-SAC wrote: Feb 11, 2021 8:11 am This thread has taken a few meandering trails with apparently similar symptoms but evidently different root causes and has gotten somewhat hard to follow...

Does the KB900235 Fix purportedly address the specific problem I reported in my original post (the 1st post in this thread) where a Single Stand-Alone Non-Clustered Server with the VM’s on Local Direct Attached Storage is getting 10,000-20,000ms I/O Delays that appears to be a Server 2019 RCT related bug?

Thanks,
Nick
Hi @Nick-SAC

Did you patch your environment?
Did it solve the original problem from the first post?

Can someone share more information about this private patch. Release notes, patch information etc.

Can someone from Veeam look in to this patch and confirm if this patch solves the original problem?

Thanks.
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

So far it seems from feedback here that the patch did not work. Please let me know if anyone finds otherwise.

I have not yet reviewed it, but it was on the short list for me to look at this week. I'll let you know as soon as I have an update.
DonZoomik
Service Provider
Posts: 368
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by DonZoomik » 1 person likes this post

stuart.s wrote: Feb 22, 2021 1:13 am I asked Microsoft Support about the private KB mentioned in the thread - i.e. KB900235
The support person said he was not able to see it listed on the official site. Does anyone have further information about this patch?
Private hotfixes often have no documentation (only internal to product team) and fictitious KB numbers (KB99999, KB98765...). I can't speak for this one but they have occasionally required test-signing (not installable on default Windows systems).
dasfliege
Service Provider
Posts: 233
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by dasfliege »

Any news here? Is someone who installed the patch working actively with MS?
We are in touch with MS, but can't test it as we're on Server 2016.
WesleyUC
Influencer
Posts: 12
Liked: 3 times
Joined: Aug 27, 2019 8:55 am
Full Name: LeslieUC
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by WesleyUC »

johan.h wrote: Feb 22, 2021 6:21 am So far it seems from feedback here that the patch did not work. Please let me know if anyone finds otherwise.

I have not yet reviewed it, but it was on the short list for me to look at this week. I'll let you know as soon as I have an update.
Hi, @johan.h

Did you had time to review the patch?
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

Not quite yet, but I will let you know as soon as I'm finished testing.
johan.h
Veeam Software
Posts: 708
Liked: 177 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by johan.h »

I have confirmation the above mentioned fix won't address this issue. I do have it on good authority the RCT issue is being worked on though.
WesleyUC
Influencer
Posts: 12
Liked: 3 times
Joined: Aug 27, 2019 8:55 am
Full Name: LeslieUC
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by WesleyUC »

Hi, is there any more info about this?
dasfliege
Service Provider
Posts: 233
Liked: 53 times
Joined: Nov 17, 2014 1:48 pm
Full Name: Florin
Location: Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by dasfliege » 2 people like this post

Not yet. Still waiting for the next gen fix.
DarrenD
Service Provider
Posts: 11
Liked: 1 time
Joined: Feb 19, 2015 5:10 pm
Full Name: Darren Durbin
Location: Hampshire, UK
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by DarrenD »

Just tagging this so I can keep track. We're seeing what appears to be the exact same issue across a small 2019 cluster with 2019 guests on HPE servers and MSA2052 storage.
Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests