Host-based backup of Microsoft Hyper-V VMs.
Post Reply
ChristineAlexa
Enthusiast
Posts: 44
Liked: 6 times
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

WeasleyUC
There are two issues going on here, for some hardware, See my testing results above.
And agreed, Neither of the issues are due to Veeam's code. Also agreed, the RCT one affects all vendors' products (that I'm aware of) using that technology.

Christine.
bhead
Influencer
Posts: 12
Liked: 6 times
Joined: Sep 30, 2020 9:18 am
Full Name: Bjoern Goerlich
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by bhead »

Hello everybody,

we're currently dealing with the exact same issue at one of our datacenter-sites:

Hyper-V 2019 Failover-Cluster with a total of 9 nodes.
The nodes are connected to the storage (All-Flash) via ISCSI on a 10 GBit/s network.
IO Performance randomly drops to nearly a complete stall.
Doing some test on any node, writing and reading data from the CSV, everything looks fine. Our network and our storage are as fast it gets.
Doing the same tests inside a random VM, writing data to and from a virtual hard disk (dynamic vhdx / static vhdx with a fixed size) will result in about 10% of the IO that we see on the parent partition.

We're currently dealing with the Microsoft Support. Right now they don't seem to have an idea on what's causing the latency and the low IO performance.

@pterpumkin: We would like to submit your MS-Ticket ID to our support contact. Maybe they can make the connection here to make some progress on both our cases? Maybe you could contact me via PM?
ChristineAlexa
Enthusiast
Posts: 44
Liked: 6 times
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

bhead:
Please see my reply earlier in this post, post383997.html#p383997

But this sounds more like the performance issue inside a HyperV VM that I documented with solutions for some hardware there with Hyperthreading vs the CBT/RCT bug that we are discussing as well. Note solving the performance in a VM does NOT solve the CBT/RCT bug, but on some hardware (all that I have personally tested), makes it happen far less frequently.

Just for a data points
- What vendor and model are the servers?
- What disk controllers and bus type are you using?
- What brand and model are the flash devices.

Thanks,
Christine
CasperGN
Novice
Posts: 6
Liked: never
Joined: Sep 29, 2020 11:59 am
Full Name: Casper Glasius-Nyborg
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by CasperGN »

I got ticket numbers from Pterpumkin.

0. 120060926001170 - Pter's own ticket.
1. 120063025001053
2. 120081024004251
3. 120040625000059
4. 119050122000550 - Veeam ticket number
5. 119100924004286
6. 120052425000074 – other backup vendor then Veeam, but same issue with performance.
7. 120081221000284
8. 120040625000059 - My own ticket which im pushing heavy on.

/Casper
CasperGN
Novice
Posts: 6
Liked: never
Joined: Sep 29, 2020 11:59 am
Full Name: Casper Glasius-Nyborg
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by CasperGN »

Hi All

Im still pushing onward with MS, only got them to say that they have escalated this internally and are awaiting updates.
Has any of you gotten any positiv outcome from your MS cases ?

/Casper
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by mkaec »

Casper,

Are you dealing with actual MS support or outsourced support? I've had a few incidents with outsourced support and they tried really hard to bury the case each time. With my last case, MS (appears to have) accidentally released the fix I was asking for. I asked the support rep to confirm the internal case he was referring to confirmed the fix and he ghosted me.
scootrz32
Novice
Posts: 4
Liked: never
Joined: Nov 17, 2009 2:25 am
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by scootrz32 »

Have you found any correlation. Like we see the issue and have Dell 14g servers with H740p controllers
ChristineAlexa
Enthusiast
Posts: 44
Liked: 6 times
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

scootrz32:
Have you looked at my research earlier in the post? Though all of my testing is with 11g, 12g, and 13g Dell servers (and the appropriate PERC controllers for the generations)
scootrz32
Novice
Posts: 4
Liked: never
Joined: Nov 17, 2009 2:25 am
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by scootrz32 »

I think I resolved ours today. I am going to test for a couple days. It was SentinelOne AV causing our issue. Once I removed it, it appears to be working.

Anyone else running S1?
CasperGN
Novice
Posts: 6
Liked: never
Joined: Sep 29, 2020 11:59 am
Full Name: Casper Glasius-Nyborg
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by CasperGN »

Hi

@mkaec Im dealing with MS directly.
@Scootrz32 We have SentinelOne AV, but we havent installed them on our HyperV 2019 hosts or this VM. Yesterday we moved one of the most affected vm's to another environment we have which is running Hyper V 2016, at the moment that setup is not used so much, so the vm have plenty of power and at the moment everything looks very good.
I noticed another thing yesterday during our storage migration for the move to Hyper V 2016, the VM's Disk Queue and latency dropped to almost nothing just when the storage migration finished. The VM where still on the same host and same storage, just a new storage volume. Im currently talking with our Storage vendor Infinidat on this issue. Because it looked very strange.

/Casper
scootrz32
Novice
Posts: 4
Liked: never
Joined: Nov 17, 2009 2:25 am
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by scootrz32 »

Once I removed it from the virtual machine. Mine seems to take snapshots and work OK.
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich »

Hi

Do you have news on your Microsoft cases ?
On my side, no significant progress

Best Regards
pterpumpkin
Enthusiast
Posts: 36
Liked: 4 times
Joined: Jun 14, 2016 9:36 am
Full Name: Pter Pumpkin
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by pterpumpkin »

Nothing yet :(
CasperGN
Novice
Posts: 6
Liked: never
Joined: Sep 29, 2020 11:59 am
Full Name: Casper Glasius-Nyborg
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by CasperGN »

My MS ticket was incorrectly closed with info that they are working on a fix, but the issue is different from what we are experiencing. But it might help others.

MS Comment
-------------------------
The instructions that my colleague Jose Ramon sent to you were to rollback and uninstall the updates since August 2020 on the Hypervisor (Not the virtual machine):

In this link you can see a discussion about this same issue that affects to backup and replication tasks.
https://docs.microsoft.com/en-us/answer ... er-au.html

The reason for that was that the team is working on a fix that will be released in the future. Currently we cannot provide a time when this will be available so you will have to wait for it. For that, and to try to mitigate the issue, my colleague suggested to delete the cumulative updates after KB4559003 (https://support.microsoft.com/en-in/help/4559003) (Do not uninstall this KB).
--------------------------

But we dont have that patch and doesnt see performance degradation doing image backup, but afterwards. But MS didnt seem to understand that, the guy who worked on our case, kept on troubleshooting as it is a VM issue.

Hope it help some of you.
yoda-ict
Lurker
Posts: 1
Liked: never
Joined: Dec 28, 2017 1:58 pm
Full Name: Frans Rampen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by yoda-ict »

Any news about this issue? Disabling Hyper-V helps but after a while we get a lot of StorageVSP Events nr 9 with description:

An I/O request for device 'C:\ClusterStorage\Volume7\ServerName\Virtual Hard Disks\Name.vhdx' took 18150 milliseconds to complete. Operation code = WRITE, Data transfer length = 524288, Status = SRB_STATUS_SUCCESS.
pterpumpkin
Enthusiast
Posts: 36
Liked: 4 times
Joined: Jun 14, 2016 9:36 am
Full Name: Pter Pumpkin
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by pterpumpkin »

MS closed my case and left us without a fix. They did this as they said it has been identified as a bug and there are no further troubleshooting steps, so there is no need to have a case open *sigh*.

I am emailing them weekly to ask if there is any update on the bug fix. This seems to be going nowhere and i have a feeling they don't actually check, they just say "no" and that they will let me know when a fix becomes available.

I have just tried deleting all of the RCT points for our affected VM and disabled our automatic daily live migration pre-emptive fix we had in place. The issue normally takes a few days to a few weeks to occur, so i will let you know how it goes.
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 1 person likes this post

On my side, the case is still open, normally a patch is in production but the person responsible for my case told me it could take several weeks
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 1 person likes this post

Small update, I just got Microsoft. The patch has been created. They are finishing testing it on their lab before releasing it.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

Hi Eluich,

Thank you for the update.

Does the patch have a reference number or any other ID?
Eluich
Influencer
Posts: 19
Liked: 18 times
Joined: Jul 06, 2020 2:31 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Eluich » 1 person likes this post

For the moment I don't have any other information
I just know that the patch concerns the VNext.
rold
Service Provider
Posts: 11
Liked: 7 times
Joined: Sep 14, 2016 12:04 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by rold »

Last update from MS on RCT performance issue ticket:

Code: Select all

1. This is not a code defect that we can fix straightaway. It is a design level change that needs approval from two CVPs before the work can officially start.
2. The 1C check-in deadline is over and given that 2C check-in deadline is on 1/8, I don’t think this will be ready by then. So my best guess is 3C (or later) for this and not earlier.
So .. maybe it will be fixed in vNext, maybe in vNextNext, maybe in some "Azure OS" only :D , maybe never.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by wishr »

This is indeed sad, but at least Microsoft is aware of the demand. Thank you for the update, Rold.
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Gostev » 2 people like this post

This just shows how much exactly Microsoft cares about on-prem Hyper-V :D I mean, it's not like they did not have an understanding of the issue over 1 year ago (see the first post of this topic). So normally, they should have secured all required approvals and planned the fix back then. But instead, it sounds like this process is starting only now? I don't get it.
rold
Service Provider
Posts: 11
Liked: 7 times
Joined: Sep 14, 2016 12:04 pm
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by rold »

Maybe you can port Veeam CBT driver to Server 2016 and later? :wink:
plinfra
Lurker
Posts: 1
Liked: never
Joined: Dec 21, 2020 12:48 pm
Full Name: PLINFRA
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by plinfra »

Same issue here:

Windows Server 2019 Datacenter (64 bit) with Hyper-V role assigned
ProLiant DL380 Gen9
5.5TB RAID6 (Created in Smart Array)

It was hosting 6VMs and currently is hosting 4 and still showing issues:
An I/O request for device 'device.vhdx' took 37186 milliseconds to complete. Operation code = WRITE16, Data transfer length = 4096, Status = SRB_STATUS_SUCCESS.
3x 2012r2
1x 2019
Andreas Neufert
VP, Product Management
Posts: 6699
Liked: 1400 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by Andreas Neufert »

Please allow me to add here that not all performance issues come from the RCT. Specifically if you have latency in the seconds area.

I had some customers that had real trouble with their ODX integration on the storage side. Create a test VM on the local disks of an Hyper-V host and compare the backup with the same VM on the storage (CSV).
ChristineAlexa
Enthusiast
Posts: 44
Liked: 6 times
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by ChristineAlexa »

plinfra wrote: Dec 21, 2020 12:56 pm Same issue here:

Windows Server 2019 Datacenter (64 bit) with Hyper-V role assigned
ProLiant DL380 Gen9
5.5TB RAID6 (Created in Smart Array)

It was hosting 6VMs and currently is hosting 4 and still showing issues:
An I/O request for device 'device.vhdx' took 37186 milliseconds to complete. Operation code = WRITE16, Data transfer length = 4096, Status = SRB_STATUS_SUCCESS.
3x 2012r2
1x 2019
Did you try any of the suggestions I listed above in this post? They help dramatically on some hardware for the Non-CBT performance issues.
DonZoomik
Service Provider
Posts: 368
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by DonZoomik »

I have noticed this issue occasionally on some VMs that run on Veeam repository's ReFS file system for reasons, but only overnight when there are a lot of backup, copy and replication jobs running at the same time. Queue depth and IO latency stay very reasonable in monitoring but Hyper-V has up to 2 minutes of IO latency according to Event Log and VMs become nearly frozen until issue clears by itself some time later.
Sidechannel attack mitigations are already disabled and classic scheduler is enabled in BCD, all VMs have HwThreadCountPerCore at 0. These VM's shouldn't have RCT enabled as these VMs have never been backed up through hypervisor (in-VM agent only).

SuperMicro SSG-6029P-E1CR24H
Broadcom MegaRAID 9361 (SuperMicro OEM) with 24*16TB SAS RAID60 (2 spans of 12) + 2*1.6TB SAS RAID1 CacheCade (SSD Caching).
mortendb
Lurker
Posts: 1
Liked: never
Joined: Nov 30, 2016 7:02 pm
Full Name: Morten Birkelund
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by mortendb »

Does anyone have any updates on this issue, i am seeing the same with my setup.
WesleyUC
Influencer
Posts: 12
Liked: 3 times
Joined: Aug 27, 2019 8:55 am
Full Name: LeslieUC
Contact:

Re: Windows Server 2019 Hyper-V VM I/O Performance Problem

Post by WesleyUC »

Andreas Neufert wrote: Dec 21, 2020 1:05 pm Please allow me to add here that not all performance issues come from the RCT. Specifically if you have latency in the seconds area.

I had some customers that had real trouble with their ODX integration on the storage side. Create a test VM on the local disks of an Hyper-V host and compare the backup with the same VM on the storage (CSV).
Correct, a lot of reactions on this forum item does not apply with the RCT and/or CBT problems that the original poster is referring to.
Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests