Windows Server 2019 Hyper-V VM I/O Performance Problem

[stephc_msft]

"How it’s possible this works in Azure and not On-Premises?"
I'd never really though about that
The host virtualization platform in Azure is Hyper-V (and broadly identical to retail Hyper-V in on-prem Windows)
They of course do NOT use veeam (or anything else that might trigger the weird VM slow IO state) to do host level backup of live VM's ...

[stephc_msft]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

[Shane77]

"How it’s possible this works in Azure and not On-Premises?"
I'd never really though about that
The host virtualization platform in Azure is Hyper-V (and broadly identical to retail Hyper-V in on-prem Windows)
They of course do NOT use veeam (or anything else that might trigger the weird VM slow IO state) to do host level backup of live VM's ...

What do they use?

[Shane77]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

I do not have Storage Replica installed.

[mkeating44]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

Can also confirm Storage Replica is not installed and would (should) never have been installed.

[steendp]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

We do not have Storage Replica installed.

[SodaPop87]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

I can confirm none of my 3 hosts have this feature installed.

[Joris360]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

We are having this issue to, both the "2. The 'other issue'" as the "3a. Occasional or even frequent, unexpected Hyper-V-StorageVSP/Admin event 8's".

We are running a Dell HCI cluster, containing of 2 AX-640 nodes with Windows Server 2019 Datacenter on it. We have no separate storage or SAN, all disks are SSDs and contained in the nodes, we use Storage Spaces Direct (hyperconverged) for syncing between them.

We have 2 separate CSV's and all VMs are aligned in the cluster (so VMs and CSV are on the same host). For backups we are also using Veeam.

The impact is mostly on the 5 Sessions Hosts of RDS environment, were we can see the explorer and other application hang while the event is generated. We tried taking backups without CBT and even disable it completely on the CSV itself, but to no avail.

Since August 2022, we have an open case for this issue with Dell, as this was a new setup and we first suspected the hardware. Since Januari 2023, Microsoft is involved in the ticket aswel, but still no solution. We regularly have to take S2D an TSR logs for them, and numerous times installed the latest updates as they said "it will be fixed in this one", but until this day the issue is still present.

I can also confirm that we don't have the Storage Replica feature installed on the nodes.

As we see numerous other people saying that live migrating of the VM's solve the problem for a while, we're thinking to schedule this via a post-backup script (like slwaldrop and SodaPop87 are doing)

We're happy to assist with logs or other information to solve this problem once and for all.

[Shane77]

Has anyone tested if this problem is with on-host backups only?

Does the problem also show itself with off-host backups?

[Ci2Group]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

Can confirm no storage replica feature is installed or used. We do use veeam replication and we are finding that in our case the is generally only happening during replication jobs.
Replication jobs and regular backup jobs do not overlap.

[mkeating44]

One thing we have found in our environment which may or may not be related - one of our high use VMs that trigger the high latency behaviour has a stack of orphaned checkpoint files in storage for the high use virtual disk files. The checkpoints didn't exist on the VM, but we've had to restore this particular VM from a backup because of virtual disk corruption this week. I have a feeling it's all related but the orphaned checkpoint files are so old I don't have any logs to check against. I know that a checkpoint merge can trigger the high volume latency, so I wonder if the checkpoint merge is successful (or maybe not based on our VM failure), but the bug can cause the checkpoint files to not be deleted when the latency increases?

Have the issue on another server in another data centre today as well.

[GabesVirtualWorld]

Since the buzzzzz now is all about the new 2025 coming Any idea if it has the same issue? @stephc_msft

[dm_ch]

Hi everyone,

I would be very interested to know if someone at MS is still taking care of the existing problem!! @stephc_msft ? We are still struggling with individual (large) VMs which have slow IO performance after the backup (after 5-10 backup cycles?). Does anybody has already expirience with Veeam12?

BR DM

[slwaldrop]

Is there any progress being made for these issues?

[karsten123]

MSFT dont give a shit. customers? wtf are customers?

[Nick-SAC]

I wonder if a robust Class Action Suit would get MS's attention...

[stephc_msft]

Well one significant blocker is neither Microsoft nor Veeam seem to be able to repro.
The common factor is a veeam backup triggering the issue (although only on certain data disks on certain VM's -- usually a SQL data one doing high active IO?)
If Veeam could consistently repro, then they could try things I have suggested, such as doing the backup read UNBUFFERED, to see if that avoided the issue.
Or to provide experimental versions using slightly different approaches such that any willing customer could try those.

I am continuing to monitor and research and do whatever I can to get enlightenment on this.

[johan.h]

We've been working with Stephen Cole and team for some time, and the unbuffered read testing is planned. Getting a consistent repro case has been hard indeed. So we're considering a way to allow testing unbuffered mode by affected customers instead.

We do not know that this would address the issue but perhaps in combination with the Microsoft October fixes already provided, it could drive some progress, and by allowing testing, this can help customers confirm behavior. Once we're confident a reliable implementation for unbuffered backup support on Hyper-V is available, we believe we could provide a registry key for testing via a private fix that allows people to test this behavior and see if it makes a difference.

Keep in mind this will take some time as it requires updating and testing a lot of different components.

[stephc_msft]

Thanks

I was just about to add that the other unknown is why whatever the trigger is (from the veeam backup), why it doesnt cause the effect on all VM's and all VM data disks (as the trigger activity or creating the backup and reading the backup is the same in all cases).
Why just on some
Seems to depend on the IO pattern.

But as you say, if we could 'experiment' and find a way to avoid whatever the trigger is, then might give some enlightenment, and might allow customers to work around the issue in the meantime.

[john.aubrey]

Thanks for the updates. It's a very frustrating situation for everyone. If there is anything we can gather to help let us know.

[mkeating44]

We've been working with Stephen Cole and team for some time, and the unbuffered read testing is planned. Getting a consistent repro case has been hard indeed. So we're considering a way to allow testing unbuffered mode by affected customers instead.

We do not know that this would address the issue but perhaps in combination with the Microsoft October fixes already provided, it could drive some progress, and by allowing testing, this can help customers confirm behavior. Once we're confident a reliable implementation for unbuffered backup support on Hyper-V is available, we believe we could provide a registry key for testing via a private fix that allows people to test this behavior and see if it makes a difference.

Keep in mind this will take some time as it requires updating and testing a lot of different components.

If you need a testing platform I'm happy to offer one. Unfortunately we can't log a case with Microsoft but I can log a case with Veeam. We have had to work around the issue by Live Migrating the main VM that triggers the issue to another host after a backup using a scheduled task.

I was just about to add that the other unknown is why whatever the trigger is (from the veeam backup), why it doesnt cause the effect on all VM's and all VM data disks (as the trigger activity or creating the backup and reading the backup is the same in all cases).
Why just on some
Seems to depend on the IO pattern.

For our VM on an S2D cluster, the issue does cause performance degradation on the Cluster Shared Volume. The other SQL servers that can trigger the issue causes latency increase on all the volumes that the affected virtual drives are on, and that impacts on other VMs using the same storage (not as bad as the VM with the issue but the disk latency is noticeable). It's as if the storage layer can't reconcile the snapshot consolidation against the CBT and it's re-reading the disk or doing some high impact task for no reason.

[mlehmann]

Well one significant blocker is neither Microsoft nor Veeam seem to be able to repro.
The common factor is a veeam backup triggering the issue (although only on certain data disks on certain VM's -- usually a SQL data one doing high active IO?)
If Veeam could consistently repro, then they could try things I have suggested, such as doing the backup read UNBUFFERED, to see if that avoided the issue.
Or to provide experimental versions using slightly different approaches such that any willing customer could try those.

I am continuing to monitor and research and do whatever I can to get enlightenment on this.

This is patently false. I told you in previous posts and DM's that I could consistently reproduce it and I offered to give you full unfettered access to our environment.

your response was, and I quote:

Thanks for the offer, but at this stage Im not sure what I could quickly look at.
I have another offer of long term access to a repro system that I am about to look at

I also have a ticket open with Microsoft that you have specifically looked at and communicated on. I have provided mountains of data 5 separate times. and then your team gets back to me weeks later asking for the exact same data again.

[mkeating44]

I was just about to add that the other unknown is why whatever the trigger is (from the veeam backup), why it doesnt cause the effect on all VM's and all VM data disks (as the trigger activity or creating the backup and reading the backup is the same in all cases).
Why just on some
Seems to depend on the IO pattern.

The issue has come up again for us, and other VMs on the same cluster volume were painfully slow until the VM with the issue was live migrated, and it worked fine after. There is definitely impact to other VM workloads on the same clustered volume.

[mccartab]

We've just found we have this issue in a recent Hyper-V deployment.

The issue was discovered through poor performance on a high IO file server which is used to host Citrix profiles. Storage response times on the file server gradually increased to a level which resulted in poor performance in Citrix.

Issue is solved immediately by live migrating to a new host until the VM is backed up again by Veeam.

[Rmachado]

Just to help if someone have the issue

We have this issue in many servers (File Servers and SQL Servers).

What we do to as a workaround is to keep vhdx with less than 2 TB in size. We have one SQL running more than 60 days without any issue - 4 TB total volume and + 1800 databases.

[karsten123]

Veeam is going to officially support proxmox. Let’s knock Hyper-V into the bin. I don’t think, MSFT needs virtualization customers anymore.

[Joris360]

If you need a testing platform I'm happy to offer one. Unfortunately we can't log a case with Microsoft but I can log a case with Veeam. We have had to work around the issue by Live Migrating the main VM that triggers the issue to another host after a backup using a scheduled task.



For our VM on an S2D cluster, the issue does cause performance degradation on the Cluster Shared Volume. The other SQL servers that can trigger the issue causes latency increase on all the volumes that the affected virtual drives are on, and that impacts on other VMs using the same storage (not as bad as the VM with the issue but the disk latency is noticeable). It's as if the storage layer can't reconcile the snapshot consolidation against the CBT and it's re-reading the disk or doing some high impact task for no reason.

We are in the same boat as you. We notice performance degradation on our HCI S2D cluster. Its mostly noticeable on the RDS servers/Linux database server but also some impact on other VM's.
We tried the live migration (even a storage move) of the VMs, but this seems not to help much for us.

This is patently false. I told you in previous posts and DM's that I could consistently reproduce it and I offered to give you full unfettered access to our environment.

your response was, and I quote:



We also have a ticket open with Microsoft that you have specifically looked at and communicated on. I have provided mountains of data 5 separate times. and then your team gets back to me weeks later asking for the exact same data again.

We have a ticket running with Dell since august 2022, and since January 2023 MS was involved. We've taken the Dell SDDC logs, Microsoft TSSv2 logs, live dumps, event viewer exports, iDRAC SupportAssist collections, etc., about 50 times now.
They analyse it, find nothing that leads to the problem and ask the same thing couple weeks later. At one point they said it is fixed in Windows Server 2022 and we should upgrade all servers to fix it (which later seems not to be true).

For the moment we are waiting for some custom instrumented code from MS for our environment. This code should gather specific data/logs for the problem.

[josues]

Hello Veeam friends,

In our environment, we have implemented Hyper-V 2022 and are facing some performance issues. A possible solution that has been helping us a lot is converting the VHDX disks from dynamic to fixed. Surprisingly, the performance of the VMs has improved significantly.

Best regards,

Josue Duarte

[Knuppel]

We are experiencing performance issues also. I couldn't place these until i saw this topic. Can't really say when it started but somewhere within the last 2 / 3 months. In that period we also upgraded our hosts to 2022, and Veeam to v12.
Just to be sure, this occurs on a 2022 host running 2019 VM's also?

During backup things get unbearably slow, but during the day we also notice a huge performance drop.

[joelg]

In our environment, we have implemented Hyper-V 2022 and are facing some performance issues. A possible solution that has been helping us a lot is converting the VHDX disks from dynamic to fixed. Surprisingly, the performance of the VMs has improved significantly.

I've been going through that process myself.. I had thought all our VM's were using fixed disks from the start, but ran a report and found out how wrong I was. I have about 50 dynamic disks remaining, but have converted about 100 to fixed so far. I haven't been able to get any verifiable metrics, but VM's feel snapier.

Just for reference, it is Microsoft's recommendation to use fixed disks for performance: https://learn.microsoft.com/en-us/windo ... -file-type

Joel