Windows Server 2019 Hyper-V VM I/O Performance Problem

[stephc_msft]

"How it’s possible this works in Azure and not On-Premises?"
I'd never really though about that
The host virtualization platform in Azure is Hyper-V (and broadly identical to retail Hyper-V in on-prem Windows)
They of course do NOT use veeam (or anything else that might trigger the weird VM slow IO state) to do host level backup of live VM's ...

[stephc_msft]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

[Shane77]

"How it’s possible this works in Azure and not On-Premises?"
I'd never really though about that
The host virtualization platform in Azure is Hyper-V (and broadly identical to retail Hyper-V in on-prem Windows)
They of course do NOT use veeam (or anything else that might trigger the weird VM slow IO state) to do host level backup of live VM's ...

What do they use?

[Shane77]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

I do not have Storage Replica installed.

[mkeating44]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

Can also confirm Storage Replica is not installed and would (should) never have been installed.

[steendp]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

We do not have Storage Replica installed.

[SodaPop87]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

I can confirm none of my 3 hosts have this feature installed.

[Joris360]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

We are having this issue to, both the "2. The 'other issue'" as the "3a. Occasional or even frequent, unexpected Hyper-V-StorageVSP/Admin event 8's".

We are running a Dell HCI cluster, containing of 2 AX-640 nodes with Windows Server 2019 Datacenter on it. We have no separate storage or SAN, all disks are SSDs and contained in the nodes, we use Storage Spaces Direct (hyperconverged) for syncing between them.

We have 2 separate CSV's and all VMs are aligned in the cluster (so VMs and CSV are on the same host). For backups we are also using Veeam.

The impact is mostly on the 5 Sessions Hosts of RDS environment, were we can see the explorer and other application hang while the event is generated. We tried taking backups without CBT and even disable it completely on the CSV itself, but to no avail.

Since August 2022, we have an open case for this issue with Dell, as this was a new setup and we first suspected the hardware. Since Januari 2023, Microsoft is involved in the ticket aswel, but still no solution. We regularly have to take S2D an TSR logs for them, and numerous times installed the latest updates as they said "it will be fixed in this one", but until this day the issue is still present.

I can also confirm that we don't have the Storage Replica feature installed on the nodes.

As we see numerous other people saying that live migrating of the VM's solve the problem for a while, we're thinking to schedule this via a post-backup script (like slwaldrop and SodaPop87 are doing)

We're happy to assist with logs or other information to solve this problem once and for all.

[Shane77]

Has anyone tested if this problem is with on-host backups only?

Does the problem also show itself with off-host backups?

[Ci2Group]

Can I just confirm for people hitting the issue -- do you have the Storage Replica feature installed on the Hyper-V host systems, even if NOT actually doing any SR replication?
And if so (and not actually using SR), can you remove that Feature.

Can confirm no storage replica feature is installed or used. We do use veeam replication and we are finding that in our case the is generally only happening during replication jobs.
Replication jobs and regular backup jobs do not overlap.

[mkeating44]

One thing we have found in our environment which may or may not be related - one of our high use VMs that trigger the high latency behaviour has a stack of orphaned checkpoint files in storage for the high use virtual disk files. The checkpoints didn't exist on the VM, but we've had to restore this particular VM from a backup because of virtual disk corruption this week. I have a feeling it's all related but the orphaned checkpoint files are so old I don't have any logs to check against. I know that a checkpoint merge can trigger the high volume latency, so I wonder if the checkpoint merge is successful (or maybe not based on our VM failure), but the bug can cause the checkpoint files to not be deleted when the latency increases?

Have the issue on another server in another data centre today as well.

[GabesVirtualWorld]

Since the buzzzzz now is all about the new 2025 coming Any idea if it has the same issue? @stephc_msft

[dm_ch]

Hi everyone,

I would be very interested to know if someone at MS is still taking care of the existing problem!! @stephc_msft ? We are still struggling with individual (large) VMs which have slow IO performance after the backup (after 5-10 backup cycles?). Does anybody has already expirience with Veeam12?

BR DM

[slwaldrop]

Is there any progress being made for these issues?

[karsten123]

MSFT dont give a shit. customers? wtf are customers?

[Nick-SAC]

I wonder if a robust Class Action Suit would get MS's attention...

[stephc_msft]

Well one significant blocker is neither Microsoft nor Veeam seem to be able to repro.
The common factor is a veeam backup triggering the issue (although only on certain data disks on certain VM's -- usually a SQL data one doing high active IO?)
If Veeam could consistently repro, then they could try things I have suggested, such as doing the backup read UNBUFFERED, to see if that avoided the issue.
Or to provide experimental versions using slightly different approaches such that any willing customer could try those.

I am continuing to monitor and research and do whatever I can to get enlightenment on this.

[johan.h]

We've been working with Stephen Cole and team for some time, and the unbuffered read testing is planned. Getting a consistent repro case has been hard indeed. So we're considering a way to allow testing unbuffered mode by affected customers instead.

We do not know that this would address the issue but perhaps in combination with the Microsoft October fixes already provided, it could drive some progress, and by allowing testing, this can help customers confirm behavior. Once we're confident a reliable implementation for unbuffered backup support on Hyper-V is available, we believe we could provide a registry key for testing via a private fix that allows people to test this behavior and see if it makes a difference.

Keep in mind this will take some time as it requires updating and testing a lot of different components.

[stephc_msft]

Thanks

I was just about to add that the other unknown is why whatever the trigger is (from the veeam backup), why it doesnt cause the effect on all VM's and all VM data disks (as the trigger activity or creating the backup and reading the backup is the same in all cases).
Why just on some
Seems to depend on the IO pattern.

But as you say, if we could 'experiment' and find a way to avoid whatever the trigger is, then might give some enlightenment, and might allow customers to work around the issue in the meantime.

[john.aubrey]

Thanks for the updates. It's a very frustrating situation for everyone. If there is anything we can gather to help let us know.