Windows Server 2019 Hyper-V VM I/O Performance Problem

[ChristineAlexa]

Bhead,
ok.. hoped my solution would work for you.. sorry that it didn't,

Christine

[kaffeine]

Hello everyone.

Unfortunately the IO issue is still present on Windows Server 2022 Hyper-V Failover Cluster. This was confirmed by the Veeam support team.

What file system where your VMs running on in the 2019 and 2022 clusters with this issue? ReFS or NTFS?

[bhead]

Hi,

no worries Christine. Thank you still.
We're using NTFS on CSV.

[bhead]

I've done some more testing today.
Here are some of my findings:

- Moving a VM to a different node brings back the IO after a backup session
- Creating a production snapshot will result in slower IO (as expected)
- Deleting a production snaphot will result in faster IO
- Running crash consistent backups will also result in slower IO
- A guest-os reboot will not change the IO speeds
- A guest-shutdown will bring back the IO performance after waiting for a couple of minutes before starting the VM
- Waiting a few minutes inbetween IO tests will result in faster IO

Something is definately happeing during or after a backup-session that will result in slower IO.
We're talking about major differences. One of my tests showed the following results:

After the backup:

Total IO - 20,55 MB/sec
Read IO - 14,19 MB/sec
Write IO - 6,07 MB/sec

After live migration:

Total IO - 460,65 MB/sec
Read IO - 322,48 MB/sec
Write IO - 138,17 MB/sec

Running another backup:

Total IO - 81,8 MB/sec
Read IO - 57,28 MB/sec
Write IO - 24,52 MB/sec

After live migration:

Total IO - 515,8 MB/sec
Read IO - 361,07 MB/sec
Write IO - 154,72 MB/sec

I was using a Windows Server 2019 guest vm with 156 GBs of memory and 6 vCPUs.
We're using all-flash storage on a 10 GBit/s ISCSI network!
The storage system is bored to death and we don't have any packet-drops on our switches.

We also reduced the hop count on the MPIO to just one hop. There are no changes loooking at latency!
We also have the latest HPE SPP 2022.09 installed on our hosts. Some of our hosts already have MS Updates (CUs) from this month.

We're now opening a ticket with Microsoft.

[ChristineAlexa]

Just making sure of one thing...
If you tried the hyperthreading disable, it has to be done to ALL nodes hosting the CSV.
Doing only one node won't help as a test.

[bhead]

Hi,

I used one node for testing. There was only one VM running on this host.
The same host was also owner of the CSV that was used to store the VM.
HT was disabled in the BIOS of the DL360.

We did not see any improvement

[ChristineAlexa]

Gotcha,
I know my testing and live environment are CSV on S2D, hence ALL nodes have to have it set, for it to work correctly.

If your CSV is shared storage, then (I would assume) only the owner node would need it set, and your test was valid.
If it was S2D, it wasn't, since all machines have to write at the same time due to drives on each node.

[bhead]

Hi,

we opened up a ticket with Microsoft this week.
We're expecting a feedback with at least some results tomorrow after the diagnostic results were reviewed.
I'll keep you updated.

Regards

[stephc_msft]

(Big struggle to get this system to let me log on again ..)
Anyway

If anyone has WS2022 or WS2019 definitively showing the RCT related issue, and would be prepared to test a private, please let me know.

[stephc_msft]

@bhead
in the tests above, which (if any) involve a CBT/RCT backup?
Or are you talking about OTHER io issues unrelated to RCT ?
Is performance after veeam backups WITHOUT CBT/RCT also showing issues?

[GabesVirtualWorld]

Hello everyone.

Unfortunately the IO issue is still present on Windows Server 2022 Hyper-V Failover Cluster. This was confirmed by the Veeam support team.
After every backup session the overall IO drops down 80% - 90%. Moving the VM to a different node brings back the IO. Another option is to reset the CBT for the VM or disk.
Yet that doesn't seem to be an acceptable solution for a larger datacenter with a large amount of VMs.
I just can't believe that neither Microsoft, nor Veeam has got no solution for this!

We ended up disabling CBT for our high IO VMs, deleting the MRT and RCT files for the virtual hard disks.
This may be suitable for smaller VMs where backup sessions are short due to small vhdx files.
Yet running backups on larger VMs will take alot more time now.

I am hoping that there might be a fix with Veeam 12 coming soon?
Maybe there's someone else who already opened up a ticket with Microsoft who has a final solution to this?

In my opinion, Microsoft and Veeam should work together on a solution! This is an awful situation.

Regards

We moved two big SQL VMs to a freshly installed Hyper-V cluster. Disabled CBT in the backup job and on the hosts. Removed the MRT and RCT files. But still have issues after backup. Only Live Migration fixes it.
Hyper-V 2019 October patch level.

[CelticDubstep]

Not sure what our long term fix will be. I've had to migrate back to Windows Server 2012 R2 which has been working flawlessly. We're a small business and don't have servers we can do live migrations to (plus we're all SAS HDD and no SSD). Switching from 2019 to 2012 R2 reduced Active Full backup from 60+ hours to 15-17 hours and improved the performance for our engineers where it took 3-5 minutes to open a model and 30+ seconds to do a basic save to central down to less than 30 seconds to load and less than 10 seconds to save. Needless to say, the engineers & their managers are much happier.

I tried to turn off CBT, HyperThreading, and various other suggestions but it only made things worse. Our main file server is nearly 8 TB. Our backup repository is 48 TB.

We're either going to be stuck on 2012 R2 past end of life or finding a different backup solution. Not sure what else to do.

[BigBadBarry]

Hi,

we opened up a ticket with Microsoft this week.
We're expecting a feedback with at least some results tomorrow after the diagnostic results were reviewed.
I'll keep you updated.

Regards

Hi Mate,

Did you get anywhere with Microsoft on this? Experiencing a similar issue and interested in the fix or recommendations on this one.

Cheers!

[Nick-SAC]

Seeing as how it’s been over 3 years since I first posted this problem and brought it to Microsoft’s attention, we’re close to 300 posts into it in this thread alone... and we’ve undoubtedly spent many thousands of collective hours on it... I’m beginning to lose hope that it’ll ever be fixed, so...


After some fairly extensive testing with some non-production Servers & VM’s the bottom line appears to be this:

On a 2019 Hyper-V Host:
Running VM’s with a Configuration Version of 9.0 : the Host generates the Event ID-9 I/O Delay Warnings
Running VM’s with a Configuration Version of 8.0 : the Host DOES NOT generate the Event ID 9 I/O Delay Warnings


So, since I now MUST deploy a couple of new Servers soon, I’m considering these options:

1) Try another Backup solution.
I really, REALLY don’t want to go this route as I absolutely love Veeam and (while I haven’t researched anything else in a long time) I’m guessing that this I/O Delay problem may still exist when using CBT with any Block-Level Backup product (and while disabling CBT eliminates the I/O Delays, it increases the Backup Job times to unacceptable durations for us).

2) Creating & keeping all the new Server’s VM’s at Configuration Version 8.0 unless & until there is a well-confirmed fix for the problem.
Although I do have a number of VM’s running Configuration Versions earlier than the Host’s and I haven’t noticed any problems in doing so, I really haven’t done much research into the potential ramifications of this rather unorthodox configuration and would appreciate any thoughts on this approach.

Anyone? Anyone?

Thanks,
Nick

[akrimmer]

Interesting that your testing showed no issues with config version 8.0 as the majority of our servers are still 8.0 and we saw major issues, specifically with our SQL servers. Out short term fix was to live migrate the VM shortly after a backup ran. The longer term fix was to create a separate 2016 cluster and migrate our "problem" VMs back to that cluster until MS has a fix.

We had a premier case open for months and after many frustrating sessions with support, they finally admitted to this being a bug and the numerous other clients who have also reported the issue. They claim the issue is still being worked on but they have no ETA on a fix. We are supposed to be a on list to be notified once fix is released but I have little hope that a fix will be released and if it is, that we would actually get notified.

[mkaec]

Is your case still open? I had Microsoft put me on a notification list for a hotfix, then close my case. Then the support rep left and the "list" was probably a task in his Outlook. I had to open a new case and prove that my original case was in before mainstream support ended in order to get the hotfix.

If the case is still open, I wouldn't wait for a notification. I'd ping the case every 6 weeks asking for a status update.

[DarrylvdPeijl]

Just to make this clear:

Microsoft has a private fix for testing, you can reach out to stephc_msft in this thread for this and pass him your case ID.

[bhead]

Is your case still open? I had Microsoft put me on a notification list for a hotfix, then close my case. Then the support rep left and the "list" was probably a task in his Outlook. I had to open a new case and prove that my original case was in before mainstream support ended in order to get the hotfix.

If the case is still open, I wouldn't wait for a notification. I'd ping the case every 6 weeks asking for a status update.

Hello,

I've got some news. I just got off the phone with a Microsoft engineer and he told me that it is a well known issue being caused by "Veeam and Microsoft"!
Right now they are expecting a bug fix to be released with the regular monthly updates in february or march this year! There will be no further notification. We are advised to keep track on the update news that are published by Microsoft every month! I was also promised to receive a summary with a confirmation of this statement by mail today!

I know there's no guarantee that there will be a fix, soon. But at least Microsoft admitted that there is a known issues and that they are working on a fix!

Let's just wait and hope.

Furthmore I did run some further tests earlier today. In our environment the issue only seems to be only when working with block-sizes smaller than 64K.

Regards

[nexusdis]

Hello,

is there a workaround to speed up veeam backup or I/O of these vm?
host: win 2019 standard, vm on raid 10 SSD
guest: win 2019 standard > 4 vhdx, 2 of these vhdx has backup speed of 4mb/s (1024GB vhdx), the other 3 vhdx are backups with 200-300mb/s over network

[fss]

@bhead
in the tests above, which (if any) involve a CBT/RCT backup?
Or are you talking about OTHER io issues unrelated to RCT ?
Is performance after veeam backups WITHOUT CBT/RCT also showing issues?

Hi Steph_msft,
we have the discussed issue. Can you send me the hotfix for testing?
My Veeam Case-Nr.05807095.
thanks a lot.
fss

[ARC841]

@Steph_msft
how are you?
Can you share somehow that private? Or need to open a case with MSFT?

[steendp]

Hi @steph_msft

I'm unable to contact you directly. I have an open case with MS and would like to receive the private hf is possible.

Thx!
Steen

[steendp]

@All

I'm told that the january update for Windows 2019 includes a fix for this issue. I'm currently fasttracking the update to our hpv-servers, but if anyone beats me to it, I'd like to receive feedback

Br Steen

[BigBadBarry]

Do you know which KB? Not yet deployed this months patches as waiting for any known issues

[steendp]

It is not listed in the KB (KB5022286). I've asked the supporter that is handling my case at MS for specific release notes, but he was unable to supply them (as well as the private fix).

[stephc_msft]

Apologies
Looks like the possible fix I thought was going to be there, is not actually in the WS2019 Jan 2023 update after all.
And still being investigated.
Will update if/when I have any definitive news.

[mkaec]

I'm starting to feel like Charlie Brown trying to kick the football.

[BigBadBarry]

Has anyone who has an open ticket with Microsoft received the update early for testing? Looking at opening a ticket but if the response is that It's still a WIP and might be available in a few months then there's not much point

[joelg]

I'm building a test environment to try it out currently. Don't want to apply it to our prod

Joel

[ChristineAlexa]

@Joelg
Did they happen to provide a "description" for the patch (to see what they think it fixes)?