Windows Server 2019 Hyper-V VM I/O Performance Problem

[Gostev]

Should I not install Veeam? My server setup is similar to the original, with 2019 Windows Server running 2 Hyper V virtual machines (one VM is 2019 and one is 2016). I can install Veeam on another HyperV server running on the same 2019 physical server; or, I could install it on a separate physical machine running 2008 R2. However, from what I have read if I'm correct, the issue is that Resilient Changed Tracking (RCT) gets enabled anyway on the 2019 server in order to track block changes in VMs and would potentially impact the performance on the 2019 server.

Is the performance noted in the forums an issue for everyone in this configuration?

I am somewhat tentative about just installing and running it not knowing if my server’s performance will be impacted

My questions are (1) should I install Veeam and (2) if so, where should I install and run it? I appreciate any guidance.

@Jrsac just installing Veeam will not do anything. The issue will appear if you start doing host-based Hyper-V backups, be it with Veeam or any other vendor, because that would enable RCT on the protected machines.

If you're really concerned about the issue, with Veeam you always have two additional options:
1. Make Veeam not to enable RCT on protected machines. But you will lose fast incremental backups and they will put more load on your production storage due to having to read all VM blocks to physically determine the changed ones.
2. Use classic agent-based backups (as if you were backing up physical servers), as opposed to host-based backups. Agent-based backup of VMs will not use RCT, but rather the Veeam's proprietary "in-guest" CBT driver.

Generally speaking, we recommend installing Veeam on a separate physical server - otherwise you will really complicate yourself the recovery when your Hyper-V server dies, taking along your backup server.

[Jrsac]

Thank you. I will try the agent approach for now and look to change if there is a fix at some point.

[Nick-SAC]

"... with Veeam you always have two additional options:
1. Make Veeam not to enable RCT on protected machines."

We tried disabling CBT at the Job Level but it had no perceptible effect on the I/O Delays

The article referenced does state that, “Veeam Backup & Replication enables CBT. You can disable it either at the host level or at the job level for troubleshooting purposes”, but I can’t find anyplace to disable it at the Host Level.

Does anybody know where that Host Level CBT ‘switch’ is located? -- Although I believe it's really RCT that's the issue on 2019.

[johan.h]

@Jrsac Sorry for not catching your question in my intial response. As Anton indicated there are always other ways to solve this. You can go agent-based, or turn off RCT per-host.

The way to stop using host-level CBT (RCT) is to go to the "Backup Infrastructure" tab, under "Managed Servers > Microsoft Hyper-V" select the cluster or standalone host in question, right click and select "Manage Volumes". You'll find the "Disable changed block tracking for this host" checkbox at the bottom. Do check that you don't have any remaining MRT/RCT files left on a VM-level afterwards, as if you were previously using this some might remain.

[johan.h]

Just a short update. I've been given a new private fix for RCT to test, targeting Windows Server 2019 RS5. I'll let you know as soon as I'm done testing. Feel free to DM me if you want to try it as well.

[hostedxl]

We're following this thread for quite a long time, and we're suffering the exactly the same problems. Since i seem to cannot DM johan.h maybe you can DM us ?

[Nick-SAC]

Hey Johan,

Thanks much for the instructions on how to disable Block Tracking at the Host level (I had been told early on by Tech Support that that wasn’t possible to do on Server 2019).

FWIW, Disabling it at the Host Level (as well as at the Job Level) did eliminate the I/O Delays entirely on our test box (interesting in itself because virtually all of the delays we had been seeing were NOT occurring during the Backup Jobs) but predictably the Job times increased so dramatically (from minutes to hours) that it’s not a viable workaround for most Production systems.

Thanks again
Nick

[Debinski]

Thought I'd chime in here:

I believe we narrowed our 2019 performance issue down to what version the actual virtual disk files (vhdx) were created on.

On our cluster, virtual disks created natively and running on 2019 do not perform well at all. Vhdx’s created on earlier versions of windows/hyper-v that we copied over and mounted on 2019, performance goes through the roof.

In contrast, we copied a 2019 vhdx to a 2012 vm, and it performed well.
So it seems to be narrowed down the abstraction layer or a driver on 2019 vm’s.

Reported this to MS as I have an open case. Will post back if we get a resolution.

I guess for now (future vms) just create a blank vhdx using 2016 or 2012, and keep copying it, mount it. then extend it, format it, etc.

(FYI, on a Win2019 S2D 6 node HCI cluster, all NVMe)

Steve

[Eluich]

Hi

You think the problem is with the virtual disk or the virtual machine where the virtual disk is attached ?

Best Regards

[Debinski]

Yup.

I don't think its the 2019 version of vhdx though (because it performs well on 2012). But rather the way the 2019 hypervisor or driver interfaces with the 2019 vhdx. Reason: Because the 2019 hypervisor with a 2012 or 2016 vhdx perform well, but not with a 2019 vhdx.

I still have an open case with MS. They thanked me for the info, But haven't heard back. Been a couple of days.

[eengland09]

I am seeing this, too. So glad I'm not the only one.... We have brand new SC5020 SAN with tons of space - brand new Dell EMC servers with 10GB nics (r640 and r630 mix in a 2019 Hyper-v Cluster) - we have been plagued with the I/O warnings - ESPECIALLY for Exchange - we noticed it when we moved from a 2013 Exchange to a 2019 DAG setup! I see lots of talk of disabling CBT/RBT - is that working for everyone??? I just opened a ticket on this.

[Eluich]

Hi eengland09

In my case I have 2 out of 7 clusters that have the problem.
The problem only occurs on VMs that host user profiles in UPD format.
All the VMs that are experiencing the problem were created natively on clusters 2016 and were migrated to 2019 via Veeam replication.
By disabling CBT I have reduced the number of times the problem occurs but as a result the backup time is very severely impacted

Best Regards

[eengland09]

Eluich,

That's interesting. I don't think we use UDP formatted stuff here. 2019 hyper-v seems to be a common denominator so far. Does this happen more for your exchange environment if you self-host? Or are you seeing it on a number of other 2019 VMs? From my research this only seems to be happening on 2019 VMs hosted on 2019 clusters (in hyper-v).

[eengland09]

Just a short update. I've been given a new private fix for RCT to test, targeting Windows Server 2019 RS5. I'll let you know as soon as I'm done testing. Feel free to DM me if you want to try it as well.

Did this fix your issue?

[eengland09]

So....I just turned of CBT at the node and job level for our exchange volumes and the I/O warnings on the nodes just started spamming like crazy. Set everything back to ON and its still spamming....this is crazy...

[Nick-SAC]

FWIW when I disabled the Block Tracking; I shut down the VM’s, Disabled it in VBR (at the Host and Jobs levels) and then deleted the respective VHD *.MRT & *.RCT Files before starting the VM’s

[eengland09]

Nic-SAC

That may be a route to go. Are you seeing that this fixed your I/O warnings on your nodes? And has it improved your exchange speed?

[Nick-SAC]

Eric,

The Production Server (a stand-alone system) that was seeing the problem has been taken out of service (for unrelated reasons).

When I disabled the Block Tracking on the Test Server, it completely eliminated the I/O Delays but since it was in a non-Production & isolated environment I can’t speak directly to any Exchange ‘speed’ impact it might have had... however, when the Production Server was in service, virtually every time the I/O Delays would occur on the Exchange Volumes there would be simultaneous Exchange DB Access/Timeout Warnings & Errors in the Logs... so yeah, we can certainly extrapolate that: No Block Tracking = No I/O Delays = No Exchange Delays = Improved Exchange Performance.

But once again; Disabling the Block Tracking increased the Backup times from minutes to hours (on a very small Server).

Nick

[Eluich]

Hi

I don't have an exchange server in my environment. The problem does not only occur on VMs in 2019 because for me it is only VMs in 2016 that are the problem.
I have other servers in 2019 with UPDs and I have no problem.

Best Regards

Eluich,

That's interesting. I don't think we use UDP formatted stuff here. 2019 hyper-v seems to be a common denominator so far. Does this happen more for your exchange environment if you self-host? Or are you seeing it on a number of other 2019 VMs? From my research this only seems to be happening on 2019 VMs hosted on 2019 clusters (in hyper-v).

[eengland09]

Eluich and Nick,

Thank you for your replies. Have any of you tried shutting down Veeam for a day or two to see if the warnings go away? I am still super hesitant to get rid of RCT/CBT and increase backup times

[eengland09]

Latest reply from Veeam is that this is a Microsoft issue and not a Veeam issue. RCT is a mechanism that exists entirely outside of Veeam.

[Gostev]

Right, see the very first post in this topic.

[DarrenD]

Just a quick note that we're still seeing this issue, but it's definitely intermittent - we had the eventid 9 issues at a customer on Friday, and they last saw the issue in May. Stopping the VM and removing the RCT files sorts the issue.

It appears the issue arises at this customer once the RCT file reaches a certain size. In this case, a 4TB disk on a file server, so the vast majority of this is static data that won't change, was up to 550MB for the RCT file and it was this disk where performance had fallen off a cliff.

If these files are bitmaps for changed files at a 16KB granularity I can't see how the RCT file should ever get that large on a largely static, un-defragged disk

[marksf]

For those who have experienced this, do you all have 10GB or above networking and/or SET as we have not experienced this issue until we have started using 25GB networking and SET instead of NIC Teaming.

[srdegeus]

We have 10GB networking (iSCSI) and don't use SET.
In our case the issue started after upgrading our 'test' 2016 cluster to 2019. Our current workaround is to live migrate our test SQL servers every morning via a scheduled task.

Not using RCT/CBT is not an option for us as while we have 10GB iSCSI, the appliance struggles to deliver 1GB. The daily backup-window already takes over 4 hours with CBT, so reading all data each backup is not an option for us. (Long term 'solution' for us is to rebuild the test cluster to Server 2016.)

[rold]

RCT issue is easy to reproduce if you are not using diskspd with file target. Use whole disk (#0 #1 etc). Its because some sort of cheating in diskspd code
Or use iometer with file and do not recreate this file each time.

[DarrenD]

Just as a quick update, and in case it helps anyone else, we thought we had this problem at two sites - it turns out it's only one site, and the other site - whilst showing identical symptoms had a different cause.

This second site had the same issues Event id 9, nearly always 10,000-15,000 ms delays, etc as mentioned by many others.

When moving the VM to a different host on the cluster didn't sort it, we looked for other issues. In the end, we found that the VHDX seeing the most of the issues ( the biggest one on the SAN at about 6TB ) was fragmented in to just under 1 million pieces.

Defragging this ( defrag.exe says all is fine but it's still 600k fragments ) and the server is back to full speed.

We've never really worried about defragging volumes on the SAN, so lesson learnt there.

[eengland09]

We have 10GB NICs for iscsi connectivity to the 10GB switch then to the SAN. And we have two 1GB nics teamed for communication between the servers in our cluster. I don't think we've ever defragged something manually here... Are you certain that has fixed your issues? Again - ours are primarily exchange related I/O Warnings.

[DarrenD]

Always hard to be 100% certain, but prior to the defrag we were seeing multiple event 9's per minute under load, and every several minutes when lightly loaded. Since the defrag we haven't seen a single error and nothing else has changed.

We still have one site where the 'move the VM to a new host' fixes the issue, so I guess that is still more of the core RCT issue being discussed here, but just wanted to share a fix for a problem that presents very similarly.

We don't have any sites with on-prem Exchange but could you be seeing it mainly there due to the sheer number of IO requests made by that server compared to others? We'd never defragged stuff on the SAN before - given the random access nature across multiple VHDs I didn't see the point.

[eengland09]

DarrenD,

Thanks for your reply. We are definitely seeing the I/O Warnings for Exchange - being that we are using a DAG and have 2 servers instead of our previous 1 server setup (which never saw I/O warnings that I could find in logs) - we are just being hit with these all day long with our DAG. I've pulled my hair out trying to find where it is coming from. My only guess is as good as yours - maybe RCT related, maybe some 2019 hyper-v Bug....I'm at a loss on what to try. We do occasionally see the I/O warnings for other 2019 VMs - very rarely. Defragging exchange volumes looks to be highly not recommended from what I've found just briefly searching :/