Slow backups since 2019 upgrade

[NIRC]

Case number #04972449

Hi
We have recently upgraded our VM servers and hosts from 2012R2 to 2019 and are seeing significantly increased backup times.
We have a cluster of 3 hosts, running a total of about 35 VM servers.
Backups of the VM's used to be taking for example between 10mins and 20mins each. A number of them are now regularly taking between 2hrs and 6hrs each. So the entire backup job of one host, what was taking about 2.5hrs is now taking close to 11hrs for one host.
The time therefore that it is taking to backup all three hosts and all VM's has skyrocketed from about 7hrs to close to 24hrs in total.
It appears to coincide with the host or VM upgrades. Nothing else has changed regarding the network infrastructure or NAS destination.
It appears to be the 'processing' stage mainly that is taking all the time. The backup job is processing about 5Tb of data and transferring about 80Gb. These numbers have not changed significantly from before the 'go slow'.
Any ideas of things I can check or change? I suspect it is something with the host/s, not the VM's themselves that is causing it, but that is just a gut feeling at this point.
Thank you.

[HannesK]

Hello,
sounds like change block tracking is not working. I remember that was normal when 2012R2 to 2016 upgrades were common. Cluster level 9 and VM level 8 must be available to make Microsoft resilient change tracking (RCT) work (might be different numbers today, they are from 2016).

Best regards,
Hannes

[NIRC]

Hi Hannes
Cluster level is currently at 10 (server 2019 level), so that should be OK

- to find out the current cluster level, open powershell on the host and run
get-cluster | select ClusterFunctionalLevel
- if you need to upgrade the cluster level after all hosts in the cluster have been upgraded to 2019, just run command
update-clusterfunctionallevel

VM config level is currently 5 for the majority of the individual VM's by the looks, which is probably not OK since everything is now 2019.
(For Server 2019 it should be now config level 9 I believe).

- to find out the current Config version for all your VM's on the host, with powershell still open on the host run
get-vm * | Format-Table Name, Version
- If need to upgrade the VM config level,
shut down the VM, right click on it in Hyper-V Manager and click on Upgrade Configuration Version
- or alternatively in powershell you can run
Update-VMVersion <vmname>

Note that if you update the VM config version, and you have replication running, both the primary host and the replication host must be running the same version of Server. If your primary host is running server 2019 and your replication host is running 2016, if you upgrade your VM config to 9, I do not believe it will then be allowed to be replicated to a replication host that is running Server 2016.

I will shut down a couple of the VM's that are taking the longer time and update their config versions and see how it goes next backup which will happen tonight.
My initial thought was something to do with the CBT as well, the time for the backup seemed like "doing a full backup" time frame instead of just 'what has changed' time frame.
Thank you kindly for the insight, will let you know how it goes tomorrow.

[NIRC]

Just a further update, I have been recommended to check and, if necessary, change our on-host /off-host proxy setting for the backup jobs as well
However I want to first just see what sort of difference upgrading the VM configuration version only makes with tonight's backup. Call me curious haha.
Tomorrow I will make the change to the on-host / off-host proxy setting as well, but if I do both at once right now, I won't really know which one of them fixes it, and I don't like that

[nmdange]

VM config level is currently 5 for the majority of the individual VM's by the looks, which is probably not OK since everything is now 2019.
(For Server 2019 it should be now config level 9 I believe).

It's definitely this... you won't get CBT until your VMs are on config version 8 or higher.

[NIRC]

OK well testing from last night shows that changing the VM config level for some of the slow VM's did not make any appreciable difference.
What took 5hrs the night before for one VM still took 4hrs 45m last night, even after upgrading its VM config version from 5.0 to 9.0.
I have now changed the on/off host proxy setting, from off-host to on-host, for the backup job.
I am pretty sure I had that set before (ie it was set to on host, not off host as it was before I just changed it), but we did recently also upgrade Veeam itself to the latest version, perhaps that upgrade also reverted the on/off host setting.
Anyways will see how it goes this eve. Note Friday today, so I won't update this for a couple of days.
Thanks all.

[NIRC]

Of course I will be upgrading all the VM config versions anyway, as they should definitely be on v9.0 now that the host and VM itself are server 2019.
I am just informing anyone reading this that _only_ upgrading the VM config version has not appeared to have changed much.

[nmdange]

Note that after the upgrade, the first backup is still a full backup, to initialize the CBT bitmap. The second backup should be much faster.

[NIRC]

Thanks nmdange
I am just noting that after changing the on-off proxy setting and upgrading all the VM Config versions from 5.0 to 9.0 and let it run for a few days, it seems OK again
The biggest change appears to have come from changing the Proxy setting for the backup jobs from off-host to on-host.
This was for sure how it was set before, so I am curious what would have changed it back to off-host. Whether that was from upgrading Windows or upgrading Veeam, I am unsure, as those two upgrades happened with a day or two of each other on the same backup server, at the same time I noticed the longer backup times.
Regardless, I would advise that it is well worth checking that setting.
In my case, changing from off-host to on-host with the backup job configuration on the Veeam server as well as updating all the VM config versions from 5.0 to 9.0 has bought the average VM backup time back down from 2 to 6 hrs to 10 to 40mins.
Thank you.

[HannesK]

Hello,

The biggest change appears to have come from changing the Proxy setting for the backup jobs from off-host to on-host.

that makes zero sense to me. While off-host proxy is the default setting, it will always fall-back to on-host proxy, because you don't have an off-host proxy as far as I see. Or do you really have an off-host proxy?

As everything is working now: if you set back to off-host, you should see similar backup times. If not, then it would be great if you have the time to open a support case and send logs to support and post the case number here.

Best regards,
Hannes

[NIRC]

Hi Hannes
For interests sake, I will set one of the the proxy setting on one of the three backup jobs (we have a cluster of 3 nodes, one backup job for each node) from on-host back to off-host and see how it performs on tonights backup.
Your understanding is correct, we do not have an off-host proxy.
I'll let you know tomorrow whether the one backup job on which I change the proxy setting is significantly slower than the other two, or not.

I see what you mean though, I have just gone in to change the proxy setting for one of the backup jobs and for sure there is also a check box right there to
"Failover to on-host backup mode if no suitable off-host proxy is available", so technically since we HAVE no off-host proxy configured, it should be using on-host anyway, even if the radio box is selected to attempt to use an off-host proxy

[NIRC]

Well changing the setting from on-host to off-host with the check box for fall back on one of the backup jobs has made no difference at all to the backup speed for that job. It is still running 'normally', aka 45mins instead of 6 or 7 hrs.
Despite my best efforts in being methodical about trying to resolve this issue, I am still no clearer what caused this issue and what fixed it unfortunately.
All I can suggest is checking the two things from above, if you run into the same issue.
Regardless, mine is back to working properly, so all good for me.
Thanks

[HannesK]

cause: CBT breaks because the backup switches from Veeam CBT driver to Microsoft native resilient change tracking (RCT)

what fixed it: switching cluster level and VM level. After switching, one slow backup is expected

[Larry s]

Is there a way to tell what type of CBT Veeam is using? We are 100% Windows Server 2019 Hyper-V with Veeam V11 and the latest service pack. I'm just trying to make sure it's using RCT.

Thanks.

[HannesK]

yes. Up to Server 2012R2, we use the Veeam CBT driver. Since Server 2016 we use Microsoft RCT.

[Highfidelitybackup]

Case number #04972449

Hi
We have recently upgraded our VM servers and hosts from 2012R2 to 2019 and are seeing significantly increased backup times.
We have a cluster of 3 hosts, running a total of about 35 VM servers.
Backups of the VM's used to be taking for example between 10mins and 20mins each. A number of them are now regularly taking between 2hrs and 6hrs each. So the entire backup job of one host, what was taking about 2.5hrs is now taking close to 11hrs for one host.

We had a similiar problem, with server 2019.
And the fix was:
"1.) You can disable a client's RSC for IPv4 traffic using PowerShell: Disable-NetAdapterRsc -Name $nic -IPv4.

2.) You can check a Hyper-V vSwitch's current RSC status using PowerShell: Get-VMSwitch -Name $vSwitch | Select *RSC*.

3.) You can disable a Hyper-V vSwitch's RSC in full using PowerShell: Set-VMSwitch -Name $vSwitch -EnableSoftwareRsc:$FALSE. This will not impact existing vSwitch connections."

I our scenerio it was enough to do step one, on our RepoServers (Virtual Machines with Server2019), after that the performance was back again.

https://www.reddit.com/r/sysadmin/comme ... ue_to_rsc/

Cheers
Peter

[nexusdis]

where I have to setup this command: Disable-NetAdapterRsc * ?
on my hyperv 2019 host or in my server 2019 vm ?

thx