Virtual Machine performance hit during backups

[josh_swr]

I've noticed recently that during the time that VBR is backing up our Windows Server VMs in Hyper V, the performance of those VMs is taking a noticeable hit. Most noticeable is that our SQL server sometimes takes a lot longer to response to querys compared to when not running a backup. I believe this may be causing issues with our ERP software which users frequently experience timeout errors with.

The host version is entirely SAS SSDs across 2 RAID arrays for OS drives and data drives, so I wouldn't expect it to be a disk bottleneck.

Server Spec:
CPUs - 2x Xeon(R) CPU E5-2620 v4
RAM - 160GB DDR4

There's 4 windows server VMs running on the host and I've got Veeam set to only process one at a time already to minimise the performance hit.

Can anyone advise how to reduce the performance impact on the VMs during backups?

[karsten123]

reduce network bandwidth for repository. maybe your network is saturated

[josh_swr]

Thanks for the suggestion, however I don't think that's the issue here. Our backup server has two NICs on separate networks, one for the main LAN where the Hyper-V server lives and a storage-specific network where the backup repository lives. If the repository NIC were being saturated it wouldn't impact anything else.

I noticed the performance hit (the SQL query example I gave before) while logged directly onto the VM being backed up, so I do not think network saturation is the problem.

[karsten123]

the 2620 v4 is pretty old. what about upgrading your metal?

[josh_swr]

Financially it's not an option right now. The CPU load doesn't get that high anyway, even during backups, so I feel it should be able to cope.

[david.domask]

Hi josh_swr,

I suppose it's related to checkpoints during backup and the performance impact that checkpoints/snapshots can have.

Any chance there are left-over checkpoints on the SQL server? I would check not only in the checkpoint manager, but also check the VM's disk settings and see if the main disk is pointing to a checkpoint instead of a normal VHDX.

Similarly, I would be curious if you see the same performance issues with Veeam Agent for Windows backups for this server. The restore narrative for Agent-based backups is quite similar to that of VMs, so if you see less of an impact during backup with the Agent, consider using that to protect this workload.

[josh_swr]

Thanks I'll try the agent backup on a test VM and see what happens. I assume the backups from an agent can still be restored to a Hyper-V instance the same way as backups created from the VM checkpoints?

[david.domask]

Correct, the only difference is you'll use Instant Recovery instead of Entire VM restore for restoring the VM itself, though functionally they will have the same results. Other restore operations will work as with VM backups.

But I would double-check on if there may be any leftover checkpoints, as that can typically introduce such behavior.

Out of curiosity, what server version is the HyperV host, and is Veeam Backup and Replication installed on a VM or on the parent partition of the Hyper-V host? The latter can introduce such issues like this which is why you must not install VBR to the HyperV host itself, but instead as a VM or a dedicated machine.

[josh_swr]

There does not appear to be any existing checkpoints so I don't think that will be an issue here.

The HyperV host is Windows Server 2016. VBR is installed on a separate dedicated server (we use our old Dell server that this HyperV host replaced) which is also running Windows Server 2016.