Backups causing high load to VM.

[JasonPGTF]

Recently I just implemented backups for all of my production VMs. There wasn't any backup done previously due to storage limitation & also license limitation.

I notice that after the backups being implemented, I've been receiving multiple complains that applications runs especially slow on the VM machines during backup hour. I've also notice high vCPU utilization during the backup period. May I know is there any way to reduce the load to the VM during backup?

[HannesK]

Hello,
just to make sure: you are doing VMware based backup and not agent based backup?

Are you doing application aware backup? If yes, the VSS snapshot commands can create a few CPU spikes. In general, you should not see load inside the VM

Do you use backup from storage snapshots? If not, you might see an IO impact as your VMs are running on VMware snapshots.

To test: simply do a VMware snapshot in VCenter without doing backup.

What backup mode do you use? Direct SAN? Hot-Add? NBD?


Best regards,
Hannes

[JasonPGTF]

Hi Hannes,

Yes they're using VMware based snapshot backup, no agent involve.

We have both app aware & non app aware backups. For the complains we receive from app user, is mainly on non app aware backups. I've not disabled the backup and there's no more complains receive from users about slowness.

I did not specifically select a backup mode, we're mostly using default settings. When I check on the backup completion details I'm seeing it's using Hot-Add.

[HannesK]

Hello,
okay, thanks for the details.

Especially for non app aware your behavior is strange. There is nothing that can create CPU spikes inside the VM as we do not even communicate with the VM. Are you sure it's about CPU?

Did you try just doing a VMware snapshot via VCenter (without doing backup) and just leaving it open for some time (similar time like backup takes) and remove the snapshot?

And just to make sure: you are not running on NFS storage?

In general: this is the point stopping to guess and ask support for help. Please post the case number for reference.

Best regards,
Hannes

[PetrM]

Hi Jason!

Yes, it's always a good idea to let our support team look for a root cause of the issue.

However, I'd like to share a couple of considerations as well:
CPU usage might be increased due to data compression performed on proxy side which acts in Hot-Add mode.
You may change compression level to Dedupe-Friendly and to check if CPU usage level becomes lower.
Another trick which might probably work for you is to decrease number of concurrent tasks at the level of proxy settings.
Moreover, an overall load level of the source infrastructure is decreased when less tasks are running in parallel.

Regarding slowness of applications on VMs, I'd suggest to check the following:
1) Try to create a snapshot on a VM and monitor how does application work when VM is running on a snapshot?
2) I'd suggest to take a look at storage I/O control option, it might be useful as long as you notice datastore read latency during backup.
3) If the slowness issue occurs during snapshot removing process at the end of backup, I'd recommend to take a look at this article.

Thanks!