High CPU usage by VeeamAgent.exe

[raphael@schitz.net]

Hello everyone,
I got a strange "issue" that i could reproduce on a test infrastructure :
My VCenter is a vm with veeam B&R installed on it (i know, i know...), the backup was all good with 4.0 but since i updated to 4.1, when the veeam job backup the vm where it is installed (the vcenter), the VeeamAgent.exe use many more cpu than for other vm.
I excluded this vm from the backup job, no more problem.
I also build a test infrastructure in my lab, the problem also occurs (even if it is not an upgraded veeam)
Any ideas ?

[Gostev]

Load can be different between VMs depending on amount of changes since last pass. Generally, it is perfectly normal for VeeamAgent.exe to load CPU heavily - because of on-the-fly compression.

This is why we are recommending multi-core physical computers, and multi-vCPU VM configurations. We recommend 4 cores for Optimal compression, and 8 cores for Best compression. Try switching to Low compression if your Veeam Backup VM has 1 vCPU.

[raphael@schitz.net]

I'll try that, thanks !

[raphael@schitz.net]

Anyway, i don't understand why this high cpu usage doesn't occur with the v4.0...

[Gostev]

Under the same conditions, 4.0 and 4.1 will produce the same CPU load. It is absolutely impossible for 4.0 NOT to load CPU heavily with the default job settings (Optimal compression) when processing VM with significant amount of changes.

One idea I have, is if you upgraded to 4.0 from previous version, then the upgraded jobs would have Low compression settings (which is what v3 Optimal is equal to in v4), which may explain why you have not seen the high CPU load before. If you re-created the job later, the new default in v4 is now Optimal compression, and thus you have started to see the CPU load.

[E.Kok]

Hi,

I'm trial-running the 4.1 and also see the 100% CPU-utilization on a Xeon 3.2 Ghz processor. Even when compression=none. So why the high utilization when the software only needs to copy the VM over the fiber SAN connection?

Veeam is installed on a physical server (HP-Proliant) with Windows 2008 64-bit with 7 GB RAM.
The Backup-Mode is VMware vStorage API 'SAN only".
Also disabling deduplication doesn't help.

Ed

[Vitaliy S.]

Hello Ed,

Could you please clarify how many jobs do you have running at the same time? Also, could you please check with Task Manager what other process might be taking all CPU resources while you run the job? Besides, do you have any real-time antivirus software installed on the backup console?

Thanks!

[Gostev]

I'm trial-running the 4.1 and also see the 100% CPU-utilization on a Xeon 3.2 Ghz processor. Even when compression=none. So why the high utilization when the software only needs to copy the VM over the fiber SAN connection?

Block hashing (for integrity control) is still performed even with deduplication disabled.
What performance are you getting?

[E.Kok]

Hello Ed,

Could you please clarify how many jobs do you have running at the same time? Also, could you please check with Task Manager what other process might be taking all CPU resources while you run the job? Besides, do you have any real-time antivirus software installed on the backup console?

Thanks!

Hi,
there is just one job. So this is the only-one running. The TaskManager shows the Veeam.Backup.Manager.exe at about 94% in the beginning of the job, and later on it is 'VeeamAgent.exe *32' on 85....90 percentage. When the job is stopped the machine idles at about 5%.

@Gustev
The "Processing rate' shows about 3 MB/s in the beginning and in the end about 9 MB/s. This is after 11 hours for a VM of 220 GB.

Ed

[Gostev]

Wow, this is totally unexpected... the numbers look very much like backup is installed in 1 vCPU VM running on weak host (I get same numbers in my lab)... may be there is some 3rd party software affecting this? Antivirus, firewall, traffic scanner? HBA driver misbehaving? Also, your processing performance is at least 10x slower then what it should be in this config (likely starved by this CPU issue).

It would help if you send our support or logs, since agent dumps some host diagnostic information such as available cores etc. - this information may help our support to troubleshoot.

[E.Kok]

@Gostev

Well, it really is a physical machine: ProLiant ML370 G4 with QLogic fiber-connection to HP-EVA-4400 SAN.
The intention is to use Veeam for replication to a disaster-recovery-site.
There is no AntiVirus, firewall or traffic scanner. And besides, for the last two; when the session is started it should purely make use of the fiber-connection to the SAN. The same HBA-driver is being used when this machine makes nightly backup's to a tape-unit. I know, Veeam recommends no other software should be on this server. But he, how may physical servers do I need in a virtualized environment? One for DC, one for backup, one for replication and one for vCenter. That is too much for a small company like we are. I do not want Veeam on a VM because I don't want it to impact the performance of our ESX-cluster. And for now Veeam is being tested during the day and the backup-software is running at night.
I already sent the log-files to support and their response was: "Probably this machine isn't enough powerful fro performing several operation simultaneously". But it really isn't doing more then one task at the time and this can clearly be seen in the Taskmanager.

Ed

[Gostev]

Can you try backup in Virtual Appliance mode with Veeam Backup installed in the 4 vCPU VM (just temporary for a test, to see the performance). I am 100% sure the issue here is with host. It is clearly starving CPU (just by looking on 5% at idle - this is waaay too much for an idling server). Did you check, if may be BIOS settings were reset to fail-safe configuration?

Actually, I have just googled up the specification for this server, and this looks to be very old hardware (released in 2004). This can be the issue indeed, our support is probably right. As per our System Requirements in the release notes document, we require modern 4 core processor for best performance. Besides CPU, another issue is very slow PC2100 memory, which is important for our engine, as it affects realtime data hashing speed.

[E.Kok]

Can you try backup in Virtual Appliance mode with Veeam Backup installed in the 4 vCPU VM (just temporary for a test, to see the performance). I am 100% sure the issue here is with host. It is clearly starving CPU (just by looking on 5% at idle - this is waaay too much for an idling server). Did you check, if may be BIOS settings were reset to fail-safe configuration?

Actually, I have just googled up the specification for this server, and this looks to be very old hardware (released in 2004). This can be the issue indeed, our support is probably right. As per our System Requirements in the release notes document, we require modern 4 core processor for best performance. Besides CPU, another issue is very slow PC2100 memory, which is important for our engine, as it affects realtime data hashing speed.

Followed your advise and built a tmp Server (W2K8) with 4 x 3.0 Ghz cores running Veeam. Tested with a small (10GB) VM, network to a share and that went fine. Processing rate was 35 MB/s. Then tat same VM over the network to a share in 'appliance mode' gave a result of 31 MB/s. The CPU's were running at about 50%. Then I created a job for backing-up our Exchange-server. The performance plunged to 6 MB/s but the processor performed the same at around 50%. At night the job failed:

Code: Select all

CreateSnapshot failed, vmRef "vm-676", timeout "1800000", snName 
"VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do 
not delete this snapshot. It is being used by Veeam Backup.", memory 
"False", quiesce "True"
Cannot create a quiesced snapshot because the create snapshot operation 
exceeded the time limit for holding off I/O in the frozen virtual machine.

Could this be caused by our backup-software ArcServe creating a snapshot of its own while there is still a snapshot made by Veeam?

It automatically restarted the job which succeeded after 13 hours with a processing rate: 5 MB/s
Because the job is scheduled for repeating every hour the next job started immediately but hung at 77% without any error and I'm unable to stop the job.
According to the TaskManager I have now four processes called "VeeamAgent.exe *32" running.

Ed

[Gostev]

The error above is a common VMware Tools quiescence issue. You should enable Veeam VSS in backup job settings (this will also disable VMware Tools quiescence automatically).

For Exchange server issue it is best to work with our support directly.

[E.Kok]

So this should be done by unchecking 'General->Backup options--Enable VMware tools quiescence'?

[nickf]

Going back to one of your earlier posts... we only have 1 physical Windows server in our ESX deployments (we host multiple separate infrastructures for clients, including banks), that server runs NetBackup and Veeam. DC's and vCenter are fine (and IMO better) running virtualised. So in your case I'd just invest in a decent physical server (we use Dell PowerEdge R610 or 710's currently) and virtualise all but Veeam and your backup host and run those on the same box.

[Vitaliy S.]

So this should be done by unchecking 'General->Backup options--Enable VMware tools quiescence'?

Yes, and don't forget to enable Veeam VSS integration for this job.

[Gostev]

Simply enable Veeam VSS, and it will take care of disabling VMware Tools quiescence automatically.