proxy cpu usage. no dedup/compression

[frinnst]

Hi!

We are experiencing very slow performance when backing up a 1.4TB 3rd-party backup repository with veeam. (Full backup only, once per week)

The VMware datastore is hosted on an old netapp 2040 nfs nas. There are no other VMs on this controller or diskshelf.
The proxy's transport is forced to NBD - if we use hotadd the VM is locked for about ~8 minutes when the VM snapshot gets removed.

The VM has 4 virtual disks
Disk 1 25GB
Disk 2 30GB
Disk 3 100GB
Disk 4 1,3TB

When running the backup, performance is what we'd expect for the first ~5%, when all disks are processed in parallel. When the smaller disks are completed the processing speed on the remaining disk stays at ~20MB/s for the duration of the backup.

We don't really see any bottlenecks except for the proxy. No matter how many cores we throw at it it still uses ~100% cpu - even with compression/dedup disabled. Is this really expected?

The bottlenecks according to veeam are these:
Source: 54%
Proxy: 44%
Network 63%
Target 35%

Needless to say, we have played with most settings to no avail. We have yet to open a ticket - thought I'd try here first.

[foggy]

Fredrik, can you check what processes on the proxy server use CPU up to 100%? What is the max concurrent tasks value for the proxy? What are the bottleneck stats with compression enabled? Since your primary bottleneck is network, you could check your network utilization. Decreasing the traffic amount by compressing it (considering enough resources on the proxy) and adding a second proxy should also help. Is it a full job run, btw?

[frinnst]

The processes that use cpu are veeamagent64.exe and veeamagent.exe *32 as well as the System process (interrupts?). Otherwise the proxy is pretty idle (47 processes while backup is running). Its a clean Win2008r2 install with only veeam software on it. VM version vmx-09, VMXNET3 nic.

As i stated previously, the data is already very compressed, deduplicated and encrypted. We have also tested with the default veeam settings with regards to compression, dedup etc without any noticeable difference.

But since we have disabled all processing (dedup, compression etc) that we can find, why is the proxy still using so much cpu?

[Gostev]

We are experiencing very slow performance when backing up a 1.4TB 3rd-party backup repository with veeam. (Full backup only, once per week)

Can you define "very slow", how much time full backup of 1.4 TB takes?

No matter how many cores we throw at it it still uses ~100% cpu - even with compression/dedup disabled. Is this really expected?

What product version and compression level are you using for this job?

When the smaller disks are completed the processing speed on the remaining disk stays at ~20MB/s for the duration of the backup.

This is normal and as much as you can get with network processing mode due to management interface throttling on ESXi. I you saying your proxy CPU is still at 100% when there is just this single remaining disk is being processed?

[frinnst]

Can you define "very slow", how much time full backup of 1.4 TB takes?

I aborted it after 11hrs

What product version and compression level are you using for this job?

v.7 and "None"

This is normal and as much as you can get with network processing mode due to management interface throttling on ESXi. I you saying your proxy CPU is still at 100% when there is just this single remaining disk is being processed?

Yes.

[foggy]

Since you have two Veeam agents running (veeamagent64.exe and veeamagent.exe *32), do you probably have proxy and repository roles both installed on a single server? Could you try to use another server as a proxy or backup repository and track bottleneck stats and proxy CPU load? What kind of CPU is installed in your host?

[frinnst]

Im not sure I understand what you mean. The proxy is a standalone VM with nothing else running on it. The server/repository is a physical machine.

The host of the proxy is an old BL460c g1 with 2x Xeon E5320.

[foggy]

The server/repository is a physical machine.

How it is added to Veeam B&R console? As a CIFS share?

veeamagent64.exe is a v7 repository agent process that receives the data from the proxy and writes it to the target folder. Having it on a proxy means that this server also owns the repository role.

[tsightler]

You might want to try enabling RSS on the server, this should help improve throughput, at least if you have multiple vCPUs. Actually, that's another good question, how many vCPUs do you have on your Veeam server?

RSS should be enabled by default at the OS level on Windows 2008R2, but for it to work you must also manually enable it on the VMXNET3 NIC in the adapter properties.

http://boerlowie.wordpress.com/2011/03/ ... -machines/

I know you mentioned the problem with Hotadd mode, but I'm curious, was the performance better with that mode or the same?