Performance issue (CPU load is 100%)

[antivir]

VBR 6.1.0.181 (64 bit) on Windows Server 2008 R2 (VM:8 CPUs, 4GB on BL460c G1)
Vsphere 5U1.
Destination Storage is NFS (1 Gbit/s).
Backup mode, default settings.

Performance:
1.VM CPU load is 100% (?????), in vSphere Client CPU Usage of host is full (8*2,5 GHz).
2.Network on NFS: 500-600 Mbits/sec (iptraf)

Is VBR so gluttonous now?

[foggy]

Andrew, I guess your jobs are processed by the default proxy installed as a part of the Veeam B&R installation. How many tasks are processed by this proxy at the time when its CPU is maxed out? Note, that typically, 2 cores are saturated by a single backup job and there is a setting for the max number of concurrent jobs per proxy, which you can tune according to your hardware.

Also, what is the compression level for these jobs and what does the bottleneck stats show for them?

[antivir]

Andrew, I guess your jobs are processed by the default proxy installed as a part of the Veeam B&R installation. How many tasks are processed by this proxy at the time when its CPU is maxed out? Note, that typically, 2 cores are saturated by a single backup job and there is a setting for the max number of concurrent jobs per proxy, which you can tune according to your hardware.

Also, what is the compression level for these jobs and what does the bottleneck stats show for them?

I have only one working job. Compression Level is Optimal, Storage optimizations is for local target.

My settings (VM is on EVA 4400):

Backup Repository:
Type: Linux Server
Server: NFS-path
Repository: Limit Maximum concurent jobs count to 8
vPower NFS: Enable vPower NFS server (this server)

Backup Proxy:
Server: this server, Max concurent tasks is 2

Backup Job:
Virtual Machines: 20 Windows VMs
Storage: NFS-repository, Advanced: Incremental (Enable synthetic fulls), Inline data deduplication is enabled, Compression Level is Optimal, Storage optimizations is for local target, CBT is enabled, Integrity checks is enabled, Exclude swap file.
Guest Processing: Enable application-aware image processing is checked.

How to find out what is bottle neck?

P.S. I see now:
Busy: Source 99% > Proxy 41% > Network 6% > Target 7%:
Primary bottleneck:Source.

Are 8 CPU bottleneck?

[dellock6]

Does not seems so, your stats show primary bottleneck is the EVA storage. If you have only one job, you have only one VM at a time beeing backed up, so it's ahrd to say a 8 cpu VM can be a bottleneck. Maybe you can check in the task manager which processes are eating up your cpu?

Luca.

[J1mbo]

Is that the host has 8 CPU (cores), and the Veeam VM have 8 vCPU assigned?

[antivir]

Does not seems so, your stats show primary bottleneck is the EVA storage. If you have only one job, you have only one VM at a time beeing backed up, so it's ahrd to say a 8 cpu VM can be a bottleneck. Maybe you can check in the task manager which processes are eating up your cpu?

Luca.

Guess EVA is not bottleneck. There are VeeamAgent.exe eats more then 90% CPU.

[antivir]

Is that the host has 8 CPU (cores), and the Veeam VM have 8 vCPU assigned?

Yes, Host BL 460c G1 has 8 CPU cores (2 sockets Xeon E5420 2.5 Ghz ans 4 cores per socket) and Veeam VM has 8 CPU sockets assigned (one core per socket in VM version 8 settings).

[J1mbo]

I would reduce that to 4 vCPU, and see what happens then.

[Vitaliy S.]

BTW - if you reduce a compression level, what will be the CPU utilization for your backup proxy?

[antivir]

BTW - if you reduce a compression level, what will be the CPU utilization for your backup proxy?

If I set low compression, there are no problem with CPU load, but sizes of VM (Used Space is 57 GB):
1. Low Compression - 52 GB
2. Optimal Compression (high CPU load) - 11 GB.

[J1mbo]

Has the vCPU count been reduced?

[BigGeorge]

Any update?
There are a way to set a maximum CPU utilization for the veeam proxy service?

Thank you

[Vitaliy S.]

Hi Giorgio,

Having your proxy server fully utilized while running multiple backup/replication jobs at the same time is absolutely normal behavior. How many jobs do you concurrently run? What are the specs of your proxy server?

Thanks!

[BigGeorge]

1 job.
The source proxy are a virtual machine win 7 pro x86 with 4 processor and 3Gb RAM, the proxy mode is "Virtual appliance" with cbt enabled and compression an dedup on the fly.
The stats see that the bottleneck is the source proxy at 99%.

Make sense to upgrade the processor number to 8 (or more if SO support them?

any ideas?

[Vitaliy S.]

Having "source" reported as a bottleneck is not bad at all, assuming that you're getting decent speed for your jobs during full and incremental runs.

If you decide to run more jobs through this proxy while having in-guest CPU resources fully utilized, then you will need to add more vCPUs or configure an additional proxy sever that will meet our recommended system requirements. These requirements can be found in the Release Notes document included with your download. Hope this helps!

[Gostev]

The stats see that the bottleneck is the source proxy at 99%.

No such bottleneck as "source proxy". Please clarify if bottleneck reported is "source" or "proxy", because these are two different possible bottleneck counter readings. I *think* you are talking about "proxy" bottleneck, but Vitaly's response above is for "source" bottleneck.

[dellock6]

The source proxy are a virtual machine win 7 pro x86 with 4 processor and 3Gb RAM, the proxy mode is "Virtual appliance" with cbt enabled and compression an dedup on the fly.

Compression and dedup are at default level, or you increased them? Higher level produce smaller backup files but put extra load on CPUs...

Luca.

[BigGeorge]

I'm sorry: the bottleneck is proxy 99%.

The dedup and compression are at default level and are "optimized for LAN target" (the repository is a third phisical windows server configured as "windows repository type".

[BigGeorge]

Compression and dedup are at default level, or you increased them? Higher level produce smaller backup files but put extra load on CPUs...

I know that, but in this case I need maximum compression for this job. But I don't want that the proxy vm uses 100% CPU for veeam process because that freeze other services witch lives on that vm!

Thank you

Giorgio

[tsightler]

With 6.1 the proxy process is given a lower priority than other system processes so it should not really conflict with other services. That being said, it sounds like you want maximum compression, but for the system to not use 100% CPU, which are two goals that are somewhat at odds with each other. One option would be to throttle the traffic using the built in throttling (either on the repository or on the network between proxies/repositories). Artificially limiting the available bandwidth would lower the load on the proxy. Other than that, you might consider using a dedicated proxy.

[J1mbo]

Since proxy CPU load seems to come up quite a bit, maybe Veeam might consider implemented a grid-type strategy for Veeam proxy agents, so that any available desktop PC could chip-in as required, to produce the highest possible compression rates by involving many processors. (I've seen this done with PDF report generation with some very impressive results.)

[dellock6]

Uhm, this would imply adding another role to the multi-role design, since proxies are also responsible for extracting data from the storage, and I suspect it's the problem behind the limit that a VM can only be processed by a single proxy, do to some locking on the files.
Also, since the IO part is always done by the proxy, moving raw data to a "compressor" role (we can call it so?) would imply basically loosing the benefits of low bandwidth consumption due to compression done in the proxy and then dehidrated data moved to the repository. Compressor role is already inside the proxy right now...

A possible idea would instead be to split the load on several proxies on a per-VM base more than a per-Job base like today, but again, compression and deduplication are done now per-Job, so the compressor role (proxy right now) would need a way to share compression/deduplication patterns with the other compressor helping it. Maybe Veeam guys already evaluated and discarded this scenario

Did I understood right your idea?

Luca.

[J1mbo]

Pretty much!

So the proxy would sends say 100 blocks to PCs for super-compression, and after some time receives the 100 blocks back. And hopefully both dT and the resultant data size are significantly less than would have been possible within the proxy.

[BigGeorge]

Another idea must be move the compression and deduplication role at the repository level.

Option 1. Dedup and compression on the proxy (now we're here). But give me the ability to set the maximum CPU usage for the proxy process (in this mode I can enforce maximum compression/dedup at the proxy that are near the source: best for WAN)
Option 2. Create an "intermediate" proxy role (flow: source->proxy_near_source(actual)->proxy_in_the_middle(new!)->repository(or destinatio_proxy for replication)->disk) to assign them the dedup/compression funcions
Option 3. Delegate to the destination the dedup/compression functions (that allow to use the CPU in the repository-server).
Option 4. At the "intermediate_proxy" (see option2) give the ability to merge stream from differents proxies (example: to allow best dedup/compression for WAN replica of more jobs running at the same time).

... good things ... but today in what manner I can optimize my job?

[foggy]

... good things ... but today in what manner I can optimize my job?

Lower the compression level. Or go with the options Tom has suggested above.

[dellock6]

As I said, moving compression and deduplication to the repository will then create a new bottleneck on network, since data as to be moved in raw format. Since VMs are becoming bigger and bigger, I think the compression and deduplication needs to happen as near as possible to the production storage.

Luca.

[BigGeorge]

As I said, moving compression and deduplication to the repository will then create a new bottleneck on network, since data as to be moved in raw format. Since VMs are becoming bigger and bigger, I think the compression and deduplication needs to happen as near as possible to the production storage.

yes I understand, but is not always true

[dellock6]

So, please describe me a use case where is better to ship raw data across the network instead of compressing/deduplicating them before going into the network.
Maybe it's me, but I'm missing a good scenario for it... 100% cpu in a proxy to me is a good value, they are designed to do that, and you can add more to balance the load.

Luca.

[J1mbo]

It really all depends on whether it's cheaper (in terms of time) to compress then send, send-compress-send or just send. Which obviously depends on compression throughput and link speeds between source and destination.

My observation was that for WAN replication, using a grid to compute could improve overall performance by using much more aggressive compression, since the limiting factor is the WAN in terms of data transmission, rather than the LAN, and the proxy CPUs in terms of compression. Of course there's a balance to be had

[Gostev]

... good things ... but today in what manner I can optimize my job?

Change compression from Best to Optimal. Comparing to Optimal, Best compression gives only 2-3% data reduction at best - at the cost of MASSIVE hit on proxy CPU. It does not really worth using, and at some point (after doing detailed compression level testing) we were close to removing this compression level completely, because of very little difference w/optimal. We kept it only because this would cause too much questions "why we are reducing functionality". We are considering replacing the compression algorithm behind this compression level for one that provides better compression for such CPU load - but for now, I would strongly recommend to simply not use Best compression unless you have monster proxy servers, and stick to Optimal instead. You really are not winning almost anything by using Best compression.