High CPU utilization

[Davd]

Hi,

I have a question regarding high CPU utilization during backup jobs. We have been using Veeam since v.4.1 and each time Veeam has released a new version, we upgraded to that version. Currently we are running v.5.0.2.230 + vSphere 5 patch (with the latest agent that addressed high CPU). Our Veeam server is virtual with 4 vCPU and 8 GB RAM running Windows Server 2008 R2. The ESX host (running vSphere 4.1) on which Veeam server is located has 2x Intel Xeon X5550 and we have 2 Dell MD1200 (SCSI DAS) connected to the ESX host. We use those 2 MD1200's as backup storage only. We have spanned all MD1200 drives (due to vmdk limitations - max 2 TB) in Veeam server so it provides us an R:\ drive with 29TB. All production VM's that we backup reside on Equallogic iSCSI SAN (PS6000 and PS4100).

Our issue is that we have had high CPU utilization when running backup jobs for quite a while. I am not sure at what exact version this started but I think it was once we moved from v.4.12 to v.5.0.0.179. Prior to this, one backup job would take somewhere about 50% of CPU meaning that we could run 2 simultaneous jobs which would make CPU to utilize almost 100%. But after one of the Veeam upgrades, one jobs takes almost all CPU (runs between 85-99%) so running 2 simultaneous jobs is no longer an option since both jobs would take longer time to perform due to the heavy load on CPU.

I have been following posts on this forum regarding other peoples CPU experiences but nobody seems to have this issue. A 4vCPU server according to other posts takes app. 50% CPU. And therefor I decide to finally post my CPU issue to see if there is something that I have missed.

Below is a print screen when I run 1 backup job and as you can see, VeeamAgent.exe is utilizing almost all CPU. All our jobs are configured to run with Optimal compression and Incremental job with "Enable synthetic fulls" + "Transform previous full backup chains into rollbacks". Storage optimize is set to Local Target, we are using CBT and we have enabled Automatic backup integrity checks. The high CPU utilization occurs only during the actual backup (creation of the .VIB file). Transforming of chain does not put a load on CPU.

We have this CPU behaviour on all our backup jobs so it’s not specific to this job.

[Gostev]

It is perfectly normal for a single job to fully saturate CPU if the source storage performance allows our engine to retrieve the data at a very high speed, which is the case here. If you look around the forum, I always state this as one of the reasons why we do not do multiple VM processing within the same job - because it is usually quite pointless (only makes each VM to run much longer off snapshot, causing much longer commits).

Sure, you might not had seen it with earlier versions of the product, because we are improving interaction with storage with every release based on what we learn from real world environments. When previously a single job was not necessarily able to realize full potential of fast storage, and you needed multiple jobs to fully saturate both storage and CPU, now you can do the same with the single job.

Looking at 178 MB/s disk I/O, you really have nothing to complain about CPU usage here

[Davd]

Hi Gostev,

Your statment makes sense and yes, I'm happy with disk I/O . I just wanted to make sure that I wasn't doing something wrong...

[Gostev]

You are doing everything right, clearly your backup and storage infrastructures are flawlessly tuned. It's those people with 4 vCPU VM only loaded 50% by a single job who should be concerned and looking for the bottleneck

Some early v5 baseline numbers (pre-patch) are as follows: full backup running at about 55 MB/s caps a single Intel Q6600 2.4 GHz (4 cores). This is for default job settings (Optimal compression, dedupe enabled). If your backup server CPU is not capped, this only means that our engine cannot read the source storage or write to target storage fast enough. v6 will actually provide real-time bottleneck statistics, so that should help to identify the weak link immediately.

Now, vSphere 5 patch is really not a regular hotfix (which is why it took so long). It brings quite major changes such as new VDDK 5.0 and even some v6 code over. There is not enough statistics for me to tell you anything specific, but there should definitely be an improvement with CPU usage with the patch.

I do have plenty of statistics for v6 though, and I can say that v6 engine uses up to 2x less CPU under the same circumstances comparing to v5. In most recent test I did in my lab, 2nd gen (Sandy Bridge) Core i5 @ 3.4 GHz capped full backup performance at 50 MB/s after I limited Veeam data mover agent to a single core via CPU affinity settings in the Task Manager. Granted that my CPU is the newest, and I have it heavily overlocked, this makes its 1 core capabilities roughly equal to 2 cores of much older Q6600. And remember that with v5, 55MB/s required all 4 cores of Q6600. Not a very scientific test, but gives you good idea.

[tsightler]

Just to add some numbers "from the field", although this varies pretty significantly based on the host hardware, my general rule of thumb, assuming fairly modern hardware, is that a 4 vCPU virtual appliance will top out around 150-200MB/sec of throughput before hitting CPU saturation. Your right in the middle of that sweet spot and, if your storage infrastructure can provide that level of throughput from a single job (which obviously yours can), then there's no surprise that your seeing near 100% CPU usage.