Comprehensive data protection for all workloads
antivir
Enthusiast
Posts: 63
Liked: never
Joined: Nov 04, 2009 2:39 pm
Full Name: Andrew
Contact:

Performance issue (CPU load is 100%)

Post by antivir »

VBR 6.1.0.181 (64 bit) on Windows Server 2008 R2 (VM:8 CPUs, 4GB on BL460c G1)
Vsphere 5U1.
Destination Storage is NFS (1 Gbit/s).
Backup mode, default settings.

Performance:
1.VM CPU load is 100% (?????), in vSphere Client CPU Usage of host is full (8*2,5 GHz).
2.Network on NFS: 500-600 Mbits/sec (iptraf)

Is VBR so gluttonous now?
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Performance issue (CPU load is 100%)

Post by foggy »

Andrew, I guess your jobs are processed by the default proxy installed as a part of the Veeam B&R installation. How many tasks are processed by this proxy at the time when its CPU is maxed out? Note, that typically, 2 cores are saturated by a single backup job and there is a setting for the max number of concurrent jobs per proxy, which you can tune according to your hardware.

Also, what is the compression level for these jobs and what does the bottleneck stats show for them?
antivir
Enthusiast
Posts: 63
Liked: never
Joined: Nov 04, 2009 2:39 pm
Full Name: Andrew
Contact:

Re: Performance issue (CPU load is 100%)

Post by antivir »

foggy wrote:Andrew, I guess your jobs are processed by the default proxy installed as a part of the Veeam B&R installation. How many tasks are processed by this proxy at the time when its CPU is maxed out? Note, that typically, 2 cores are saturated by a single backup job and there is a setting for the max number of concurrent jobs per proxy, which you can tune according to your hardware.

Also, what is the compression level for these jobs and what does the bottleneck stats show for them?
I have only one working job. Compression Level is Optimal, Storage optimizations is for local target.

My settings (VM is on EVA 4400):

Backup Repository:
Type: Linux Server
Server: NFS-path
Repository: Limit Maximum concurent jobs count to 8
vPower NFS: Enable vPower NFS server (this server)

Backup Proxy:
Server: this server, Max concurent tasks is 2

Backup Job:
Virtual Machines: 20 Windows VMs
Storage: NFS-repository, Advanced: Incremental (Enable synthetic fulls), Inline data deduplication is enabled, Compression Level is Optimal, Storage optimizations is for local target, CBT is enabled, Integrity checks is enabled, Exclude swap file.
Guest Processing: Enable application-aware image processing is checked.

How to find out what is bottle neck?

P.S. I see now:
Busy: Source 99% > Proxy 41% > Network 6% > Target 7%:
Primary bottleneck:Source.

Are 8 CPU bottleneck?
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by dellock6 »

Does not seems so, your stats show primary bottleneck is the EVA storage. If you have only one job, you have only one VM at a time beeing backed up, so it's ahrd to say a 8 cpu VM can be a bottleneck. Maybe you can check in the task manager which processes are eating up your cpu?

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Performance issue (CPU load is 100%)

Post by J1mbo »

Is that the host has 8 CPU (cores), and the Veeam VM have 8 vCPU assigned?
antivir
Enthusiast
Posts: 63
Liked: never
Joined: Nov 04, 2009 2:39 pm
Full Name: Andrew
Contact:

Re: Performance issue (CPU load is 100%)

Post by antivir »

dellock6 wrote:Does not seems so, your stats show primary bottleneck is the EVA storage. If you have only one job, you have only one VM at a time beeing backed up, so it's ahrd to say a 8 cpu VM can be a bottleneck. Maybe you can check in the task manager which processes are eating up your cpu?

Luca.
Guess EVA is not bottleneck. There are VeeamAgent.exe eats more then 90% CPU.
antivir
Enthusiast
Posts: 63
Liked: never
Joined: Nov 04, 2009 2:39 pm
Full Name: Andrew
Contact:

Re: Performance issue (CPU load is 100%)

Post by antivir »

J1mbo wrote:Is that the host has 8 CPU (cores), and the Veeam VM have 8 vCPU assigned?
Yes, Host BL 460c G1 has 8 CPU cores (2 sockets Xeon E5420 2.5 Ghz ans 4 cores per socket) and Veeam VM has 8 CPU sockets assigned (one core per socket in VM version 8 settings).
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Performance issue (CPU load is 100%)

Post by J1mbo »

I would reduce that to 4 vCPU, and see what happens then.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Performance issue (CPU load is 100%)

Post by Vitaliy S. »

BTW - if you reduce a compression level, what will be the CPU utilization for your backup proxy?
antivir
Enthusiast
Posts: 63
Liked: never
Joined: Nov 04, 2009 2:39 pm
Full Name: Andrew
Contact:

Re: Performance issue (CPU load is 100%)

Post by antivir »

Vitaliy S. wrote:BTW - if you reduce a compression level, what will be the CPU utilization for your backup proxy?
If I set low compression, there are no problem with CPU load, but sizes of VM (Used Space is 57 GB):
1. Low Compression - 52 GB
2. Optimal Compression (high CPU load) - 11 GB.
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Performance issue (CPU load is 100%)

Post by J1mbo »

Has the vCPU count been reduced?
BigGeorge
Veeam ProPartner
Posts: 27
Liked: 4 times
Joined: Jan 31, 2012 2:00 pm
Full Name: Giorgio Colucci
Location: Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by BigGeorge »

Any update?
There are a way to set a maximum CPU utilization for the veeam proxy service?

Thank you
Giorgio
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Performance issue (CPU load is 100%)

Post by Vitaliy S. »

Hi Giorgio,

Having your proxy server fully utilized while running multiple backup/replication jobs at the same time is absolutely normal behavior. How many jobs do you concurrently run? What are the specs of your proxy server?

Thanks!
BigGeorge
Veeam ProPartner
Posts: 27
Liked: 4 times
Joined: Jan 31, 2012 2:00 pm
Full Name: Giorgio Colucci
Location: Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by BigGeorge »

1 job.
The source proxy are a virtual machine win 7 pro x86 with 4 processor and 3Gb RAM, the proxy mode is "Virtual appliance" with cbt enabled and compression an dedup on the fly.
The stats see that the bottleneck is the source proxy at 99%.

Make sense to upgrade the processor number to 8 (or more if SO support them?

any ideas?
Giorgio
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Performance issue (CPU load is 100%)

Post by Vitaliy S. »

Having "source" reported as a bottleneck is not bad at all, assuming that you're getting decent speed for your jobs during full and incremental runs.

If you decide to run more jobs through this proxy while having in-guest CPU resources fully utilized, then you will need to add more vCPUs or configure an additional proxy sever that will meet our recommended system requirements. These requirements can be found in the Release Notes document included with your download. Hope this helps!
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Performance issue (CPU load is 100%)

Post by Gostev »

BigGeorge wrote:The stats see that the bottleneck is the source proxy at 99%.
No such bottleneck as "source proxy". Please clarify if bottleneck reported is "source" or "proxy", because these are two different possible bottleneck counter readings. I *think* you are talking about "proxy" bottleneck, but Vitaly's response above is for "source" bottleneck.
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by dellock6 »

BigGeorge wrote:The source proxy are a virtual machine win 7 pro x86 with 4 processor and 3Gb RAM, the proxy mode is "Virtual appliance" with cbt enabled and compression an dedup on the fly.
Compression and dedup are at default level, or you increased them? Higher level produce smaller backup files but put extra load on CPUs...

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
BigGeorge
Veeam ProPartner
Posts: 27
Liked: 4 times
Joined: Jan 31, 2012 2:00 pm
Full Name: Giorgio Colucci
Location: Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by BigGeorge »

I'm sorry: the bottleneck is proxy 99%.

The dedup and compression are at default level and are "optimized for LAN target" (the repository is a third phisical windows server configured as "windows repository type".
Giorgio
BigGeorge
Veeam ProPartner
Posts: 27
Liked: 4 times
Joined: Jan 31, 2012 2:00 pm
Full Name: Giorgio Colucci
Location: Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by BigGeorge »

dellock6 wrote:Compression and dedup are at default level, or you increased them? Higher level produce smaller backup files but put extra load on CPUs...
I know that, but in this case I need maximum compression for this job. But I don't want that the proxy vm uses 100% CPU for veeam process because that freeze other services witch lives on that vm!

Thank you

Giorgio
Giorgio
tsightler
VP, Product Management
Posts: 6035
Liked: 2860 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Performance issue (CPU load is 100%)

Post by tsightler »

With 6.1 the proxy process is given a lower priority than other system processes so it should not really conflict with other services. That being said, it sounds like you want maximum compression, but for the system to not use 100% CPU, which are two goals that are somewhat at odds with each other. One option would be to throttle the traffic using the built in throttling (either on the repository or on the network between proxies/repositories). Artificially limiting the available bandwidth would lower the load on the proxy. Other than that, you might consider using a dedicated proxy.
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Performance issue (CPU load is 100%)

Post by J1mbo »

Since proxy CPU load seems to come up quite a bit, maybe Veeam might consider implemented a grid-type strategy for Veeam proxy agents, so that any available desktop PC could chip-in as required, to produce the highest possible compression rates by involving many processors. (I've seen this done with PDF report generation with some very impressive results.)
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by dellock6 »

Uhm, this would imply adding another role to the multi-role design, since proxies are also responsible for extracting data from the storage, and I suspect it's the problem behind the limit that a VM can only be processed by a single proxy, do to some locking on the files.
Also, since the IO part is always done by the proxy, moving raw data to a "compressor" role (we can call it so?) would imply basically loosing the benefits of low bandwidth consumption due to compression done in the proxy and then dehidrated data moved to the repository. Compressor role is already inside the proxy right now...

A possible idea would instead be to split the load on several proxies on a per-VM base more than a per-Job base like today, but again, compression and deduplication are done now per-Job, so the compressor role (proxy right now) would need a way to share compression/deduplication patterns with the other compressor helping it. Maybe Veeam guys already evaluated and discarded this scenario :P

Did I understood right your idea? :)

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Performance issue (CPU load is 100%)

Post by J1mbo »

Pretty much!

So the proxy would sends say 100 blocks to PCs for super-compression, and after some time receives the 100 blocks back. And hopefully both dT and the resultant data size are significantly less than would have been possible within the proxy.
BigGeorge
Veeam ProPartner
Posts: 27
Liked: 4 times
Joined: Jan 31, 2012 2:00 pm
Full Name: Giorgio Colucci
Location: Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by BigGeorge »

Another idea must be move the compression and deduplication role at the repository level.

Option 1. Dedup and compression on the proxy (now we're here). But give me the ability to set the maximum CPU usage for the proxy process (in this mode I can enforce maximum compression/dedup at the proxy that are near the source: best for WAN)
Option 2. Create an "intermediate" proxy role (flow: source->proxy_near_source(actual)->proxy_in_the_middle(new!)->repository(or destinatio_proxy for replication)->disk) to assign them the dedup/compression funcions
Option 3. Delegate to the destination the dedup/compression functions (that allow to use the CPU in the repository-server).
Option 4. At the "intermediate_proxy" (see option2) give the ability to merge stream from differents proxies (example: to allow best dedup/compression for WAN replica of more jobs running at the same time).

... good things ... but today in what manner I can optimize my job? :)
Giorgio
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Performance issue (CPU load is 100%)

Post by foggy »

BigGeorge wrote:... good things ... but today in what manner I can optimize my job? :)
Lower the compression level. ;) Or go with the options Tom has suggested above.
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by dellock6 » 1 person likes this post

As I said, moving compression and deduplication to the repository will then create a new bottleneck on network, since data as to be moved in raw format. Since VMs are becoming bigger and bigger, I think the compression and deduplication needs to happen as near as possible to the production storage.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
BigGeorge
Veeam ProPartner
Posts: 27
Liked: 4 times
Joined: Jan 31, 2012 2:00 pm
Full Name: Giorgio Colucci
Location: Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by BigGeorge »

dellock6 wrote:As I said, moving compression and deduplication to the repository will then create a new bottleneck on network, since data as to be moved in raw format. Since VMs are becoming bigger and bigger, I think the compression and deduplication needs to happen as near as possible to the production storage.
yes I understand, but is not always true :)
Giorgio
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Performance issue (CPU load is 100%)

Post by dellock6 »

So, please describe me a use case where is better to ship raw data across the network instead of compressing/deduplicating them before going into the network.
Maybe it's me, but I'm missing a good scenario for it... 100% cpu in a proxy to me is a good value, they are designed to do that, and you can add more to balance the load.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Performance issue (CPU load is 100%)

Post by J1mbo »

It really all depends on whether it's cheaper (in terms of time) to compress then send, send-compress-send or just send. Which obviously depends on compression throughput and link speeds between source and destination.

My observation was that for WAN replication, using a grid to compute could improve overall performance by using much more aggressive compression, since the limiting factor is the WAN in terms of data transmission, rather than the LAN, and the proxy CPUs in terms of compression. Of course there's a balance to be had :)
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Performance issue (CPU load is 100%)

Post by Gostev »

BigGeorge wrote:... good things ... but today in what manner I can optimize my job? :)
Change compression from Best to Optimal. Comparing to Optimal, Best compression gives only 2-3% data reduction at best - at the cost of MASSIVE hit on proxy CPU. It does not really worth using, and at some point (after doing detailed compression level testing) we were close to removing this compression level completely, because of very little difference w/optimal. We kept it only because this would cause too much questions "why we are reducing functionality". We are considering replacing the compression algorithm behind this compression level for one that provides better compression for such CPU load - but for now, I would strongly recommend to simply not use Best compression unless you have monster proxy servers, and stick to Optimal instead. You really are not winning almost anything by using Best compression.
Post Reply

Who is online

Users browsing this forum: No registered users and 54 guests