-
- Enthusiast
- Posts: 63
- Liked: never
- Joined: Nov 04, 2009 2:39 pm
- Full Name: Andrew
- Contact:
Performance issue (CPU load is 100%)
VBR 6.1.0.181 (64 bit) on Windows Server 2008 R2 (VM:8 CPUs, 4GB on BL460c G1)
Vsphere 5U1.
Destination Storage is NFS (1 Gbit/s).
Backup mode, default settings.
Performance:
1.VM CPU load is 100% (?????), in vSphere Client CPU Usage of host is full (8*2,5 GHz).
2.Network on NFS: 500-600 Mbits/sec (iptraf)
Is VBR so gluttonous now?
Vsphere 5U1.
Destination Storage is NFS (1 Gbit/s).
Backup mode, default settings.
Performance:
1.VM CPU load is 100% (?????), in vSphere Client CPU Usage of host is full (8*2,5 GHz).
2.Network on NFS: 500-600 Mbits/sec (iptraf)
Is VBR so gluttonous now?
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Performance issue (CPU load is 100%)
Andrew, I guess your jobs are processed by the default proxy installed as a part of the Veeam B&R installation. How many tasks are processed by this proxy at the time when its CPU is maxed out? Note, that typically, 2 cores are saturated by a single backup job and there is a setting for the max number of concurrent jobs per proxy, which you can tune according to your hardware.
Also, what is the compression level for these jobs and what does the bottleneck stats show for them?
Also, what is the compression level for these jobs and what does the bottleneck stats show for them?
-
- Enthusiast
- Posts: 63
- Liked: never
- Joined: Nov 04, 2009 2:39 pm
- Full Name: Andrew
- Contact:
Re: Performance issue (CPU load is 100%)
I have only one working job. Compression Level is Optimal, Storage optimizations is for local target.foggy wrote:Andrew, I guess your jobs are processed by the default proxy installed as a part of the Veeam B&R installation. How many tasks are processed by this proxy at the time when its CPU is maxed out? Note, that typically, 2 cores are saturated by a single backup job and there is a setting for the max number of concurrent jobs per proxy, which you can tune according to your hardware.
Also, what is the compression level for these jobs and what does the bottleneck stats show for them?
My settings (VM is on EVA 4400):
Backup Repository:
Type: Linux Server
Server: NFS-path
Repository: Limit Maximum concurent jobs count to 8
vPower NFS: Enable vPower NFS server (this server)
Backup Proxy:
Server: this server, Max concurent tasks is 2
Backup Job:
Virtual Machines: 20 Windows VMs
Storage: NFS-repository, Advanced: Incremental (Enable synthetic fulls), Inline data deduplication is enabled, Compression Level is Optimal, Storage optimizations is for local target, CBT is enabled, Integrity checks is enabled, Exclude swap file.
Guest Processing: Enable application-aware image processing is checked.
How to find out what is bottle neck?
P.S. I see now:
Busy: Source 99% > Proxy 41% > Network 6% > Target 7%:
Primary bottleneck:Source.
Are 8 CPU bottleneck?
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Performance issue (CPU load is 100%)
Does not seems so, your stats show primary bottleneck is the EVA storage. If you have only one job, you have only one VM at a time beeing backed up, so it's ahrd to say a 8 cpu VM can be a bottleneck. Maybe you can check in the task manager which processes are eating up your cpu?
Luca.
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Veteran
- Posts: 261
- Liked: 29 times
- Joined: May 03, 2011 12:51 pm
- Full Name: James Pearce
- Contact:
Re: Performance issue (CPU load is 100%)
Is that the host has 8 CPU (cores), and the Veeam VM have 8 vCPU assigned?
-
- Enthusiast
- Posts: 63
- Liked: never
- Joined: Nov 04, 2009 2:39 pm
- Full Name: Andrew
- Contact:
Re: Performance issue (CPU load is 100%)
Guess EVA is not bottleneck. There are VeeamAgent.exe eats more then 90% CPU.dellock6 wrote:Does not seems so, your stats show primary bottleneck is the EVA storage. If you have only one job, you have only one VM at a time beeing backed up, so it's ahrd to say a 8 cpu VM can be a bottleneck. Maybe you can check in the task manager which processes are eating up your cpu?
Luca.
-
- Enthusiast
- Posts: 63
- Liked: never
- Joined: Nov 04, 2009 2:39 pm
- Full Name: Andrew
- Contact:
Re: Performance issue (CPU load is 100%)
Yes, Host BL 460c G1 has 8 CPU cores (2 sockets Xeon E5420 2.5 Ghz ans 4 cores per socket) and Veeam VM has 8 CPU sockets assigned (one core per socket in VM version 8 settings).J1mbo wrote:Is that the host has 8 CPU (cores), and the Veeam VM have 8 vCPU assigned?
-
- Veteran
- Posts: 261
- Liked: 29 times
- Joined: May 03, 2011 12:51 pm
- Full Name: James Pearce
- Contact:
Re: Performance issue (CPU load is 100%)
I would reduce that to 4 vCPU, and see what happens then.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Performance issue (CPU load is 100%)
BTW - if you reduce a compression level, what will be the CPU utilization for your backup proxy?
-
- Enthusiast
- Posts: 63
- Liked: never
- Joined: Nov 04, 2009 2:39 pm
- Full Name: Andrew
- Contact:
Re: Performance issue (CPU load is 100%)
If I set low compression, there are no problem with CPU load, but sizes of VM (Used Space is 57 GB):Vitaliy S. wrote:BTW - if you reduce a compression level, what will be the CPU utilization for your backup proxy?
1. Low Compression - 52 GB
2. Optimal Compression (high CPU load) - 11 GB.
-
- Veteran
- Posts: 261
- Liked: 29 times
- Joined: May 03, 2011 12:51 pm
- Full Name: James Pearce
- Contact:
Re: Performance issue (CPU load is 100%)
Has the vCPU count been reduced?
-
- Veeam ProPartner
- Posts: 27
- Liked: 4 times
- Joined: Jan 31, 2012 2:00 pm
- Full Name: Giorgio Colucci
- Location: Italy
- Contact:
Re: Performance issue (CPU load is 100%)
Any update?
There are a way to set a maximum CPU utilization for the veeam proxy service?
Thank you
There are a way to set a maximum CPU utilization for the veeam proxy service?
Thank you
Giorgio
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Performance issue (CPU load is 100%)
Hi Giorgio,
Having your proxy server fully utilized while running multiple backup/replication jobs at the same time is absolutely normal behavior. How many jobs do you concurrently run? What are the specs of your proxy server?
Thanks!
Having your proxy server fully utilized while running multiple backup/replication jobs at the same time is absolutely normal behavior. How many jobs do you concurrently run? What are the specs of your proxy server?
Thanks!
-
- Veeam ProPartner
- Posts: 27
- Liked: 4 times
- Joined: Jan 31, 2012 2:00 pm
- Full Name: Giorgio Colucci
- Location: Italy
- Contact:
Re: Performance issue (CPU load is 100%)
1 job.
The source proxy are a virtual machine win 7 pro x86 with 4 processor and 3Gb RAM, the proxy mode is "Virtual appliance" with cbt enabled and compression an dedup on the fly.
The stats see that the bottleneck is the source proxy at 99%.
Make sense to upgrade the processor number to 8 (or more if SO support them?
any ideas?
The source proxy are a virtual machine win 7 pro x86 with 4 processor and 3Gb RAM, the proxy mode is "Virtual appliance" with cbt enabled and compression an dedup on the fly.
The stats see that the bottleneck is the source proxy at 99%.
Make sense to upgrade the processor number to 8 (or more if SO support them?
any ideas?
Giorgio
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Performance issue (CPU load is 100%)
Having "source" reported as a bottleneck is not bad at all, assuming that you're getting decent speed for your jobs during full and incremental runs.
If you decide to run more jobs through this proxy while having in-guest CPU resources fully utilized, then you will need to add more vCPUs or configure an additional proxy sever that will meet our recommended system requirements. These requirements can be found in the Release Notes document included with your download. Hope this helps!
If you decide to run more jobs through this proxy while having in-guest CPU resources fully utilized, then you will need to add more vCPUs or configure an additional proxy sever that will meet our recommended system requirements. These requirements can be found in the Release Notes document included with your download. Hope this helps!
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Performance issue (CPU load is 100%)
No such bottleneck as "source proxy". Please clarify if bottleneck reported is "source" or "proxy", because these are two different possible bottleneck counter readings. I *think* you are talking about "proxy" bottleneck, but Vitaly's response above is for "source" bottleneck.BigGeorge wrote:The stats see that the bottleneck is the source proxy at 99%.
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Performance issue (CPU load is 100%)
Compression and dedup are at default level, or you increased them? Higher level produce smaller backup files but put extra load on CPUs...BigGeorge wrote:The source proxy are a virtual machine win 7 pro x86 with 4 processor and 3Gb RAM, the proxy mode is "Virtual appliance" with cbt enabled and compression an dedup on the fly.
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Veeam ProPartner
- Posts: 27
- Liked: 4 times
- Joined: Jan 31, 2012 2:00 pm
- Full Name: Giorgio Colucci
- Location: Italy
- Contact:
Re: Performance issue (CPU load is 100%)
I'm sorry: the bottleneck is proxy 99%.
The dedup and compression are at default level and are "optimized for LAN target" (the repository is a third phisical windows server configured as "windows repository type".
The dedup and compression are at default level and are "optimized for LAN target" (the repository is a third phisical windows server configured as "windows repository type".
Giorgio
-
- Veeam ProPartner
- Posts: 27
- Liked: 4 times
- Joined: Jan 31, 2012 2:00 pm
- Full Name: Giorgio Colucci
- Location: Italy
- Contact:
Re: Performance issue (CPU load is 100%)
I know that, but in this case I need maximum compression for this job. But I don't want that the proxy vm uses 100% CPU for veeam process because that freeze other services witch lives on that vm!dellock6 wrote:Compression and dedup are at default level, or you increased them? Higher level produce smaller backup files but put extra load on CPUs...
Thank you
Giorgio
Giorgio
-
- VP, Product Management
- Posts: 6035
- Liked: 2860 times
- Joined: Jun 05, 2009 12:57 pm
- Full Name: Tom Sightler
- Contact:
Re: Performance issue (CPU load is 100%)
With 6.1 the proxy process is given a lower priority than other system processes so it should not really conflict with other services. That being said, it sounds like you want maximum compression, but for the system to not use 100% CPU, which are two goals that are somewhat at odds with each other. One option would be to throttle the traffic using the built in throttling (either on the repository or on the network between proxies/repositories). Artificially limiting the available bandwidth would lower the load on the proxy. Other than that, you might consider using a dedicated proxy.
-
- Veteran
- Posts: 261
- Liked: 29 times
- Joined: May 03, 2011 12:51 pm
- Full Name: James Pearce
- Contact:
Re: Performance issue (CPU load is 100%)
Since proxy CPU load seems to come up quite a bit, maybe Veeam might consider implemented a grid-type strategy for Veeam proxy agents, so that any available desktop PC could chip-in as required, to produce the highest possible compression rates by involving many processors. (I've seen this done with PDF report generation with some very impressive results.)
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Performance issue (CPU load is 100%)
Uhm, this would imply adding another role to the multi-role design, since proxies are also responsible for extracting data from the storage, and I suspect it's the problem behind the limit that a VM can only be processed by a single proxy, do to some locking on the files.
Also, since the IO part is always done by the proxy, moving raw data to a "compressor" role (we can call it so?) would imply basically loosing the benefits of low bandwidth consumption due to compression done in the proxy and then dehidrated data moved to the repository. Compressor role is already inside the proxy right now...
A possible idea would instead be to split the load on several proxies on a per-VM base more than a per-Job base like today, but again, compression and deduplication are done now per-Job, so the compressor role (proxy right now) would need a way to share compression/deduplication patterns with the other compressor helping it. Maybe Veeam guys already evaluated and discarded this scenario
Did I understood right your idea?
Luca.
Also, since the IO part is always done by the proxy, moving raw data to a "compressor" role (we can call it so?) would imply basically loosing the benefits of low bandwidth consumption due to compression done in the proxy and then dehidrated data moved to the repository. Compressor role is already inside the proxy right now...
A possible idea would instead be to split the load on several proxies on a per-VM base more than a per-Job base like today, but again, compression and deduplication are done now per-Job, so the compressor role (proxy right now) would need a way to share compression/deduplication patterns with the other compressor helping it. Maybe Veeam guys already evaluated and discarded this scenario
Did I understood right your idea?
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Veteran
- Posts: 261
- Liked: 29 times
- Joined: May 03, 2011 12:51 pm
- Full Name: James Pearce
- Contact:
Re: Performance issue (CPU load is 100%)
Pretty much!
So the proxy would sends say 100 blocks to PCs for super-compression, and after some time receives the 100 blocks back. And hopefully both dT and the resultant data size are significantly less than would have been possible within the proxy.
So the proxy would sends say 100 blocks to PCs for super-compression, and after some time receives the 100 blocks back. And hopefully both dT and the resultant data size are significantly less than would have been possible within the proxy.
-
- Veeam ProPartner
- Posts: 27
- Liked: 4 times
- Joined: Jan 31, 2012 2:00 pm
- Full Name: Giorgio Colucci
- Location: Italy
- Contact:
Re: Performance issue (CPU load is 100%)
Another idea must be move the compression and deduplication role at the repository level.
Option 1. Dedup and compression on the proxy (now we're here). But give me the ability to set the maximum CPU usage for the proxy process (in this mode I can enforce maximum compression/dedup at the proxy that are near the source: best for WAN)
Option 2. Create an "intermediate" proxy role (flow: source->proxy_near_source(actual)->proxy_in_the_middle(new!)->repository(or destinatio_proxy for replication)->disk) to assign them the dedup/compression funcions
Option 3. Delegate to the destination the dedup/compression functions (that allow to use the CPU in the repository-server).
Option 4. At the "intermediate_proxy" (see option2) give the ability to merge stream from differents proxies (example: to allow best dedup/compression for WAN replica of more jobs running at the same time).
... good things ... but today in what manner I can optimize my job?
Option 1. Dedup and compression on the proxy (now we're here). But give me the ability to set the maximum CPU usage for the proxy process (in this mode I can enforce maximum compression/dedup at the proxy that are near the source: best for WAN)
Option 2. Create an "intermediate" proxy role (flow: source->proxy_near_source(actual)->proxy_in_the_middle(new!)->repository(or destinatio_proxy for replication)->disk) to assign them the dedup/compression funcions
Option 3. Delegate to the destination the dedup/compression functions (that allow to use the CPU in the repository-server).
Option 4. At the "intermediate_proxy" (see option2) give the ability to merge stream from differents proxies (example: to allow best dedup/compression for WAN replica of more jobs running at the same time).
... good things ... but today in what manner I can optimize my job?
Giorgio
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Performance issue (CPU load is 100%)
Lower the compression level. Or go with the options Tom has suggested above.BigGeorge wrote:... good things ... but today in what manner I can optimize my job?
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Performance issue (CPU load is 100%)
As I said, moving compression and deduplication to the repository will then create a new bottleneck on network, since data as to be moved in raw format. Since VMs are becoming bigger and bigger, I think the compression and deduplication needs to happen as near as possible to the production storage.
Luca.
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Veeam ProPartner
- Posts: 27
- Liked: 4 times
- Joined: Jan 31, 2012 2:00 pm
- Full Name: Giorgio Colucci
- Location: Italy
- Contact:
Re: Performance issue (CPU load is 100%)
yes I understand, but is not always truedellock6 wrote:As I said, moving compression and deduplication to the repository will then create a new bottleneck on network, since data as to be moved in raw format. Since VMs are becoming bigger and bigger, I think the compression and deduplication needs to happen as near as possible to the production storage.
Giorgio
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Performance issue (CPU load is 100%)
So, please describe me a use case where is better to ship raw data across the network instead of compressing/deduplicating them before going into the network.
Maybe it's me, but I'm missing a good scenario for it... 100% cpu in a proxy to me is a good value, they are designed to do that, and you can add more to balance the load.
Luca.
Maybe it's me, but I'm missing a good scenario for it... 100% cpu in a proxy to me is a good value, they are designed to do that, and you can add more to balance the load.
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Veteran
- Posts: 261
- Liked: 29 times
- Joined: May 03, 2011 12:51 pm
- Full Name: James Pearce
- Contact:
Re: Performance issue (CPU load is 100%)
It really all depends on whether it's cheaper (in terms of time) to compress then send, send-compress-send or just send. Which obviously depends on compression throughput and link speeds between source and destination.
My observation was that for WAN replication, using a grid to compute could improve overall performance by using much more aggressive compression, since the limiting factor is the WAN in terms of data transmission, rather than the LAN, and the proxy CPUs in terms of compression. Of course there's a balance to be had
My observation was that for WAN replication, using a grid to compute could improve overall performance by using much more aggressive compression, since the limiting factor is the WAN in terms of data transmission, rather than the LAN, and the proxy CPUs in terms of compression. Of course there's a balance to be had
-
- Chief Product Officer
- Posts: 31816
- Liked: 7302 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Performance issue (CPU load is 100%)
Change compression from Best to Optimal. Comparing to Optimal, Best compression gives only 2-3% data reduction at best - at the cost of MASSIVE hit on proxy CPU. It does not really worth using, and at some point (after doing detailed compression level testing) we were close to removing this compression level completely, because of very little difference w/optimal. We kept it only because this would cause too much questions "why we are reducing functionality". We are considering replacing the compression algorithm behind this compression level for one that provides better compression for such CPU load - but for now, I would strongly recommend to simply not use Best compression unless you have monster proxy servers, and stick to Optimal instead. You really are not winning almost anything by using Best compression.BigGeorge wrote:... good things ... but today in what manner I can optimize my job?
Who is online
Users browsing this forum: No registered users and 54 guests