Comprehensive data protection for all workloads
Post Reply
chris352
Novice
Posts: 9
Liked: never
Joined: Oct 14, 2011 10:34 pm
Full Name: Chris Oleson
Contact:

Veeam Local Replication Slow

Post by chris352 » Feb 05, 2013 5:10 pm

Hello all,

I have an issue and I believe its due to our proxy.

I have a replication job that runs twice daily during business hours. This job processes 30ish VM's that have a total size of 3.6TB.

What I'm seeing is that the job is processed one VM at a time. Each job takes roughly 5-20min depending on the VM as expected.

I see that each VM is using the same source/target proxy.
IE: Using source proxy: 10.10.10.33(nbd)
Using target proxy: 10.10.10.33(nbd)

That particular proxy is used for each VM and is a 4 core proxy set to allow 4 simultaneous jobs.
We do have 2 other proxy's. One with 4 cores and one with 8. (I will be changing them all to 8 this evening).

Any reason why they are being processed sequentially? (Is this working as intended?).

I'm also seeing the target being the bottleneck at 99%, however the jobs are being processed quickly.

See screenshots below of one job after another and the time stamp.


Image
Image

tsightler
VP, Product Management
Posts: 5289
Liked: 2144 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Local Replication Slow

Post by tsightler » Feb 05, 2013 5:29 pm

Veeam does not support parallel processing within a single job. If you want to have parallel processing of VMs you should split them into multiple jobs. Also, since this is local replication you'll want to disable any compression on the job. Having the "target" as the bottleneck is normal since, with local replication, writing to the target will almost always be the slowest part of the operation.

pcrebe
Enthusiast
Posts: 94
Liked: 1 time
Joined: Dec 06, 2010 10:41 pm
Full Name: CARLO
Contact:

Re: Veeam Local Replication Slow

Post by pcrebe » Feb 05, 2013 8:46 pm

Why disable compression?

Thanks,
Carlo

tsightler
VP, Product Management
Posts: 5289
Liked: 2144 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Local Replication Slow

Post by tsightler » Feb 05, 2013 9:15 pm

Compression is just wasting CPU resources since this is local replication and thus both the source and target proxy are the same server. Not much sense for one process on the proxy to spend CPU cycles compressing the data only for another process on the very same proxy to spend cycles decompressing it.

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Veeam Local Replication Slow

Post by Yuki » Feb 06, 2013 3:52 am

Hmmm i want to hear from Veeam guys on this subject. We have a proxy VM deployed on all our hosts. And even for local replication we have compression enabled. Proxies have 8 cores and are never pegged at 100% CPU (except when we push to remote site).

As to the original poster: you want to deploy proxies local to VMs that are being processed so that they can use Hot-Add mode through host's I/O stack.

tsightler
VP, Product Management
Posts: 5289
Liked: 2144 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Local Replication Slow

Post by tsightler » Feb 06, 2013 5:12 am

Yuki wrote:Hmmm i want to hear from Veeam guys on this subject. We have a proxy VM deloyed on all our hosts. And even for local replication we have compression enabled. Proxies have 8 cores and are never pegged at 100% CPU (except when we push to remote site).
I think you'll find that I count as one of the "Veeam guys". :mrgreen: I currently work for Veeam as a Solutions Architect focused primarily on B&R. I spend my days training customers and partners on how to deploy B&R.

The way our architecture works is very simple, a VeeamAgent.exe process is started on the source proxy, and a VeeamAgent.exe process is started on the target proxy and they connect to each other. The source proxy compresses the data, sends it to the target proxy, which then decompresses the data. If the same server is used for both the source and target proxy it doesn't really change the process, the server will have two VeeamAgent.exe processes which connect to each other locally. It doesn't make a lot of sense to waste the CPU cycles to send compressed data between two processes on the same VM.

If you need more proof, here's a quick log of a full replication job of a small VM with compression enabled:

Code: Select all

2/5/2013 11:14:59 PM :: Job started at 2/5/2013 11:14:56 PM
2/5/2013 11:14:59 PM :: Building VM list
2/5/2013 11:15:43 PM :: VM size: 45.0 GB (26.6 GB used)
2/5/2013 11:15:43 PM :: Changed block tracking is enabled
2/5/2013 11:15:45 PM :: Preparing next VM for processing
2/5/2013 11:15:45 PM :: Processing 'srv01'
2/5/2013 11:29:40 PM :: All VMs have been processed
2/5/2013 11:29:41 PM :: Load: Source 10% > Proxy 66% > Network 17% > Target 99%
2/5/2013 11:29:41 PM :: Primary bottleneck: Target
2/5/2013 11:29:41 PM :: Job finished at 2/5/2013 11:29:41 PM
and then with compression disabled:

Code: Select all

2/5/2013 11:34:42 PM :: Job started at 2/5/2013 11:34:40 PM
2/5/2013 11:34:42 PM :: Building VM list
2/5/2013 11:35:25 PM :: VM size: 45.0 GB (26.6 GB used)
2/5/2013 11:35:25 PM :: Changed block tracking is enabled
2/5/2013 11:35:27 PM :: Preparing next VM for processing
2/5/2013 11:35:27 PM :: Processing 'srv01'
2/5/2013 11:48:55 PM :: All VMs have been processed
2/5/2013 11:48:57 PM :: Load: Source 5% > Proxy 32% > Network 38% > Target 99%
2/5/2013 11:48:57 PM :: Primary bottleneck: Target
2/5/2013 11:48:57 PM :: Job finished at 2/5/2013 11:48:57 PM
Notice that the second job with compression disabled finished slightly faster (it could have been even faster, but the target, a slow iSCSI array, was the bottleneck for both runs) however, the CPU load was cut almost in half by disabling compression.

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Veeam Local Replication Slow

Post by Yuki » Feb 06, 2013 6:07 am

But it still makes sense to use compression on LAN when Proxies mount disks via Hot-Add, right? Especially if proxy is not a bottleneck (so compression is not the slowest link, yet provides performance gain by sending less data due to compression).

v.eremin
Product Manager
Posts: 16126
Liked: 1314 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam Local Replication Slow

Post by v.eremin » Feb 06, 2013 8:27 am

As Tom has already said, If we took local replication(1-proxy model) as an example, It wouldn’t make any sense to use compression, since whatever proxy method were to be used, compression and decompression process would be taking place within one proxy server. In other words, it’s nothing but time/resource wasting.

One should always remember that Veeam B&R is using two-agent based architecture. And all of features like throttling rules, compression, etc, are implemented between them. In case of on-site replication(1-proxy model), the source-side agent and the target-side agent will be started on the same backup proxy and the process of compression/decompression won’t leave the bounds of this machine.

Meanwhile, it completely stands to reason to utilize compression functionality in case of on-site backup, for instance. In this scenario, the source-side agent is started on the proxy server, and the target-side agent is started on the Windows or Linux repository server. Data will be compressed on the proxy server, then flow though the LAN, and after it will be decompressed on the target side.

Hope this helps.
Thanks.

foggy
Veeam Software
Posts: 17700
Liked: 1481 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam Local Replication Slow

Post by foggy » Feb 06, 2013 11:10 am

Yuki wrote:But it still makes sense to use compression on LAN when Proxies mount disks via Hot-Add, right? Especially if proxy is not a bottleneck (so compression is not the slowest link, yet provides performance gain by sending less data due to compression).
It might make sense to use compression if different proxies are used to read data from the source and write it to the target, which is the case when the source proxy does not have direct access to the target datastore for hotadd.

tsightler
VP, Product Management
Posts: 5289
Liked: 2144 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam Local Replication Slow

Post by tsightler » Feb 06, 2013 5:42 pm

Yuki wrote:But it still makes sense to use compression on LAN when Proxies mount disks via Hot-Add, right? Especially if proxy is not a bottleneck (so compression is not the slowest link, yet provides performance gain by sending less data due to compression).
That question cannot be answered definitively with more information. There is always some overhead in compression, even if the proxy is not the "bottleneck" it uses processing resources than can impact the throughput of other portions in the chain, just like the example I posted above. In neither case was the "proxy" the bottleneck, but yet disabling compression, and thus lowering the overall load on the proxy, still allowed the replication to be slightly faster.

In general, compression is only a win for replication (as stated above, it's different for backups) when "Network" is the bottleneck. If "Network" is not the bottleneck, then compression is probably just using CPU resources for no benefit. For a 1Gb LAN there might be some advantage to using compression, especially if you are running mulitple concurrent jobs, but it's worth testing. Once you get to 10Gb, there's rarely cases where compression makes sense.

However, none of that applies to my original post, which was specifically addressing the case of a single proxy used as both source and target (whether hotadd/network/direct SAN/whatever). In this case, compression is never warranted in any scenario that I can come up with.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], foggy, tsightler and 42 guests