Veeam Local Replication Slow

[chris352]

Hello all,

I have an issue and I believe its due to our proxy.

I have a replication job that runs twice daily during business hours. This job processes 30ish VM's that have a total size of 3.6TB.

What I'm seeing is that the job is processed one VM at a time. Each job takes roughly 5-20min depending on the VM as expected.

I see that each VM is using the same source/target proxy.
IE: Using source proxy: 10.10.10.33(nbd)
Using target proxy: 10.10.10.33(nbd)

That particular proxy is used for each VM and is a 4 core proxy set to allow 4 simultaneous jobs.
We do have 2 other proxy's. One with 4 cores and one with 8. (I will be changing them all to 8 this evening).

Any reason why they are being processed sequentially? (Is this working as intended?).

I'm also seeing the target being the bottleneck at 99%, however the jobs are being processed quickly.

See screenshots below of one job after another and the time stamp.

[tsightler]

Veeam does not support parallel processing within a single job. If you want to have parallel processing of VMs you should split them into multiple jobs. Also, since this is local replication you'll want to disable any compression on the job. Having the "target" as the bottleneck is normal since, with local replication, writing to the target will almost always be the slowest part of the operation.

[pcrebe]

Why disable compression?

Thanks,
Carlo

[tsightler]

Compression is just wasting CPU resources since this is local replication and thus both the source and target proxy are the same server. Not much sense for one process on the proxy to spend CPU cycles compressing the data only for another process on the very same proxy to spend cycles decompressing it.

[Yuki]

Hmmm i want to hear from Veeam guys on this subject. We have a proxy VM deployed on all our hosts. And even for local replication we have compression enabled. Proxies have 8 cores and are never pegged at 100% CPU (except when we push to remote site).

As to the original poster: you want to deploy proxies local to VMs that are being processed so that they can use Hot-Add mode through host's I/O stack.

[tsightler]

Hmmm i want to hear from Veeam guys on this subject. We have a proxy VM deloyed on all our hosts. And even for local replication we have compression enabled. Proxies have 8 cores and are never pegged at 100% CPU (except when we push to remote site).

I think you'll find that I count as one of the "Veeam guys". I currently work for Veeam as a Solutions Architect focused primarily on B&R. I spend my days training customers and partners on how to deploy B&R.

The way our architecture works is very simple, a VeeamAgent.exe process is started on the source proxy, and a VeeamAgent.exe process is started on the target proxy and they connect to each other. The source proxy compresses the data, sends it to the target proxy, which then decompresses the data. If the same server is used for both the source and target proxy it doesn't really change the process, the server will have two VeeamAgent.exe processes which connect to each other locally. It doesn't make a lot of sense to waste the CPU cycles to send compressed data between two processes on the same VM.

If you need more proof, here's a quick log of a full replication job of a small VM with compression enabled:

Code: Select all

2/5/2013 11:14:59 PM :: Job started at 2/5/2013 11:14:56 PM
2/5/2013 11:14:59 PM :: Building VM list
2/5/2013 11:15:43 PM :: VM size: 45.0 GB (26.6 GB used)
2/5/2013 11:15:43 PM :: Changed block tracking is enabled
2/5/2013 11:15:45 PM :: Preparing next VM for processing
2/5/2013 11:15:45 PM :: Processing 'srv01'
2/5/2013 11:29:40 PM :: All VMs have been processed
2/5/2013 11:29:41 PM :: Load: Source 10% > Proxy 66% > Network 17% > Target 99%
2/5/2013 11:29:41 PM :: Primary bottleneck: Target
2/5/2013 11:29:41 PM :: Job finished at 2/5/2013 11:29:41 PM

and then with compression disabled:

Code: Select all

2/5/2013 11:34:42 PM :: Job started at 2/5/2013 11:34:40 PM
2/5/2013 11:34:42 PM :: Building VM list
2/5/2013 11:35:25 PM :: VM size: 45.0 GB (26.6 GB used)
2/5/2013 11:35:25 PM :: Changed block tracking is enabled
2/5/2013 11:35:27 PM :: Preparing next VM for processing
2/5/2013 11:35:27 PM :: Processing 'srv01'
2/5/2013 11:48:55 PM :: All VMs have been processed
2/5/2013 11:48:57 PM :: Load: Source 5% > Proxy 32% > Network 38% > Target 99%
2/5/2013 11:48:57 PM :: Primary bottleneck: Target
2/5/2013 11:48:57 PM :: Job finished at 2/5/2013 11:48:57 PM

Notice that the second job with compression disabled finished slightly faster (it could have been even faster, but the target, a slow iSCSI array, was the bottleneck for both runs) however, the CPU load was cut almost in half by disabling compression.

[Yuki]

But it still makes sense to use compression on LAN when Proxies mount disks via Hot-Add, right? Especially if proxy is not a bottleneck (so compression is not the slowest link, yet provides performance gain by sending less data due to compression).

[veremin]

As Tom has already said, If we took local replication(1-proxy model) as an example, It wouldn’t make any sense to use compression, since whatever proxy method were to be used, compression and decompression process would be taking place within one proxy server. In other words, it’s nothing but time/resource wasting.

One should always remember that Veeam B&R is using two-agent based architecture. And all of features like throttling rules, compression, etc, are implemented between them. In case of on-site replication(1-proxy model), the source-side agent and the target-side agent will be started on the same backup proxy and the process of compression/decompression won’t leave the bounds of this machine.

Meanwhile, it completely stands to reason to utilize compression functionality in case of on-site backup, for instance. In this scenario, the source-side agent is started on the proxy server, and the target-side agent is started on the Windows or Linux repository server. Data will be compressed on the proxy server, then flow though the LAN, and after it will be decompressed on the target side.

Hope this helps.
Thanks.

[foggy]

But it still makes sense to use compression on LAN when Proxies mount disks via Hot-Add, right? Especially if proxy is not a bottleneck (so compression is not the slowest link, yet provides performance gain by sending less data due to compression).

It might make sense to use compression if different proxies are used to read data from the source and write it to the target, which is the case when the source proxy does not have direct access to the target datastore for hotadd.

[tsightler]

But it still makes sense to use compression on LAN when Proxies mount disks via Hot-Add, right? Especially if proxy is not a bottleneck (so compression is not the slowest link, yet provides performance gain by sending less data due to compression).

That question cannot be answered definitively with more information. There is always some overhead in compression, even if the proxy is not the "bottleneck" it uses processing resources than can impact the throughput of other portions in the chain, just like the example I posted above. In neither case was the "proxy" the bottleneck, but yet disabling compression, and thus lowering the overall load on the proxy, still allowed the replication to be slightly faster.

In general, compression is only a win for replication (as stated above, it's different for backups) when "Network" is the bottleneck. If "Network" is not the bottleneck, then compression is probably just using CPU resources for no benefit. For a 1Gb LAN there might be some advantage to using compression, especially if you are running mulitple concurrent jobs, but it's worth testing. Once you get to 10Gb, there's rarely cases where compression makes sense.

However, none of that applies to my original post, which was specifically addressing the case of a single proxy used as both source and target (whether hotadd/network/direct SAN/whatever). In this case, compression is never warranted in any scenario that I can come up with.