Improving SOBR offload performance?

[bhnelson]

Using Amazon S3 for capacity tier.

I've been doing testing with the new SOBR 'copy' mode introduced in v10. With a (I think) fairly basic SOBR setup, I'm seeing around 700Mbps offload speed. Based on our current in-house solution I should be able to get 2-3 times that. The in-house solution (powershell w/AWS modules) uses the same storage, compute, and network path as Veeam is using. We have a 4Gbps internet connection. What should I be expecting out of Veeam?

I don't see any resource starvation on the Veeam server (acting as the S3 gateway). The offload job lists bottlenecks as target 40%, proxy 20%, source & network 17%. But those seem to vary wildly between runs.

Are there any knobs to turn or other suggestions? I've seen some references to S3ConcurrentTaskLimit registry key but no real info on how or when to set it. I've also seen mentions of Veeam block size having an effect. I've tried using 4k blocks (instead of 1k) which resulted in a modest improvement but still much less than I'd expect.

Thanks!

[HannesK]

Hello,

or other suggestions?

yes, support is always the first point to check out to ensure that everything is configured correctly (e.g. repository tasks)... what is your case number?

The S3ConcurrentTaskLimit was describe here and it would more be to limit the speed. I think we should aim for higher speed. How many VMs are you uploading in parallel and how many tasks have you set on your repository?

I remember that Azure Blob has some upload limits which are below 1GBit/s. Did you already ask Amazon whether there is any throttling from their side? I see similar speed to AWS here: post312587.html#p312475

Thanks,
Hannes

[bhnelson]

I saw that thread. That's precisely where I first saw S3ConcurrentTaskLimit mentioned. Well, there and also here. I wondered if there were more general usage 'suggestions' beyond that specific situation. Particularly I wondered if it might ever need increased. That doesn't seem to be the case though. More on that below.

The limits in Amazon S3 are transaction-based. They're on the order of 3,500 PUTs/sec though, which is huge. See here. In theory there is no particular bandwidth limit for S3. I verified that with them at ReInvent. (As an aside, there is however a 1.25Gbps limit per VPN connection to AWS.)

In my case, it turns out resource monitor was tricking me. Even though the server never went much above 75-80% CPU, there was apparently some CPU limitation happening. Does VeeamAgent.exe have some built-in nice-ness going on that would hide actual demand? Regardless, I added additional cores to the Veeam VM (ie acting proxy) and that un-corked it straight away. I was able to get 3+Gbps to Amazon while offloading 4-5 VMs at once. Pretty good!

After the dust settled though, I did find a legit bottleneck in VeeamAgent.exe. I won't call it a 'problem' as such but it would be a good RFE target. For a single VM's offload, it gets one VeeamAgent.exe process, which ends up CPU-bound on one thread internally. In my environment, this tops offload throughput right around an ironic 100MB/s. As a side effect (in general), it doesn't seem to get anywhere near S3ConcurrentTaskLimit number of connections. The most I've seen was around 20-25 at once but usually less. Additionally, this seems to make Veeam's 'bottleneck reporting' completely bonkers. For sure, proxy CPU is the limiting factor in this case, but it's often the smallest bottleneck percentage reported. And those number vary wildly between jobs.

Here's a pic showing that one thread chugging away. 16 cores so 6.25% CPU = 100% of one core.

[Gostev]

Thanks for the follow up and sharing the resolution, I'm glad you're getting great performance now!

As far as your RFE, I have confirmed with the devs that this bottleneck is already addressed in the next major release branch (v11).

[bhnelson]

Nice!

I can't recall if I've seen it elsewhere: Any rough idea of when we'll see v11 release? I don't imagine it'll be soon.

[veremin]

Unfortunately, we cannot provide any ETA right now. Thanks!

[Gostev]

I can't recall if I've seen it elsewhere: Any rough idea of when we'll see v11 release? I don't imagine it'll be soon.

Our major release cadence is annual (plus-minus a couple of months), so this should give you an approximate idea. We just don't try to estimate the specific month until we get into the beta stage.

[ajgball]

Not sure this is actually fixed in v11 still seems to bottle neck if you have the one backup job going up to object storage

[veremin]

Have you had a support ticket opened for slow performance of offload process? If so, can you post its number so we can check debug logs and see where actual bottleneck resides? Thanks!

[stuart_little1874]

Can you tell me if you added the additional Cores to the Server acting as the Veeam Proxy, Repository, or Gateway?

We have a similar issue where it's crawling along at 3 MB/s.

I'll log a call, upload some logs and Post the Case ID here ...

[stuart_little1874]

Case #04785518 — Slow SOBR off-load

[veremin]

Thanks, Stuart, for sharing the support ticket number - now let's wait and see what support team come up with after debug log investigation. Thanks!

[bhnelson]

OP here. Finally upgraded to Veeam 11.latest a few weeks ago. Disappointed to report that single file offload performance really hasn't improved much.

It no longer seems to be cpu-bound on a single thread. Instead though, it seems to be bursting on a fairly constant cycle. Literally going between 0 and about 2Gbps with a 50% duty cycle. It really seems to me like it's throttling itself, though I've confirmed I have no throttling options set. I've confirmed it's almost certainly not a resource problem.

I'll maybe open a support case at some point but it's not a priority at the moment. I wanted to post an update though.

[veremin]

Unfortunately it's rather hard to comment on performance topics without seeing the current infrastructure and debug logs. So if this situation causes more issues, kindly open a ticket and post its number here - will see how we can help you. Thanks!