Discussions related to using object storage as a backup target.
Post Reply
bhnelson
Enthusiast
Posts: 34
Liked: 9 times
Joined: Oct 01, 2018 5:32 pm
Full Name: Brian Nelson
Contact:

Improving SOBR offload performance?

Post by bhnelson »

Using Amazon S3 for capacity tier.

I've been doing testing with the new SOBR 'copy' mode introduced in v10. With a (I think) fairly basic SOBR setup, I'm seeing around 700Mbps offload speed. Based on our current in-house solution I should be able to get 2-3 times that. The in-house solution (powershell w/AWS modules) uses the same storage, compute, and network path as Veeam is using. We have a 4Gbps internet connection. What should I be expecting out of Veeam?

I don't see any resource starvation on the Veeam server (acting as the S3 gateway). The offload job lists bottlenecks as target 40%, proxy 20%, source & network 17%. But those seem to vary wildly between runs.

Are there any knobs to turn or other suggestions? I've seen some references to S3ConcurrentTaskLimit registry key but no real info on how or when to set it. I've also seen mentions of Veeam block size having an effect. I've tried using 4k blocks (instead of 1k) which resulted in a modest improvement but still much less than I'd expect.

Thanks!
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Improving SOBR offload performance?

Post by HannesK »

Hello,
or other suggestions?
yes, support is always the first point to check out to ensure that everything is configured correctly (e.g. repository tasks)... what is your case number?

The S3ConcurrentTaskLimit was describe here and it would more be to limit the speed. I think we should aim for higher speed. How many VMs are you uploading in parallel and how many tasks have you set on your repository?

I remember that Azure Blob has some upload limits which are below 1GBit/s. Did you already ask Amazon whether there is any throttling from their side? I see similar speed to AWS here: post312587.html#p312475

Thanks,
Hannes
bhnelson
Enthusiast
Posts: 34
Liked: 9 times
Joined: Oct 01, 2018 5:32 pm
Full Name: Brian Nelson
Contact:

Re: Improving SOBR offload performance?

Post by bhnelson » 1 person likes this post

I saw that thread. That's precisely where I first saw S3ConcurrentTaskLimit mentioned. Well, there and also here. I wondered if there were more general usage 'suggestions' beyond that specific situation. Particularly I wondered if it might ever need increased. That doesn't seem to be the case though. More on that below.

The limits in Amazon S3 are transaction-based. They're on the order of 3,500 PUTs/sec though, which is huge. See here. In theory there is no particular bandwidth limit for S3. I verified that with them at ReInvent. (As an aside, there is however a 1.25Gbps limit per VPN connection to AWS.)

In my case, it turns out resource monitor was tricking me. Even though the server never went much above 75-80% CPU, there was apparently some CPU limitation happening. Does VeeamAgent.exe have some built-in nice-ness going on that would hide actual demand? Regardless, I added additional cores to the Veeam VM (ie acting proxy) and that un-corked it straight away. I was able to get 3+Gbps to Amazon while offloading 4-5 VMs at once. Pretty good!

After the dust settled though, I did find a legit bottleneck in VeeamAgent.exe. I won't call it a 'problem' as such but it would be a good RFE target. For a single VM's offload, it gets one VeeamAgent.exe process, which ends up CPU-bound on one thread internally. In my environment, this tops offload throughput right around an ironic 100MB/s. As a side effect (in general), it doesn't seem to get anywhere near S3ConcurrentTaskLimit number of connections. The most I've seen was around 20-25 at once but usually less. Additionally, this seems to make Veeam's 'bottleneck reporting' completely bonkers. For sure, proxy CPU is the limiting factor in this case, but it's often the smallest bottleneck percentage reported. And those number vary wildly between jobs.

Here's a pic showing that one thread chugging away. 16 cores so 6.25% CPU = 100% of one core.
Image
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving SOBR offload performance?

Post by Gostev »

Thanks for the follow up and sharing the resolution, I'm glad you're getting great performance now!

As far as your RFE, I have confirmed with the devs that this bottleneck is already addressed in the next major release branch (v11).
bhnelson
Enthusiast
Posts: 34
Liked: 9 times
Joined: Oct 01, 2018 5:32 pm
Full Name: Brian Nelson
Contact:

Re: Improving SOBR offload performance?

Post by bhnelson »

Nice!

I can't recall if I've seen it elsewhere: Any rough idea of when we'll see v11 release? I don't imagine it'll be soon.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Improving SOBR offload performance?

Post by veremin »

Unfortunately, we cannot provide any ETA right now. Thanks!
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving SOBR offload performance?

Post by Gostev » 1 person likes this post

bhnelson wrote: Jun 15, 2020 4:36 pmI can't recall if I've seen it elsewhere: Any rough idea of when we'll see v11 release? I don't imagine it'll be soon.
Our major release cadence is annual (plus-minus a couple of months), so this should give you an approximate idea. We just don't try to estimate the specific month until we get into the beta stage.
ajgball
Influencer
Posts: 11
Liked: never
Joined: Nov 05, 2009 12:18 pm
Full Name: AJ GB
Contact:

Re: Improving SOBR offload performance?

Post by ajgball »

Not sure this is actually fixed in v11 still seems to bottle neck if you have the one backup job going up to object storage
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Improving SOBR offload performance?

Post by veremin »

Have you had a support ticket opened for slow performance of offload process? If so, can you post its number so we can check debug logs and see where actual bottleneck resides? Thanks!
stuart_little1874
Influencer
Posts: 22
Liked: 4 times
Joined: Jul 22, 2020 1:25 pm
Full Name: Stuart Little
Contact:

Re: Improving SOBR offload performance?

Post by stuart_little1874 »

Can you tell me if you added the additional Cores to the Server acting as the Veeam Proxy, Repository, or Gateway?

We have a similar issue where it's crawling along at 3 MB/s.

I'll log a call, upload some logs and Post the Case ID here ...
stuart_little1874
Influencer
Posts: 22
Liked: 4 times
Joined: Jul 22, 2020 1:25 pm
Full Name: Stuart Little
Contact:

Re: Improving SOBR offload performance?

Post by stuart_little1874 »

Case #04785518 — Slow SOBR off-load
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Improving SOBR offload performance?

Post by veremin »

Thanks, Stuart, for sharing the support ticket number - now let's wait and see what support team come up with after debug log investigation. Thanks!
bhnelson
Enthusiast
Posts: 34
Liked: 9 times
Joined: Oct 01, 2018 5:32 pm
Full Name: Brian Nelson
Contact:

Re: Improving SOBR offload performance?

Post by bhnelson »

OP here. Finally upgraded to Veeam 11.latest a few weeks ago. Disappointed to report that single file offload performance really hasn't improved much.

It no longer seems to be cpu-bound on a single thread. Instead though, it seems to be bursting on a fairly constant cycle. Literally going between 0 and about 2Gbps with a 50% duty cycle. It really seems to me like it's throttling itself, though I've confirmed I have no throttling options set. I've confirmed it's almost certainly not a resource problem.

I'll maybe open a support case at some point but it's not a priority at the moment. I wanted to post an update though.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Improving SOBR offload performance?

Post by veremin »

Unfortunately it's rather hard to comment on performance topics without seeing the current infrastructure and debug logs. So if this situation causes more issues, kindly open a ticket and post its number here - will see how we can help you. Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests