Offload performance of a single VIB or VBK tops out at 10MB/sec

[ConradGoodman]

I am evaluating the speed at which our longstanding Capacity Tier Offloads are running.

We have various jobs that run in a 24 hour period.

If we offload a bunch of VMs, we can use the most of our 1Gbit Internet connection, offloading at 800Mbps+

When we offload a diff backup from a database, that is hundreds of GB or even over a 1TB, the speed drops down to 12MB/sec or about 100mbit.

We are now seeing issues where we take more than a day to offload a backup.

I cannot find any settings that would allow me to parallelise this task for more performance, are there any limitations in Veeam or AWS S3 that I can use to tune this up?

[Andreas Neufert]

By default Veeam is creates per task slot in use, 64 parallel S3 operations. You can increase this value for a test.
Important point is that in total you do not go higher than 3000 operations in parallel. Means S3ConcurrentTaskLimit multiplied with the maximum task slots setting on the Object Repository wizard should not go higher than 3000.

S3ConcurrentTaskLimit
The number of concurrent tasks while working with archive tier (concurrent HTTP requests for data download/upload from/to S3). By default = 64, which means that the job processes data within 64 parallel threads. Can be set either on Veeam B&R and passed to all extents/gateways or set directly on the extent/gateway (the one on Veeam B&R has a higher priority).

Linux:
/etc/VeeamAgentConfig
S3ConcurrentTaskLimit = 64

Windows:
Reg Key
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
S3ConcurrentTaskLimit
(DWORD)
64

[ConradGoodman]

Thank you Andreas.

I also noticed that we had set in Network Traffic Rules: DISABLED: Use multiple upload streams per job. Will this be having an impact on S3? Sounds like maybe it won't from your message... But when we are out of change freeze I would like to experiment with it.

[ConradGoodman]

in Get-NetTCPConnection I am seeing only 12-13 connections established to the proxy server at any one time...

There are ~170 in total, of which most are in TimeWait.

So it looks like there are a lot of concurrent connections, but it just isn't sending data through them... If we aren't getting the 64, would increasing the above reg key make any difference?

[ConradGoodman]

E.g this is truncated, there are many in TimeWait.

PS Get-NetTCPConnection | Where-Object { $_.RemotePort -eq 8080 }

LocalAddress LocalPort RemoteAddress RemotePort State AppliedSetting
------------ --------- ------------- ---------- ----- --------------
*.*.*.40 62407 *.*.*.31 8080 Established Datacenter
*.*.*.40 51296 *.*.*.31 8080 Established Datacenter
*.*.*.40 51294 *.*.*.31 8080 Established Internet
*.*.*.40 51289 *.*.*.31 8080 Established Internet
*.*.*.40 51288 *.*.*.31 8080 Established Datacenter
*.*.*.40 51287 *.*.*.31 8080 Established Internet
*.*.*.40 51286 *.*.*.31 8080 Established Datacenter
*.*.*.40 51285 *.*.*.31 8080 Established Datacenter
*.*.*.40 51284 *.*.*.31 8080 Established Datacenter
*.*.*.40 51283 *.*.*.31 8080 Established Datacenter
*.*.*.40 51282 *.*.*.31 8080 Established Datacenter
*.*.*.40 51281 *.*.*.31 8080 Established Datacenter
*.*.*.40 51274 *.*.*.31 8080 TimeWait
*.*.*.40 51273 *.*.*.31 8080 TimeWait
*.*.*.40 51272 *.*.*.31 8080 TimeWait
*.*.*.40 51271 *.*.*.31 8080 TimeWait
*.*.*.40 51270 *.*.*.31 8080 TimeWait
*.*.*.40 51269 *.*.*.31 8080 TimeWait
*.*.*.40 51268 *.*.*.31 8080 TimeWait
*.*.*.40 51267 *.*.*.31 8080 TimeWait
*.*.*.40 51266 *.*.*.31 8080 TimeWait
*.*.*.40 51265 *.*.*.31 8080 Established Datacenter

[Andreas Neufert]

Network Traffic rule connection count is not influencing the S3 offload.

Maybe it is time to open a support ticket so that the team can have a look what is going on. Please add here the support case number.

[ConradGoodman]

[ConradGoodman]

Network Traffic rule connection count is not influencing the S3 offload.

Maybe it is time to open a support ticket so that the team can have a look what is going on. Please add here the support case number.

Case #07524429, Thanks.

[Andreas Neufert]

For your agent screenshot.
Agents are transport service agents that receive data or write data.

For example if you backup a VM a target agent is started and writes to your storage. The agent is stopped then. Then an agent is started to offload the data to object storage.

In the RTS log you see current spread. Google a bit for the location.

Each of these agents when used for cloud offloading pushes data in parallel up to 64 S3 operations by default to the object storage.

[ConradGoodman]

I had a look at this, support have asked me to increase the repo from 4 to 8 tasks, which makes sense... Except, in the RTS log for the current upload we are only using 1 of 4 slots anyway.

In the peak of the log, where we are offloading other VMs at the same time, the local repo consumes 4 slots, and the object storage 70..

Latest log:
[LOG REMOVED BY MODERATOR]

[Andreas Neufert]

Hi Conrad,

I am not sure how to help you further.
As explained when you offload a single backup only one task slot is used.
In order to speed up the task, you can increase the concurrent s3 operations setting (see above) to see if this speed up things.

The general speed depends on many things like source storage, available bandwith, target storage,...

Log files are not allowed in the forum as this is not a support forum and I removed those.

[ConradGoodman]

Fair enough. For posterity the Object Storage is able to use more than 1 task, as it uses 70 tasks at peak in the log... But for 1 offload it is only using 1 task, there is no setting that you have suggested that will influence this as far as I understand.

I will post the solution from support if there is one. I'm doubtful though, seeing as they want us to increase the local repo tasks from 4 to 8, which as you say won't have any effect.

[ConradGoodman]

Reg Key
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
S3ConcurrentTaskLimit
(DWORD)
64

Just checked on this one... A previous support ticket advised us to change it to 1, as we had issues with restores from Object Storage. It was then tuned to 10 and left there.

I will try doubling it to 20 and see if we get the throughput we need, thanks again.

[Andreas Neufert]

Jup this is the reason why you experience low performance.
With AWS S3 you can for sure use the 64 concurrent tasks setting.

Overall you should limit the task slot amount on the repository to maximum of 50.
Not sure why support advised you to go with this, but I would check on root cause and not limit processing to that low value.

[ConradGoodman]

It was for a historical issue back in v11a. We did a big database restore test, but it would not restore from S3. Got a load of time outs. We tuned this setting down and then could do the restore, and that was the end of the matter!

It's obviously had this knock on effect. I will tune it up until the link is fully used, and then document internally that we might have to drop it back down to restore big backups.

[Andreas Neufert]

I don´t think that it is an issue now with the latest release. v11a is more than 3 years old code and we just started there with object storage. Tons of improvement happened since then.

[ConradGoodman]

I assumed that would be the case. Thanks again.