Offload speeds

[ctg49]

Looking to get some information on offload speeds.

Right now we've got a pretty substantial amount of data that's being offloaded (on the order of 70TB, we're ~30TB in) and I had a couple questions.

We're seeing speeds right now ranging between 20-50MB/s. Bottleneck is usually source, which in this case is a handful of locations, some on the actual VEEAM server (housing two 45TB NTFS volumes), some on a separate storage device. All are backed by ye olden platter disks. Now, my confusion is that whether i'm uploading from multiple disk groups or one disk group, the speeds are almost always the same, which tells me the disk speeds aren't the actual bottleneck. In addition, they aren't really spiking the disk usage in any real way (reading ~30MB/s in resource manager), far lower usage than the disks are actually capable of. So this tells me the bottleneck on the source side has to lie in either the processing of the data, or the method by which it's being processed on the software side.

Given that CPU and RAM usage are minimal, is there something in the VEEAM software which is governing/limiting the processing for offloads? Is this something that can be tweaked, given more resources to, or sped up in some way? I'd rather have shorter time frames when this offloading is able to just cap out system usage, rather than it running for weeks/months on end.

Thanks!

[anthonyspiteri79]

Hey there Chris.

Couple of questions before giving you a specific reply.

1. Are the repos and SOBR configured for per-VM backup chain?
2. What Object Storage are you Offloading to?

[ctg49]

Hey there Chris.

Couple of questions before giving you a specific reply.

1. Are the repos and SOBR configured for per-VM backup chain?
2. What Object Storage are you Offloading to?

1) Yep, per-VM. In all instances, these are actually jobs broken out as per-server or per-volume backups. (long term retention for file servers)
2) Wasabi, an S3 compatible vendor.

[anthonyspiteri79]

Hey Chris.

Per VM backup chains are going to be the most optimal configuration for offload process, so no worries there. Are you seeing multiple veeamagent.exe processes sending data to Wasabi? With regards to Wasabi (who we know well) generally the upload speeds are dictated by local to remote connectivity and also the speed of the service Wasabi allow.

As an example I have just seen upload to another S3-compatible Object Storage max out at about 40Mb/s. While uploading to some AWS regions, i've only goteen 10Mb/s.

Like any initial seed, the good news is that once the bulk of the data is up, the subsequent incremental changes will happen pretty quickly.

[ctg49]

Yep, I can confirm I'm seeing multiple veeamagent.exe processes, each looks like they're accessing a different job's file to upload at a time, so that appears to be working as expected. I'm just curious I guess as to why it's marking the bottleneck of source if the limitation is on the destination side throttling the bandwidth. I'll note that the speeds are faster if I'm uploading from multiple source repositories at once on different systems (40-50MB/s vs 20-30MB/s), which would imply that the source is indeed the bottleneck as the job states, since adding additional sources speeds things up.