Sobr offload and organization

[theviking84]

Please see https://helpcenter.veeam.com/docs/backu ... ml?ver=100

It says to use one bucket per sobr rather than same bucket different folders for each sobr. Due to metadata size and performance. I've asked on here in the past and was told you can do different folders same bucket and it was fine.

So question there is how big of a deal/performance impact is it to use one bucket multiple folders vs one per sobr?

Second question is offload traffic flow. Veeam will start data mover on each performance extent and data from there to bucket, rather than elsewhere first such as veeam server? I did not specify a gateway server in the sobr/object storage settings. Is that correct if that is the traffic flow I want?

[theviking84]

A specific on the offload traffic flow. If the performance extent is a Linux repo and has only an account with permissions to the backup folder. Is that still going to be able to offload from here to s3? It is not a proxy only a Linux repo. It can't be made a proxy as I believe in v11 a Linux repo that is immutable can't be a proxy.

[Gostev]

So question there is how big of a deal/performance impact is it to use one bucket multiple folders vs one per sobr?

It's a really big deal for object storage scalability indeed, so you should not be using one bucket for multiple SOBR under any circumstances with any decent amount of data in each SOBR. There's actually no point to even contemplate doing this, because it costs nothing to create additional dedicated buckets for each SOBR to use.

By the way, there are NO folders in object storage in principle because there's no file system! What you see as "folders" are merely similar object name prefixes, which some object storage explorers show as folders for convenience, even if they are really not.

Second question is offload traffic flow. Veeam will start data mover on each performance extent and data from there to bucket, rather than elsewhere first such as veeam server? I did not specify a gateway server in the sobr/object storage settings. Is that correct if that is the traffic flow I want?

Yes, that is correct. Permissions don't matter as everything the data mover needs to do is read backup files data from that directory [which it does have access to], before passing this data into the object storage (which is a totally different storage system, so Linux accounts permissions are irrelevant at this point).

[theviking84]

Thank you Gostev!

I'm relieved to hear it will work OK to move from the vm to s3 directly and the regular repo permissions are fine.

I will definitely make two buckets then, one per sobr. I was just thinking of simplifying, but there are also forum threads here that people were told it was fine to use folders on different sobrs with same bucket. Here are two examples on this forum where we were sent the wrong direction. But I know things change as more info is learned and more data goes into buckets. Perhaps the metadata and scalability issues with one bucket wasn't known yet.

object-storage-f52/multiple-sobrs-with- ... 67285.html

object-storage-f52/2-buckets-or-one-and ... 70726.html

[Gostev]

Thanks for linking these topics, I updated both.

You're right, two things changed since the early days of our object storage integration:

1. A huge number of new vendors released their own object storage solutions having [as expected for v1] much worse scalability than those few established vendors that existed for many years now.

2. Some of our customers have been using object storage long enough to start approaching 1PB of backups from multiple SOBR in a single bucket, which apparently may result in performance issues even with the established object storage vendors like Amazon S3.