One bucket per job

[matteu]

Hello,

Our S3 provider best practice is to use 1 bucket per job and 1 user per bucket...
It's the first S3 provider best practice I see with this recommendation.

Is it something you often see ? It's lot's of overhead management when you have lot's of job... and need to create 1 repository per job too :/

[Mildur]

Hi Matteu

Our recommendation is to follow the storage vendor’s best practices — they know how their appliance works best with our products.
That said, with small sized VMs I expect you can use a single bucket for multiple backup jobs.
Some Object Storage appliances struggle with the number of objects they have to store and manage, not with the number of protected workloads. For example, 500 × 5 GB VMs is less challenging than 5 × 50 TB VMs.

As an alternative to 1 bucket = 1 Job: Did your Object Storage vendor consider multi-bucket repositories in their best practices?
- Multi-bucket repositories

This functionality is available for S3-compatible repositories that do not use our Smart Object Storage API. It is enabled by default.
You create the first bucket and connect it to the backup server. In the repository settings, you can define how many workloads should be stored per bucket. Once the threshold is reached, the backup server will automatically create a new bucket on the object storage appliance via the S3 API.

Keep in mind, that the following workloads do not support multi-bucket repositories:
- Cloud Connect Repositories
- Unstructured Data
- Entra ID Log



Best,
Fabian

[matteu]

Thanks for this detailed answer.

I asked them for this feature but they confirm me the best practice is to use 1 bucket / job.

I will follow their best practice for sure, but I just would like to hear there are others provider with the same recommendation...

[edh]

Hi,

As service provider, and storage provider with S3 system.

Until Veeam give SP the .reg for establish the object size between 8-32MB to all tenants, limit your bucket to 100TB.

THe bucket problem become bigger with the List / Massive listing index problems from "legacy" s3 providers, new systems works better due NVME 5.0 advantages.

There's a lot of S3 system backends, the main problems are the metadata (HEAD) request, not the PUT/GET request. Also DELETE are intensive due IO saturation, some S3 providers got delete markers that "help" doing shadow fast deletions.

Regards,
Manuel