Multiple S3-Buckets in SOBR Capacity Tier

[straessle]

Hello,
I created a SOBR and used one S3-Bucket in the Capacity Tier.
Our S3-Bucket is running on a Scality-Ring.
During a review of the Scality-Ring, it was proposed to use more then one bucket to get better performance.
I have configured another 3 buckets, created a Repo for each of them and in the SOBR I selected all 4 Repos.
This was done before the weekend and even after 3 days, Veeam is writing only to on bucket.
Could anyone please explain why?

[sfirmes]

@straessle welcome to the forums and thanks for reaching out. What version of VBR are you using?

Steve

[straessle]

@sfirmes: This is a new installation, we are using 12.1.2.172.

[Mildur]

Hi Marc

What is the backup chain format configured on the performance tier extends?
Single backup file or per-machine backup files?
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Best,
Fabian

[straessle]

Hi Fabian,
The Repo for the performance tier is using "per-machine backup files".
But I don't think this matters in this case. This SOBR has been created for SQL backups using the SQL Plugin.
The backups are created using the SSMS. Unfortunately I don't have access to SSMS and therefore no details about how it's configured.
I'm currently creating a ticket. As soon as I have an answer, I'll post it here.
Kind Regards,
Marc

[straessle]

After setting a limit, lower than actual usage, it started using the other Repos.
As easy as that.

[Gostev]

Could you please clarify what limit do you mean? Thanks

[straessle]

At first there was no limit set on all the Repos and the first Repo1 usage was at more than 140 TB.
Then I set a limit to 130 TB to Repo1.
After this no new backups were written to Repo1, they were written to the other 3 Repos.

[Gostev]

Now it's clear, thank you.

When you have multiple buckets, SOBR will round robin machines between them. However, I now see that all your machines have already been backed up into the single bucket and there's nothing to round robin until new machines are added to the job. Any new machines however will round robin across all available buckets, but for each individual machines all its backups will stay in the same bucket.

In other words, with the current design you're supposed to create SOBR with multiple buckets before you point any backup jobs to it. This will spread all machines evenly across all buckets.

@veremin please ask QA to reproduce what OP is seeing though. Something tells we could be in an uncharted/untested territory. The soft limit OP has set is supposed to be tier-wide, not per-bucket. I suspect SOBR might have reacted in a weird way and is doing something we never tested (adding buckets buckets to existing SOBR + enabling limit).

Regardless, with the settings above when 130TB soft limit is set after SOBR has already stored 140TB, I would expect SOBR to refuse all backups because its used space is above soft limit? But per OP it does let backup jobs write to it.

[veremin]

Regardless, with the settings above when 130TB soft limit is set after SOBR has already stored 140TB, I would expect SOBR to refuse all backups because its used space is above soft limit?

The soft limit is the setting of repository (extent) not SOBR. When an extent hit this mark, it stopped accepting new backups.

@veremin please ask QA to reproduce what OP is seeing though. Something tells we could be in an uncharted/untested territory.

We will discuss with the QA team the scenario above (adding extents and enabling limits on them) and I will post back our findings.

Thanks!

[tyler.jurgens]

Sounds like default SOBR behavior to me. New extents added to a SOBR won't get used until either an Active Full backup is taken, or new VMs are added to the backup job. I've used that soft setting before to force backups onto other extents when I could not manage the backup jobs directly (eg: Cloud Connect repositories where the client is unmananged).

As far as I am aware, there are no better balancing mechanisms for a SOBR.

The other option would be to manually move backups to another extent, but that's really only feasible with a block based repository. I don't think that would even be possible with a S3 repo.

[Gostev]

The soft limit is the setting of repository (extent) not SOBR.

Ah, thanks. I forgot about that. In that case all is working exacty as it should.

[veremin]

Correct, but the QA team is going to re-verify this scenario regardless.

This SOBR has been created for SQL backups using the SQL Plugin.

One question, though, what is the type of backups? Plug-in ones, VM ones, both? I'm wondering cause the behavior and scenarios to test might differ a bit.

Thanks!

[RubinCompServ]

The other option would be to manually move backups to another extent, but that's really only feasible with a block based repository. I don't think that would even be possible with a S3 repo.

I don't have any experience with multiple buckets in the Capacity Tier, but it looks like I have the option to place my Capacity Tier S3 extent in Maintenance Mode and then Evacuate Backups. Would that move all of the backups from that bucket to the other S3 buckets? Or would it try to spread everything back over all of the extents? (Obviously, this conversation is based on the assumption that no immutability is in play here.)

[veremin]

We will confirm this scenario as well, just to ensure 100% accuracy, and I will update on this thread with the outcome. Thanks!

[veremin]

@RubinCompServ During evacuation, capacity extents are scheduled the same way as when offloading a new chain. If there are no limits set on repository, each backup chain will be placed on an extent with fewer chains. If limits are set (or in the case of S3-integrated repositories that report free/used space to us), we will select an extent with the maximum available free space.

Thanks!

[RubinCompServ]

@veremin

So would this be a viable way of migrating from one S3 provider to another: place both buckets in the capacity tier, put one in maintenance mode, and evacuate the backups? This would move all of the S3 data to the other bucket, which could be on the other provider, correct?

[Gostev]

Right, smooth migrations between different object storage providers was basically the one and only use case for adding the Seal functionality.

However, we believe that most customers will choose to instead just keep existing backups in place on a Sealed extent and let them expire on the "old" provider. This way you don't have to move TBs of backups around, which can take a lot of time and - as any data management operations - comes with an added risk of data loss/corruption. Plus, with some object storage this means incurring significant egress charges.