Proper way to seed S3 storage

[NorthGuard]

We have an offsite datacenter, running minio with S3 buckets.

I have a couple of clients with enough data that trying to perform the initial backups would literally take weeks.

What is the proper way to seed backups for offsite storage? Can I just backup copy to a USB drive, connect the USB to our datacenter, copy into a bucket, then map the backups for the client?

[Mildur]

Hi Northguard

The data format on a USB drive repository and object storage are different.
- USB drive repository: backup files (vbk, vib)
- chunks of restore points as objects

Seeding could work, if you have another object storage at your clients location.
1) Create a backup copy job to this local object storage.
2) Transfer the object storage to your datacenter
3) Copy the objects 1:1 to the new bucket. Same structure most be preserved
4) Connect the client to the new bucket and run a rescan
5) Map the copy job with the seeded backup data.

Please note, this process will not work with immutable S3 buckets (versioning, object lock).
Also I recommend to run the process in a test run before using it on all customers.

Best,
Fabian

[NorthGuard]

What is the process for immutable S3 buckets as that is what we are using.

[EWMarco]

Unfortunately I do not think you'll manage to get around the process taking weeks.

We had tried incorporating S3 in existing backups. Turning on immediate offloading and moving after X days at the same time and we gave up on the effort after several months (!).

The only viable way IMO is to start offloading all the new backups immediately and when your initial retention policy is reached, activate move (if that is desired). Otherwise offloading will start at the oldest available restore points and as such will start pruning relatively immediately which causes deletes. Those will be the bane of your existence.

Just as an aside, did you configure 4MB block size and did you remember to keep the buckets small? I don't know if it'll be the same for minio but from what I know both NetApp and Cloudian do very badly with larger bucket sizes (we keep ours at 100TB now). We are also now switching from 1MB to 4MB because the amount of small objects at 1MB was killing us.

[ratkinsonuk]

For info, Amazon offer a service called SnowBall that allows customers to send hard drives to be loaded onto their storage. That may be a better option if you have TB's of data.

[RubinCompServ]

Just as an aside, did you configure 4MB block size and did you remember to keep the buckets small? I don't know if it'll be the same for minio but from what I know both NetApp and Cloudian do very badly with larger bucket sizes (we keep ours at 100TB now). We are also now switching from 1MB to 4MB because the amount of small objects at 1MB was killing us.

Can you provide more info about this? Thanks!

[Mildur]

Veeam writes small unique objects (Default Setting 1MB block size) instead of a big backup file to object storage. On-Premise Object Storage appliances normally have a limit on how many objects a single bucket can handle without performance issues.

Therefore it's important to follow the object storage vendors recommendation for Veeam Backup & Replication. In most cases you have a number of max supported objects per bucket. To lower the amount of objects written by Veeam for the same restore point, the block size of Veaam's backup files can be changed from 1MB to 4MB.

The block size can be configured in the job settings: Storage optimization

Please reach out to your object storage vendor if you want to know their recommendation. Every vendor should have a guide or kb page where they provide the recommendation especially for their appliance.

Best,
Fabian

[ratkinsonuk]

Fabian, what do you think to this...

"Amazon S3 adds 8 KB of storage for the name of the object + 32 KB of storage for index and related metadata"
(https://www.revolgy.com/insights/blog/h ... -amazon-s3)

Given the millions of files Veeam stores per backup job, this becomes very relevant.

Cheers, Rob

[ratkinsonuk]

Actually, ignore that. It sounds large, but it's like 100Mb which is nothing compared to the Tb's of data storage.

Rob.

[EWMarco]

Can you provide more info about this? Thanks!

I'm not sure how... The gist is that NetApp's object management DB suffers if you have many small objects. Deletion is also slow. We have a 4.5 PB S3 backing our environment and at 1MB we reached over 2.5 billion objects and performance was atrocious. Large buckets led tu enumeration issues. We could no longer edit the buckets in GUI due to timeouts.

[rk@rnt]

For info, you could also use Higabon Mobile Data Safe to transfer large amounts of Data physically and securely from one location to the other.

[sfirmes]

You should also check with the object storage vendor to see if they offer any native tools to seed data and if it has been tested with VBR. Unfortunately, I am unaware of MinIO offering such a tool at this time.

Thanks

Steve