-
- Service Provider
- Posts: 108
- Liked: 8 times
- Joined: Jan 19, 2022 4:48 pm
- Contact:
Proper way to seed S3 storage
We have an offsite datacenter, running minio with S3 buckets.
I have a couple of clients with enough data that trying to perform the initial backups would literally take weeks.
What is the proper way to seed backups for offsite storage? Can I just backup copy to a USB drive, connect the USB to our datacenter, copy into a bucket, then map the backups for the client?
I have a couple of clients with enough data that trying to perform the initial backups would literally take weeks.
What is the proper way to seed backups for offsite storage? Can I just backup copy to a USB drive, connect the USB to our datacenter, copy into a bucket, then map the backups for the client?
-
- Product Manager
- Posts: 10339
- Liked: 2759 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Proper way to seed S3 storage
Hi Northguard
The data format on a USB drive repository and object storage are different.
- USB drive repository: backup files (vbk, vib)
- chunks of restore points as objects
Seeding could work, if you have another object storage at your clients location.
1) Create a backup copy job to this local object storage.
2) Transfer the object storage to your datacenter
3) Copy the objects 1:1 to the new bucket. Same structure most be preserved
4) Connect the client to the new bucket and run a rescan
5) Map the copy job with the seeded backup data.
Please note, this process will not work with immutable S3 buckets (versioning, object lock).
Also I recommend to run the process in a test run before using it on all customers.
Best,
Fabian
The data format on a USB drive repository and object storage are different.
- USB drive repository: backup files (vbk, vib)
- chunks of restore points as objects
Seeding could work, if you have another object storage at your clients location.
1) Create a backup copy job to this local object storage.
2) Transfer the object storage to your datacenter
3) Copy the objects 1:1 to the new bucket. Same structure most be preserved
4) Connect the client to the new bucket and run a rescan
5) Map the copy job with the seeded backup data.
Please note, this process will not work with immutable S3 buckets (versioning, object lock).
Also I recommend to run the process in a test run before using it on all customers.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Service Provider
- Posts: 108
- Liked: 8 times
- Joined: Jan 19, 2022 4:48 pm
- Contact:
Re: Proper way to seed S3 storage
What is the process for immutable S3 buckets as that is what we are using.
-
- Service Provider
- Posts: 48
- Liked: 7 times
- Joined: Feb 20, 2023 9:28 am
- Full Name: Marco Glavas
- Contact:
Re: Proper way to seed S3 storage
Unfortunately I do not think you'll manage to get around the process taking weeks.
We had tried incorporating S3 in existing backups. Turning on immediate offloading and moving after X days at the same time and we gave up on the effort after several months (!).
The only viable way IMO is to start offloading all the new backups immediately and when your initial retention policy is reached, activate move (if that is desired). Otherwise offloading will start at the oldest available restore points and as such will start pruning relatively immediately which causes deletes. Those will be the bane of your existence.
Just as an aside, did you configure 4MB block size and did you remember to keep the buckets small? I don't know if it'll be the same for minio but from what I know both NetApp and Cloudian do very badly with larger bucket sizes (we keep ours at 100TB now). We are also now switching from 1MB to 4MB because the amount of small objects at 1MB was killing us.
We had tried incorporating S3 in existing backups. Turning on immediate offloading and moving after X days at the same time and we gave up on the effort after several months (!).
The only viable way IMO is to start offloading all the new backups immediately and when your initial retention policy is reached, activate move (if that is desired). Otherwise offloading will start at the oldest available restore points and as such will start pruning relatively immediately which causes deletes. Those will be the bane of your existence.
Just as an aside, did you configure 4MB block size and did you remember to keep the buckets small? I don't know if it'll be the same for minio but from what I know both NetApp and Cloudian do very badly with larger bucket sizes (we keep ours at 100TB now). We are also now switching from 1MB to 4MB because the amount of small objects at 1MB was killing us.
-
- Expert
- Posts: 126
- Liked: 18 times
- Joined: Dec 10, 2018 10:59 am
- Full Name: Robert Atkinson
- Contact:
Re: Proper way to seed S3 storage
For info, Amazon offer a service called SnowBall that allows customers to send hard drives to be loaded onto their storage. That may be a better option if you have TB's of data.
-
- Service Provider
- Posts: 360
- Liked: 89 times
- Joined: Mar 16, 2015 4:00 pm
- Full Name: David Rubin
- Contact:
Re: Proper way to seed S3 storage
Can you provide more info about this? Thanks!EWMarco wrote: ↑Sep 25, 2023 5:37 am Just as an aside, did you configure 4MB block size and did you remember to keep the buckets small? I don't know if it'll be the same for minio but from what I know both NetApp and Cloudian do very badly with larger bucket sizes (we keep ours at 100TB now). We are also now switching from 1MB to 4MB because the amount of small objects at 1MB was killing us.
-
- Product Manager
- Posts: 10339
- Liked: 2759 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Proper way to seed S3 storage
Veeam writes small unique objects (Default Setting 1MB block size) instead of a big backup file to object storage. On-Premise Object Storage appliances normally have a limit on how many objects a single bucket can handle without performance issues.
Therefore it's important to follow the object storage vendors recommendation for Veeam Backup & Replication. In most cases you have a number of max supported objects per bucket. To lower the amount of objects written by Veeam for the same restore point, the block size of Veaam's backup files can be changed from 1MB to 4MB.
The block size can be configured in the job settings: Storage optimization
Please reach out to your object storage vendor if you want to know their recommendation. Every vendor should have a guide or kb page where they provide the recommendation especially for their appliance.
Best,
Fabian
Therefore it's important to follow the object storage vendors recommendation for Veeam Backup & Replication. In most cases you have a number of max supported objects per bucket. To lower the amount of objects written by Veeam for the same restore point, the block size of Veaam's backup files can be changed from 1MB to 4MB.
The block size can be configured in the job settings: Storage optimization
Please reach out to your object storage vendor if you want to know their recommendation. Every vendor should have a guide or kb page where they provide the recommendation especially for their appliance.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Expert
- Posts: 126
- Liked: 18 times
- Joined: Dec 10, 2018 10:59 am
- Full Name: Robert Atkinson
- Contact:
Re: Proper way to seed S3 storage
Fabian, what do you think to this...
"Amazon S3 adds 8 KB of storage for the name of the object + 32 KB of storage for index and related metadata"
(https://www.revolgy.com/insights/blog/h ... -amazon-s3)
Given the millions of files Veeam stores per backup job, this becomes very relevant.
Cheers, Rob
"Amazon S3 adds 8 KB of storage for the name of the object + 32 KB of storage for index and related metadata"
(https://www.revolgy.com/insights/blog/h ... -amazon-s3)
Given the millions of files Veeam stores per backup job, this becomes very relevant.
Cheers, Rob
-
- Expert
- Posts: 126
- Liked: 18 times
- Joined: Dec 10, 2018 10:59 am
- Full Name: Robert Atkinson
- Contact:
Re: Proper way to seed S3 storage
Actually, ignore that. It sounds large, but it's like 100Mb which is nothing compared to the Tb's of data storage.
Rob.
Rob.
-
- Service Provider
- Posts: 48
- Liked: 7 times
- Joined: Feb 20, 2023 9:28 am
- Full Name: Marco Glavas
- Contact:
Re: Proper way to seed S3 storage
I'm not sure how... The gist is that NetApp's object management DB suffers if you have many small objects. Deletion is also slow. We have a 4.5 PB S3 backing our environment and at 1MB we reached over 2.5 billion objects and performance was atrocious. Large buckets led tu enumeration issues. We could no longer edit the buckets in GUI due to timeouts.
-
- Service Provider
- Posts: 2
- Liked: never
- Joined: Feb 07, 2024 8:16 am
- Full Name: Ridel K.
- Location: Germany
- Contact:
Re: Proper way to seed S3 storage
For info, you could also use Higabon Mobile Data Safe to transfer large amounts of Data physically and securely from one location to the other.
-
- Veeam Software
- Posts: 321
- Liked: 150 times
- Joined: Jul 24, 2018 8:38 pm
- Full Name: Stephen Firmes
- Contact:
Re: Proper way to seed S3 storage
You should also check with the object storage vendor to see if they offer any native tools to seed data and if it has been tested with VBR. Unfortunately, I am unaware of MinIO offering such a tool at this time.
Thanks
Steve
Thanks
Steve
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
Who is online
Users browsing this forum: No registered users and 5 guests