[Feature Request] Object Storage Migration (Migrate Bucket A to Bucket B)

[kevlahau]

This is our experience—keep in mind that we all recognize that Blob is not S3 and that many S3 implementations are not alike.

We migrated a 200TB bucket from Wasabi to our Minio, and it worked correctly. The migration took a weekend because Wasabi limits the number of concurrent connections per access key. We achieved an incoming traffic of 14Gbps, limited by our backend.

To do this, we made sure that the bucket had the same name in both the source and destination to maintain the same paths at the Veeam Office 365 level. We used a migration tool that copies the object metadata and can handle hundres threads.

Obviously, while the data is being copied, you must remove it from the source VBO to ensure there is no activity.

Once all the objects have been copied, you create a new storage and perform a resync. When the first task finishes, you will see all the old points.

This method is not officially supported, so I recommend testing it with your own tenant and with just a few GBs.

What was the migration tool that you used??

Tks Kevlahau

[edh]

Flexify IO, it works like a charm in our case

[LukasK]

Hi,
is there an update to the original feature request?
An official way would be very interesting for us, as we cannot scale our S3 systems infinitely and offload migrations to other data center areas are very likely.

BR
Lukas

[Polina]

Hi Lukas,

It's on the roadmap, but with no updates at this point.

[Bjoern_ch]

@Polina
it would be nice if you could implement it without the need of a rescan like it is today. Migrating buckets is easy but the rescan of the new bucket (especially SP/Temas/OD) often takes a lot of time and let us miss SLA`s...

[DanielJ]

"On the roadmap" has been known to mean several years or never. We need to have a solution for this now. Flexify.io was suggested. Any other suggestions?

[Bjoern_ch]

@DanielJ
If it is just about migrating your data to a different bucket, there are plenty of solutions available for this.
We have around 2.5PB in 300 buckets and most of the data was migrated at some point to a different/new on-prem S3 storage system.
We use Rsync for this. Very easy and reliable.
But after migrating the data you need to create a new repo (with your new bucket) and therefore the data needs to be rescanned.