[Feature Request] - Object Storage Migration (Migrate Bucket A to Bucket B)

[DE&C]

We need a solution to migrate data between buckets.


Situation
• Object Storage is used for the backup job.

Challenge
• The bucket used by the backup job must be migrated to a new bucket.

Veeam has no way to migrate from object storage to another object storage. This is not supported by Veeam (https://www.veeam.com/kb3067). And it doesn’t work with a simple “copy from bucket a to bucket b”, because the IDs of the object change.


Feature request
• Build in solution, that allows to migrate VB365 backup data from object storage to another object storage.
• This should work for the primary backup job and a copy job.
• On a copy job it must work with object-lock enabled (if enabled on the source and destination bucket)


Examples
• Replace onprem storage (vendor A) with another vendor (B).
• Replace onprem storage (vendor A) with another version of the product / or new product that needs a new cluster (still vendor A, but now it is a new system)
• Migrate between cloud providers (from provider A to provider B)


Optional request: bulk deletion (without object-lock)
• Option to activate bulk deletion of the data in the old bucket after everything is migrated.
• Option to set how many objects should be deleted at the same time (100, 1000 etc.) to make sure the object storage can handle it.
• Of course this is only for buckets without object-lock

[JWH@VO]

can you not just use a tool to copy a full S3 bucket to a new bucket ?
or from the first main Veeam folder with all the repository data in there ? to a new S3 location.

JW Heins

[DE&C]

@Heins: No, as described in the Feature request, Veeam uses the IDs of the objects (therefore it doesn't work to just copy objects, because the IDs will change).

[Gostev]

Hmm, but why would a copying tool rename objects instead of just copying them?

[DE&C]

@Anton:
I mixed up a few things from my VBR feature request - sorry

VB365: It looks like it VB365 uses "name" not "ID" for the backup job. This would make sense, because no version ID is used. My colleague crosschecked the logs from the object storage and confirms this.

Problem (why I opened the feature request): We tested the migration multiple times with a copy from a bucket A to a bucket B. Even from a Storage-1 "bucket-1" to a storage-B "bucket-1" to keep the bucket name the same. On the point where we synchronize the repository we have always the error "unrecognized Guid format". Therefore we expect, that Veeam has something hard-coded or is using something, that changes after the copy (initially I thought the ID like in VBR, but of course, that makes no sense if versioning is deactivated). Therefore I thought that this is a Veeam limitation.

Can you or someone from product management please clarify: Should a copy with a 3rd party tool (s3cmd, cyberduck etc.) work for a bucket migration?
- if yes: I will open a veeam case and edit the feature request
- if no: I will change the background information on the feature request

sorry for the confusion and thanks in advance for your inputs

[Polina]

Hi William,

We're currently looking into some of the possible cross-bucket migration scenarios, however, at the moment officially there's no supported/reliable way of doing it. In some cases, we see that the repository structure was changed after the migration which breaks the compatibility.

[DE&C]

@Polina: Thank you very much for your reply.

Does this mean, that in general it should work? No limitation if the bucket is named differently (new bucket) as long as the structure is the same?

This information would already help a lot

[Polina]

The bucket name doesn't matter, but the entire structure with all objects and their properties must be identical to the initial repository.
If you want to run any migration tests, note that it's safer to remove the repository from the VB365 console before starting any copy/migration to a new bucket.

[DE&C]

@Polina: Thank you for the reply and your input. I will try a new migration next week with different tools.

[chrisflyckelen]

+1

A customer stores over 600 tb in an Azure storage account and would like to migrate to Wasabi due to a massive cost reduction.
And we had several customer scenarios where aging out the data was the only way, but this results in higher costs as well.

Thanks in advance,
Christian

[cdhanson88]

+1

This should be a supported feature given Veeams stance on mobility. Businesses need to be able to choose a storage provider without being locked in.

[DanielJ]

+1. Obvious and required functionality.

[fklein]

+1. (At least moving from one S3 to another S3)

[sandsturm]

+1 (copy from S3 bucket to another S3 bucket)

[YouGotServered]

+1 - We have our VBR bucket and VBMS365 buckets in one Wasabi tenant. I am in a situation where we have (don't ask) "accidental long-term object locking" for years on a VBR bucket. We essentially have to delete our entire account and start fresh since the buckets can't be deleted due to immutability (by design). Since we can't migrate our MS365 data cleanly to another bucket in another account, we have to re-upload from scratch to a new account and let the old one age out which is very expensive for us.

[MichaB]

Was any solution\workaround found for this?

[Mildur]

Hi Micha

Welcome to the forum.
It's still not supported to migrate data to another object storage.
Could you please describe your own use case to us why you need to migrate data?

Best,
Fabian

[MichaB]

Hello Fabian,

Thank you, let me clarify,
We have a S3 Compatible bucket on our own on-prem storage with a multiple Object Storage repositories, this storage and bucket is at full capacity and cannot be extended at all. We have another S3 Compatible bucket on a different free storage and would like to move some of the Object Storage repositories there without losing any historical backups.

[Bjoern_ch]

Hi,
some time ago (when there was no immutability yet) we tested an exit strategy for our buckets.
Migrating Azure Blobs to Wasabi S3 worked totally fine. We used flexify.io for this. Jobs ran successful afterwards again and restore of all data was also successful. Did not notice any issues.
Best regards,
Bjoern

[jasonede]

We're interested in this too to move between providers for better compatibility and to get rid of versioning warnings on all our backup jobs in VBO.
We need to move the data that has multi-year retention and don't want to be paying for storage twice for those years.
What tools have people used successfully to move between providers? (I can see flexify.io from above post) I've seen that can't just download/upload the data using s3cmd and the like.