Adding object storage for existing backups

[OzFenric]

Hi, this is just a clarification query to confirm my understanding is correct.

1. We can't add object storage to an existing Backup Repository. It has to be provisioned when you create the Backup Repository.

2. Existing data in a Backup Repository without object storage can be migrated (using Move-VBOEntityData, KB3067) to a Backup Repository with Object Storage. The full backup data will be stored in the Backup Repository until it gets offloaded to the Object Storage. This process might take significant time if there is a lot of backup data.

3. We cannot migrate directly from one VBO instance where Backup Repositories do not use Object Storage, to another instance with Backup Repositories extended with Object Storage. Moving a customer to a different VBO server is only supported like for like - with Object Storage to Object Storage, or without-to-without.

4. The best solution would be to migrate data in the existing instance to a repository with object storage, then migrate that object storage to the new instance.

Thanks in advance

[DanielJ]

I would say that the "best" solution - if you can afford the time and storage - is to set up a new repository with object storage, and new jobs pointing to that repository. Disable the old jobs and let them age out. The full backup will take some time but will most probably be faster than migrating data, and you will eliminate the risk of various possible migration problems. This is what we ended up doing in our current move to S3 storage. Of course, if you have long retention times it might not be viable to let it age out. We had one year at the most.

[sumeet]

I did run a migrate for our client during the Christmas break.
The block storage repository was 21 TB in size and it took 14 days to migrate the data to the new object storage repository.

I had to change the script in KB3067 and add sleep in the loop as in our previous migrations we observed that sometimes after Move-VBOEntityData forks off a job, it takes a while (mostly when the CPU is heavily loaded with existing move jobs), for the status to change as Running, this causes the script to fork off many more jobs, causing the server to become unresponsive.

Also, since the remaining jobs were on separate repo, I wanted them to run their daily backups, hence had to modify the script to not disable all the jobs and manually disable the job whose repo was being migrated. This required me to limit the number of threads that this migration could use (instead of using half the available threads).

We have lot more data to be migrated to object storage, slowly getting there, but the benefits of getting rid of JetDB repositories is worth.

[OzFenric]

Thanks Sumeet. This is the way we're heading also and your experience is valuable. We'll be migrating a full repository with a significant amount of data.

I have one more question: how do you detect the progress of the offloading? Where can you get information on how much data has been uploaded to object storage beyond watching I/O on the destination account?

Thanks in advance
Dan

[sumeet]

Hi Dan,

It is difficult to track the progress, relying on the objects that have been migrated does not help because each object has varying amount of data. So we simply monitor the storage I/O on the destination. Also, what we observed is that the objects that have few huge files migrate faster, as compared to objects which have thousand of tiny files. You can generate storage consumption report from Veeam UI and it will show the amount of data the has been pulled out of block storage repository and the amount of data dumped on object storage.

Please be aware that this is not a copy operation. The move command migrates the data from block to object repository. So for worst case scenario, I take a snapshot of the Virtual machine prior to starting the migration. With this, I can revert back to the snapshot and I have all the data. Yes we understand the risk that reverting the snapshot, will cause us to loose the backup data of rest of the jobs that have been running. Since our retention period is 7 years, we believe loosing few days of backup data is fine as compared to last few years. Also due to snapshot, there is some minor performance impact on the IO read operations (in our case VMware is the hypervisor).

The benefit of this move command is that it can be run multiple times on the same repository. So it will not start over as all the data that is successfully migrated no longer exists in the old block repository. And I have had to do this, because in my last migration, the server did run of resources, hence I had to reboot the server and start the script execution.

Also note that post the migration, the total size of your object storage repository might be less than the block repository. Because of few factors - one, there is better compression on object storage. Two - JetDB does not shrink when data is removed.

Hope this helps.

Thanks,
-Sumeet.

[Mike Resseler]

Hi All,

Since this happened to be a topic that I discussed last week with some architects... Are you using S3-compatible on-premises that has a certain size limit? And what would be a good solution for you guys (like extending a repository with a new folder / bucket or maybe something different?

We had a lot of open questions, ideas, but most of those were very futuristic .

[sumeet]

Hey Mike,

Yes we are using S3 compatible on-premise object storage.
When we started out few years back with M365, we did not have this on-premise S3 capability. As we grew and the amount of data that had to be handled, we realized that switching to S3 not only makes VBM perform better, but also the amount of resources (CPU/Memory) that a VBM/proxy server requires is less. Hence we got on-premise S3 capability.
All our new M365 clients are on S3 storage, but old clients need to be migrated to S3 and this is where we have to go through this exercise.

This last weekend we migrated one of our clients Teams object repo to object storage. The repo was few TBs in size, but had to be done in short notice as we realized that after switching to the new Teams Export API, if we perform this repo migration, the full sync that is required post repo migration will add on to the cost involved with the new Teams API. Hence the repo migration had to be done prior to the Teams API switch. Now all our clients Teams repo is on object storage. Calling out the scenario as different cases we have to handle.

Some of our clients block repositories are 50+ TB in size. Migrating this much amount of data to S3 will require us to disable the job for few weeks (4+ weeks). Even with adding more CPU and memory, we have observed that the migration have very minimal benefit, most scenarios JetDB causes the server to run out of memory.

Having mentioned all this, if you ask me for a good solution, I will request you for something like Backup Copy option. Have a option like Migrate repository from block to object storage. The primary benefit of the option has to be that the daily job can continue to run, while the repository continues to migrate from block to object storage. Once the migration completes, I have an option to compare both the repositories are same and then switch the job to point to the new object storage repo.

Thanks,
-Sumeet.

[sumeet]

Hey Mike,

Watched VBR v12 release yesterday - fantastic work - congratulations to your team.

The backup move option is what we need in VBM, that's what I explained in my last reply of this thread.

Thanks,
-Sumeet.

[Mike Resseler]

Understood sumeet, however, it is not the same scenario as what you have. Now, we are working on something that would allow the migration from Jet DB to OSR while the jobs continue to run (which means you won't miss the SLA of customers).

[sumeet]

Hi Mike,

Thanks.
Looking forward to this.

Cheers,
-Sumeet.

[sumeet]

Hi Mike,

Hope you are well. Checking if you have any timelines on which this migration feature can be made available to us? I understand that giving a definite time for something that you are working on will not be possible, but just want to get an estimate if it can/may be available in 6 months or a year or more.

With some of clients on drive repository (Jet DB) which is more than 60+ TB in size (we have separate repositories for each backup of mailboxes, OD, SP, archive and Teams. This 60+ TB repositories are such individual ones, not all combined together), managing them is getting difficult (proxy resources issues cpu + memory, restore explorers fail to open when backups are running or timeout). Hence with some timelines, we can plan accordingly.
Also, since you have mentioned that something is being worked upon, we have stopped all manual migrations, as mentioned in my earlier posts.

Thanks,
-Sumeet.

[Polina]

Hi Sumeet,

We expect to deliver it by the end of the year if things go well.

[sumeet]

Hi Polina,

Thanks for the update. Good to know.
This feature will save us lot of time.

Regards,
-Sumeet.

[sumeet]

Hi Mike, Polina,
Hope you are well.
I see 7.1 released today/yesterday - congratulations.

Checking on this requirement of migration from block to object storage based repository. Being so close to the end of year, by any chance we get this?
Year end is a good time for us to perform such migrations.

Thanks,
-Sumeet.

[Polina]

Hi Sumeet,

We don't plan to release any new functionality this year; stay tuned for updates in '24.

Thanks

[sumeet]

Hi Polina,

It would have been good to have this now, as it gives us few weeks around Christmas to let this migration run.

No worries, will wait and hopefully have this ready in 24.

If this is not going to get implemented, please let us know, as this will help us plan and continue our existing process to perform the migration.

Thanks,
-Sumeet.

[sumeet]

Hi Polina,

Any updates on this new functionality.
As mentioned earlier, it is important that we have this capability as it will help us plan the growth of our old clients that are on block storage.

Thanks,
-Sumeet.

[Mildur]

Hello Sumeet

There was no new release since your last comment. The mentioned feature (migrate backups while jobs are running) is still scheduled for our upcoming release (v8). But no release date we can provide today.

Best,
Fabian

PS:
I see you are managing clients. Please register yourself in our service provider user group if you are a service provider. This gives you access to our hidden service provider sub forums where service provider products are discussed: Apply for the Cloud and Service Providers user group

[sumeet]

Hello Fabian,

Yes aware that no new release since last comment. What I wanted to know is that the mentioned feature (migrate backups while jobs are running) is still scheduled and not dropped off the upcoming release.
Managing some of our clients which are 100+ TB of backup block data (on JetDB) is slowly getting difficult. Hence want to know that this feature will be made available.

The reason I keep asking this question in few months time, is to know if for any reason you have dropped this mentioned feature, we get to know.

Thanks for pointing out the service provider user group. I have registered today.

Thanks,
-Sumeet.

[Polina]

Hi Sumeet,

It's still expected in v8, not dropped, not overlooked.

[sumeet]

Thanks, appreciate you and your teams help. Will wait.

[sumeet]

Hi Polina, Fabian,

I apologies as I keep going back to this one. But it is important for us to know this as some of our block repositories have grown to an extent that it is not easy to manage.
So will help to get some rough timelines on when can we expect this feature of migrating from block to object storage.
This will help us in planning our current storage growth and be better prepared when this feature is available.

Thanks,
-Sumeet.