No more mixing immutable & non-immutable extents in a SOBR

[RubinCompServ]

First off, I fully understand why putting immutable and non-immutable extents in a SOBR is a Bad Idea. I realize that there is no way to guarantee that data gets written to the proper extent, and immutability will go haywire. I get it, and I support it.

That said, VBR v11 allowed the different extents to be in the same SOBR, and we took advantage of that. All of our customers get non-immutable storage as part of their base backup package. If, at some point in the future, they decide to upgrade to immutable storage, it was easy to build an immutable repo, add it as an extent to the customer's existing SOBR, seal the old extent (to allow the old backups to age out), and all future backups would be immutable. This was a simple method that was virtually invisible to the customer - they kept their existing repository in BEM, they continued to use their existing backups job, and - other than an additional Full backup - they didn't see any change in the Self-Service portal.

When we were forced to upgrade to v12.1, all of that broke: the upgrade literally failed if mixed extents are in the same SOBR, forcing me to pull the non-immutable extent out of the SOBR before I could upgrade (which I had to do because I had already upgraded BEM to v12 because of the security issue, and it turns out that BEM v12 can't actually fully manage older versions of VBR). So now I have a repo with ~50TB of non-immutable backup data on it and a SOBR containing another 250TB of data. The customer can no longer see their oldest data (GFS from 2023), because it doesn't live in one of the SOBR's extents.

I opened case #07273940 with support, and their answer was "we make it easy to move data from one repository to another" and, while that is very true, and very unhelpful, because it will take DAYS to move the 50TB of data to the immutable repository, which translates to days during which the customer won't get any backups, because the "easy method of moving backup data" also disables the backup job while it moves it. But that's an issue for Support to address.

My question here is: going forward, what is the recommended method for the above scenario? As near as I can tell, the only thing we can do is build a new SOBR with an immutable extent, publish a new repo to the customer, recreate all of their backup jobs to point to the new extent, and manually delete their old backups once they've exceeded their retention period.

[RubinCompServ]

Three weeks and no responses? Has nobody had to move a customer from a non-immutable repo to an immutable repo?

[IvanK]

Hi David,

It's not clear if this scenario is shared hosted VBR + EM self-service or if this involves Cloud Connect. I do see you mention "in the Self-Service portal" but still want to confirm it, since Cloud SOBR and regular SOBR are two different beasts to some degree.

If that's a shared hosted VBR + EM SSP, then it's also important to understand whether you are using VCD or not because the available toolkit would be slightly different from the tenant self-service perspective, depending on if it's vSphere or VCD.

The steps you describe though, sound valid, let's see if there is any way to optimize that or allow tenants see older unmapped backups. I reached out to you via email. If you'd like to discuss your case and show me the current state, feel free to reply there. Then we are going to share the results here on forums.

[RubinCompServ]

@IvanK,

...shared hosted VBR + EM self-service or if this involves Cloud Connect...understand whether you are using VCD or not...

This is not Cloud Connect related. This is for the Veeam Self-Service portal that is a component of the BEM that allows VCD tenants to log in using their VCD creds and manage their own backups. We have no customers with vSphere backups; all customers are VCD tenants.

I reached out to you via email.

Can you do so again please? I don't see anything from you. Thanks!

[Henrik.Grevelund]

Hi David

I dont understand why this topic isn't gloving with responses. Being a service provider and a consultant i have the exact same issue as you.
Every customer who didn't upgrade to immutable repo's in version 11 is going to have this issue.

I have customer with 9 extents running on windows servers, which we where planning to migrate to 7 new linux boxes. 4 PB in total.
Expecting to add them to the SOBR and seal the old over time.
The SOBR is also using copy to a capacity tier. So writing everything to a new SOBR will also start a new offload

I really hope Veeam can change this ?
So count this as a +1 on this feature request

[IvanK]

Hi Henrik,

Mixing immutable and non-immutable performance extents was disallowed to avoid user misconfigurations and misunderstandings.

As the SOBR placement logic (for both DBOS and block-based performance tier) operates on a per-machine basis, it was possible to run into a situation where some machines within one job are placed on immutable extents, while others are on non-immutable extents. This was confusing and could lead users to believe their data was immutable when it was not.

Regarding possible workarounds, there are two options:
1. Remove non-immutable extents from SOBR add new immutable extents and trigger active full on the existing jobs, so it will be written on the new performance tier extents. According to the QA, this operation will preserve data on the capacity tier and allow continuing the existing capacity tier chains (and keep object re-use intact). Mind that those tests were done on v12.1.2 with upgraded chains.
2. Stay on v11 until the migration from non-immutable to immutable extents is completed, then upgrade.

David's situation is more complex due to the use of VCD self-service portal and the removal of original backup jobs, if I'm not mistaken.
However, it's still possible to re-create the job with the same VMs, point it to the same SOBR with a new performance tier and map it to the original backup chain on this SOBR (while the actual backups are present only on capacity tier of this SOBR). This way, a new active full will be written on the performance extent, while the capacity tier data will be re-used.
If jobs are created on VBR server directly (not via self-service portal), make sure to use the Set-VBRvCloudOrganizationJobMapping cmdlet to make job to VCD tenant and corresponding backups visible to the customer.

[Henrik.Grevelund]

Hi Ivan,

I did a small test trying your optin 1.
In the latest version i was able to delete all the old extents, and then i was allowed to add an immutable extent. I wasn't able to test the offload.

I found that all the old backup where listed as missing.
Since we use the offload as copy to ensure 2 copies of the backup, then using this option would remove the option to restore fromt the local copy, since Veeam doesn't know where it is.

Beside that, this will cause that we have to start a FULL backup of the entire environement on the same day.

So please create a way to get this mix-setup back, so we can use a controlled rollover.

[RubinCompServ]

@Henrik,

First off: I'm with you. I think that Veeam should allow you to create the hybrid SOBR that existed beforehand. I think they could have solved their problem by sealing every non-immutable extent in the SOBR, which will allow for an easy migration in the future, as well (especially since v11 is no longer supported). However, as I was not, am not, and (likely) will not be part of the Development team, I can only surmise that there were valid reasons for not going that route.

That said, if you still have the non-immutable repositories configured within Veeam (even if they aren't in the SOBR), Veeam should still be aware of the local copy of the backups; you can find them in the "Disk (Imported)" section of the left-side navigation pane. It's not nearly as pretty, but it does work.

As for having to start a new Full backup for the entire environment, keep in mind that that would have happened anyway, as Veeam will automatically do an Active Full if the current Full is located on a Sealed extent.

[Henrik.Grevelund]

Hi David,

My main concern is the offload off 3 PB. In the old links i got from support is was about creating a new repo, with the same extents, but converted to immutable. That will cause a new upload.
Sealing extents one bye one, could work over time, but not taking a Full of everything.

Just got a new link, with a page that was updated 7/18/2024, seems like a new way that might work. Have to test it.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Support case # 07313689

[RubinCompServ]

Henrik,

I didn't even consider the offload cost to S3. Unfortunately in my case, my non-immutable repos are Windows machines and I'm having a hard time just getting the backups over to a Linux machine for the Performance tier.

[Henrik.Grevelund]

Hi,

I have succesfully converted the non-immutable to an immutable setup keping the offload intact, using the new method desribed here:
https://helpcenter.veeam.com/docs/backu ... ml?ver=120
Using Veeam version 12.1.2.172.

In the test i created 1 standard linux repository, and a S3 compatible repository.
Created a SOBR adding both, the capacity tier in copy mode.
Created 2 backup jobs, wrote a couple of backups.

followed the precedure created by Veeam.
I did not remove the offloaded data from configuration, only local repo.
I created the new immutable repositories using the same path as the original(not mentioned in the procedure)

After the procedure, i started the backup, and it ran an incremental backup.
The offload job did go trough all the restore points, but didn't transfer any data from the old restore points.
So this might take a while in a production setup, but it should be OK.

[RubinCompServ]

That's great news Henrik, thanks for letting us know!