Veeam Immutable Off-Site SOBR almost full, what's the best way to add storage?

[sarnold]

I have a Veeam repository setup consisting of an on-site ReFS SOBR (2 extents) for our main backup jobs [100TB capacity total, 66TB free], and then an off-site immutable XFS SOBR (2 extents) at one of our locations [65TB capacity total, 4TB free]. The off-site location keeps 14 days of backups and then GFS backups weekly/monthly/yearly. This has worked for quite some time without issues, but recently, the off-site immutable Linux storage has been creeping up quickly in storage usage, even with fast clone (used by both our on-site and off-site repos). I thought that fast clone would have prevented storage from being an issue especially since we only keep 14 days and then the rest are GFS (which should be using spaceless synthetic fulls), but free space has suddenly quickly dropped on the off-site repo.

I do happen to have two more servers available with the same amount of storage as the two servers being used currently in the off-site repo, so I could install Linux on those two, configure them the same way as the first two extents, and add those 2 to the off-site SOBR, making it have a total of 4 extents. However, since the backups for all the VMs already exist on the first 2 extents, I can't see a way to rebalance the data across them, expecially since synthetic fast clone is in use and I use immutability (which also makes GFS backups immutable, as I learned today), so moving generally isn't possible I believe.

What's the best way to handle this? My initial thought was to maybe "seal" the first 2 off-site SOBR extents, forcing all backups to the 2 new extents (if I go this "adding servers" route), and then once I have enough history, remove and rebuild the second 2 extents and then add them back. But then I end up in the same situation where the other 2 systems now have all the backups and the first 2 won't receive any.

Open to ideas! This environment is running Veeam 12.0.0.1420 (P20230718).

[tyler.jurgens]

You can move backups between repositories in v12 without incurring space usage increases - however, if those files are immutable you won't be able to remove them from the LHR without getting in as root and removing the immutable flag within the LHR itself.

https://community.veeam.com/blogs-and-p ... ories-4003

The other way to rebalance the SOBR (after adding the new extents) is seal the old extents and run an Active Full Backup on your backup copy job. That will force Veeam to target the new extents with whatever job you have ran.

You should also dig into why you're getting such a large increase in usage. Are your incrementals larger than expected? XFS will give you incredible space savings, but it won't magically stop data growth in your VMs from causing data growth on your backup repositories.

[sarnold]

You can move backups between repositories in v12 without incurring space usage increases - however, if those files are immutable you won't be able to remove them from the LHR without getting in as root and removing the immutable flag within the LHR itself.

It seems like I can move between repositories, but not extents.

The other way to rebalance the SOBR (after adding the new extents) is seal the old extents and run an Active Full Backup on your backup copy job.

True, but then I end up with the same problem as I described in my original post where the second two servers now house all the backup copy job content and the first 2 won't get anything because the chains now exist on the second two servers for each backed up resource.

[sarnold]

You should also dig into why you're getting such a large increase in usage.

I'm now suspicious that XFS fast cloning has stopped working... I think that might be the issue. I just realized our synthetic full creation has been running for nearly 14 hours...and it should take more like 10 minutes. Has anyone seen this before? Any recommendations on how to fix this if this is indeed the case?