V12 SOBR Rebalance

[mkretzer]

I have a few questions about SOBR rebalance. As far as i understand SOBR rebalancing is the only way to balance space usage between SOBR extends. All other ways to rebalance without losing block clone benefits require moving between different SOBR or Jobs if I am correct.
Questions:
- Can we cancel the rebalancing? If not, it would be a disaster if suddenly 1,5 PB of data is being rebalanced on a very unbalanced SOBR (it would take days and for all that time backups are not possible)?
- How smart is the rebalancing?
o Will it take the smallest backup chains first?
o Will it concentrate on that chain, or will it do many chains at once?
- The only way I found to steer the procedure is to put some extends in maintenance more beforehand. Will putting some extends in maintenance mode and then out of maintenance mode after the process started help keep at least some backup jobs working?

Markus

[HannesK]

Hello Markus,
there is a reason why that functionality is hidden behind CTRL

We know, that we need to improve and I recommend not using it in your scenario (I added you to the existing feature request after you posted in the other V13 thread).

EDIT: [No, it cannot be cancelled in V12.] -> it can be cancelled. See below

Best regards,
Hannes

[mkretzer]

Hello Hannes,

ok - is there any other way to move files without loosing block cloning in a SOBR?

Markus

[HannesK]

Hello Markus,
not really besides of really ugly hacks (moving it to another repository and moving it back)

Best regards,
Hannes

[mdiver]

I also already though about how to cancel the procedure once being identified to run for several weeks e.g.

What would happen if VBR or the repo server(s) are rebooted while the procedure is running?
Will the backups be left in unconsistent state?
I would assume the process to be resilient against crashed - just like backups being rolled back after a crash during runtime.

Thanks
Mike

[veremin]

Actually you can cancel the SOBR rebalancing: find its session and click stop. Thanks!

[mkretzer]

@veremin nice - what happens when we start again? Will it pick new backups or continue the old ones?

[Gostev]

It is safe to guess we will create a new rebalance plan, and it will be slightly different as the SOBR situation will have changed somewhat due to the previously moved and newly created backups.

[mkretzer]

Ok i hope there will be a better way to do this - maybe even before V13

[mdiver]

I can confirm that the rebalancing can be stopped. I just tried it within a medium sized environment. Takes a few minutes as it finishes the VMs currently in process. Also the process seems to obey the limit for repo parallelity.
I've let it run to ~50% (112 VMs were identified to be copied) and stopped it. Once started again afterwards some 46 VMs were identified for the next run.

Though one thing does not make sense to men: An almost full extent had to be cleared of backups, but the rebalance even filled it up further (0,5TB free -> 0,1TB free). Shouldn't I expect the most filled up extent to get a relief?
Also the process complained about "No scale-out repository extents have sufficient disk space to store the backup file.". But two of three extents had >30TB free. Only one was almost full.
I would assume this error during rebalancing at least doesn't break anything.

Maybe someone can comment on the logic behind the rebalancing?

[HannesK]

sounds more like a bug than a feature... do you maybe have a support case for that behavior?

[tyler.jurgens]

A not so elegant way of rebalancing is to force a new active full backup for your VMs. The SOBR, if set up with per-VM backups, will then find a SOBR with free space to house that new VM backup.

Its not elegant, and I daresay a bit gross given the additional space requirements of doing that new active full, but it would work. Given time to work through the retention policy would allow the old extent to slowly clear while the new extent would take the new heavy lifting. Not always possible of course and you're at the whims of the SOBR extent targeting algorithms, which as I understand them don't always target the extent with the most free space. One could run this for a few VMs and then once you're happier with the results, perform either per VM balancing, or try the SOBR rebalance feature to minimize its run time.

SOBR balancing and migrations are why I've advocated for a 'soft seal' of sort for SOBR extents. Let any incremental backups target that extent, but any new VM backups/active full backups would never be allowed. One could soft seal the loaded extents and over time see backups drain stop to other extents.

[mkretzer]

@tjurgens-s2d Is that a new feature of V12? At least for V11 it does not select a new extend if there is at least some space free on the one with the previous chain!

[mdiver]

To my knowledge this functionality was there since the early days of the SOBR.
https://helpcenter.veeam.com/archive/ba ... ement.html
If VBR "thinks" the VBK would not fit (synthetic), it switched to another extent. This could even be fine-tuned with a RegKey: https://community.veeam.com/discussion- ... cluded-531 when fast-cloning methods are in place.

[mkretzer]

Any news if there is some new rebalancing option is planned for 12a?

[tyler.jurgens]

@mkretzer - you have to seal the extent and make sure your SOBR is setup to create a new full backup if the extent is unavailable. If you have that check box set in your SOBR properties, when you seal or place an extent into maintenance mode, any job that has a backup chain on that extent will automatically target a different extent in your SOBR. I migrated off 6 different ReFS extents onto different XFS extents this way.


Keep in mind that also means if you have an extent offline for any other reason and a backup job starts when that extent is offline, you will start a new active full backup elsewhere.

If you don't have the extent sealed, any new backups may target that extent. I'd like to be able to have extents still function for incremental backups/synthetic fulls, but never ingest brand new backups - but that feature doesn't exist.

You can of course use the powershell command to move any backups for an individual tenant from one extent to another. Its a little less intrusive than SOBR balancing, as it only disables that one tenant during the process.

[mkretzer]

Thats just inacceptable for our case - our Repos are up to a PB big, our individual backups files up to 40 TB. Doing this just because a repo gets "a bit too full" is extremly inefficient, especially since with new active full backup is existing on two extends for the retention time. Also, the current mode of rebalacing when an extend is full (non-fast cloned synthetic) leads to a very bad situation sometimes where Veeam moves to many backups to a new extend and 3 weeks later that extend is 99 % filled. With every such switch total backup capacity we can hold decreases as we use block cloning each time. In the worst case the only way to "fix" this is to reduce retention and manually move some backups (but not all).

[HannesK]

Markus, the request is clear and makes sense. There won't be anything new around online-rebalance in 12a though. We will look into this topic after 12a.

[mkretzer]

Hello Hannes,
6 months later - is there hope for online backup migration in V13? Especially rebalancing and emptying extends while no synthetics are running...

[HannesK]

Hello,
unfortunately, there is only little hope for V13, but let's see.

Best regards,
Hannes