Comprehensive data protection for all workloads
Post Reply
mkretzer
Veeam Legend
Posts: 1144
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

V12 SOBR Rebalance

Post by mkretzer »

I have a few questions about SOBR rebalance. As far as i understand SOBR rebalancing is the only way to balance space usage between SOBR extends. All other ways to rebalance without losing block clone benefits require moving between different SOBR or Jobs if I am correct.
Questions:
- Can we cancel the rebalancing? If not, it would be a disaster if suddenly 1,5 PB of data is being rebalanced on a very unbalanced SOBR (it would take days and for all that time backups are not possible)?
- How smart is the rebalancing?
o Will it take the smallest backup chains first?
o Will it concentrate on that chain, or will it do many chains at once?
- The only way I found to steer the procedure is to put some extends in maintenance more beforehand. Will putting some extends in maintenance mode and then out of maintenance mode after the process started help keep at least some backup jobs working?

Markus
HannesK
Product Manager
Posts: 14314
Liked: 2887 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: V12 SOBR Rebalance

Post by HannesK »

Hello Markus,
there is a reason why that functionality is hidden behind CTRL :-)

We know, that we need to improve and I recommend not using it in your scenario (I added you to the existing feature request after you posted in the other V13 thread).

EDIT: [No, it cannot be cancelled in V12.] -> it can be cancelled. See below

Best regards,
Hannes
mkretzer
Veeam Legend
Posts: 1144
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12 SOBR Rebalance

Post by mkretzer »

Hello Hannes,

ok - is there any other way to move files without loosing block cloning in a SOBR?

Markus
HannesK
Product Manager
Posts: 14314
Liked: 2887 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: V12 SOBR Rebalance

Post by HannesK »

Hello Markus,
not really besides of really ugly hacks (moving it to another repository and moving it back)

Best regards,
Hannes
mdiver
Veeam Legend
Posts: 201
Liked: 33 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: V12 SOBR Rebalance

Post by mdiver » 1 person likes this post

I also already though about how to cancel the procedure once being identified to run for several weeks e.g.

What would happen if VBR or the repo server(s) are rebooted while the procedure is running?
Will the backups be left in unconsistent state?
I would assume the process to be resilient against crashed - just like backups being rolled back after a crash during runtime.

Thanks
Mike
veremin
Product Manager
Posts: 20282
Liked: 2257 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: V12 SOBR Rebalance

Post by veremin »

Actually you can cancel the SOBR rebalancing: find its session and click stop. Thanks!
mkretzer
Veeam Legend
Posts: 1144
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12 SOBR Rebalance

Post by mkretzer »

@veremin nice - what happens when we start again? Will it pick new backups or continue the old ones?
Gostev
Chief Product Officer
Posts: 31503
Liked: 6689 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V12 SOBR Rebalance

Post by Gostev »

It is safe to guess we will create a new rebalance plan, and it will be slightly different as the SOBR situation will have changed somewhat due to the previously moved and newly created backups.
mkretzer
Veeam Legend
Posts: 1144
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12 SOBR Rebalance

Post by mkretzer »

Ok i hope there will be a better way to do this - maybe even before V13 :-(
mdiver
Veeam Legend
Posts: 201
Liked: 33 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: V12 SOBR Rebalance

Post by mdiver » 1 person likes this post

I can confirm that the rebalancing can be stopped. I just tried it within a medium sized environment. Takes a few minutes as it finishes the VMs currently in process. Also the process seems to obey the limit for repo parallelity.
I've let it run to ~50% (112 VMs were identified to be copied) and stopped it. Once started again afterwards some 46 VMs were identified for the next run.

Though one thing does not make sense to men: An almost full extent had to be cleared of backups, but the rebalance even filled it up further (0,5TB free -> 0,1TB free). Shouldn't I expect the most filled up extent to get a relief?
Also the process complained about "No scale-out repository extents have sufficient disk space to store the backup file.". But two of three extents had >30TB free. Only one was almost full.
I would assume this error during rebalancing at least doesn't break anything.

Maybe someone can comment on the logic behind the rebalancing?
HannesK
Product Manager
Posts: 14314
Liked: 2887 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: V12 SOBR Rebalance

Post by HannesK »

sounds more like a bug than a feature... do you maybe have a support case for that behavior?
tyler.jurgens
Veeam Legend
Posts: 289
Liked: 128 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: V12 SOBR Rebalance

Post by tyler.jurgens » 1 person likes this post

A not so elegant way of rebalancing is to force a new active full backup for your VMs. The SOBR, if set up with per-VM backups, will then find a SOBR with free space to house that new VM backup.

Its not elegant, and I daresay a bit gross given the additional space requirements of doing that new active full, but it would work. Given time to work through the retention policy would allow the old extent to slowly clear while the new extent would take the new heavy lifting. Not always possible of course and you're at the whims of the SOBR extent targeting algorithms, which as I understand them don't always target the extent with the most free space. One could run this for a few VMs and then once you're happier with the results, perform either per VM balancing, or try the SOBR rebalance feature to minimize its run time.

SOBR balancing and migrations are why I've advocated for a 'soft seal' of sort for SOBR extents. Let any incremental backups target that extent, but any new VM backups/active full backups would never be allowed. One could soft seal the loaded extents and over time see backups drain stop to other extents.
Tyler Jurgens
Veeam Legend x2 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @tylerjurgens.bsky.social
mkretzer
Veeam Legend
Posts: 1144
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12 SOBR Rebalance

Post by mkretzer »

@tjurgens-s2d Is that a new feature of V12? At least for V11 it does not select a new extend if there is at least some space free on the one with the previous chain!
mdiver
Veeam Legend
Posts: 201
Liked: 33 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: V12 SOBR Rebalance

Post by mdiver »

To my knowledge this functionality was there since the early days of the SOBR.
https://helpcenter.veeam.com/archive/ba ... ement.html
If VBR "thinks" the VBK would not fit (synthetic), it switched to another extent. This could even be fine-tuned with a RegKey: https://community.veeam.com/discussion- ... cluded-531 when fast-cloning methods are in place.
mkretzer
Veeam Legend
Posts: 1144
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12 SOBR Rebalance

Post by mkretzer »

Any news if there is some new rebalancing option is planned for 12a?
tyler.jurgens
Veeam Legend
Posts: 289
Liked: 128 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: V12 SOBR Rebalance

Post by tyler.jurgens » 1 person likes this post

@mkretzer - you have to seal the extent and make sure your SOBR is setup to create a new full backup if the extent is unavailable. If you have that check box set in your SOBR properties, when you seal or place an extent into maintenance mode, any job that has a backup chain on that extent will automatically target a different extent in your SOBR. I migrated off 6 different ReFS extents onto different XFS extents this way.
Image

Keep in mind that also means if you have an extent offline for any other reason and a backup job starts when that extent is offline, you will start a new active full backup elsewhere.

If you don't have the extent sealed, any new backups may target that extent. I'd like to be able to have extents still function for incremental backups/synthetic fulls, but never ingest brand new backups - but that feature doesn't exist.

You can of course use the powershell command to move any backups for an individual tenant from one extent to another. Its a little less intrusive than SOBR balancing, as it only disables that one tenant during the process.
Tyler Jurgens
Veeam Legend x2 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @tylerjurgens.bsky.social
mkretzer
Veeam Legend
Posts: 1144
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12 SOBR Rebalance

Post by mkretzer » 2 people like this post

Thats just inacceptable for our case - our Repos are up to a PB big, our individual backups files up to 40 TB. Doing this just because a repo gets "a bit too full" is extremly inefficient, especially since with new active full backup is existing on two extends for the retention time. Also, the current mode of rebalacing when an extend is full (non-fast cloned synthetic) leads to a very bad situation sometimes where Veeam moves to many backups to a new extend and 3 weeks later that extend is 99 % filled. With every such switch total backup capacity we can hold decreases as we use block cloning each time. In the worst case the only way to "fix" this is to reduce retention and manually move some backups (but not all).
HannesK
Product Manager
Posts: 14314
Liked: 2887 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: V12 SOBR Rebalance

Post by HannesK » 1 person likes this post

Markus, the request is clear and makes sense. There won't be anything new around online-rebalance in 12a though. We will look into this topic after 12a.
mkretzer
Veeam Legend
Posts: 1144
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V12 SOBR Rebalance

Post by mkretzer »

Hello Hannes,
6 months later - is there hope for online backup migration in V13? Especially rebalancing and emptying extends while no synthetics are running...
HannesK
Product Manager
Posts: 14314
Liked: 2887 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: V12 SOBR Rebalance

Post by HannesK » 1 person likes this post

Hello,
unfortunately, there is only little hope for V13, but let's see.

Best regards,
Hannes
Post Reply

Who is online

Users browsing this forum: Google [Bot], Semrush [Bot] and 45 guests