SOBR Rebalance doesn't take extent size into account for free space calculations

[ChristineAlexa]

SOBR Rebalance seems to assume all extents are the same size, so that they all try to get the same free space IN TB vs the same free space as PERCENT off the extent size

Not sure if I have missed a feature setting or request at this point

I'm still on 12.0.x but I don't believe that has an effect on this.

I have a SOBR with 3 extents and they are all in performance mode. All are REFS, 64K with dedup, etc working properly, so simple file copies aren't options.
1 x 10TB, 8.7TB free
1 x 11TB, 8.9TB free
1 x 22 TB, 8.4TB free

Unfortunately, even after a rebalance the balance seems to optimize for giving the all the same amount of free SIZE (as seen above). Because of this, most of the disk load is on one extent instead of spread across then, as well as my data loss risk.

Is there a way to set the calculation to use % free space so that I end I have something roughly like the below so they are all proportionally filled the same.
1 x 10TB, 6TB free
1 x 11TB, 6.65TB free
1 x 22TB, 13.30 TB free


Thanks in advance
Christine

[RobMiller86]

In my experience, the SOBR rebalance doesn't work well. I even opened a ticket and received some unhelpful and confusing responses. You would think it would balance out according to free space % but it doesn't, at least in practice. We generally can't run it all the way to the end due to the length of time it takes and it stops all backup jobs during that operation.

IMO, what should happen is that when its kicked off, it determines what needs to move where to optimize it, and then begin moving. If it were to form a plan, and then execute the plan sequentially, in a manner where it is continually improving by moving backups from the extent with the lowest amount of free space %, to the extent with the highest amount of free space %, then it would be far superior to how it works now, and you could run it overnight, stop it the next morning, and no matter what there would be some form of improvement.

I've run this overnight for 12 hours, stopped it the next morning, and end up worse off than when I started. I then had to make non-SOBR repos on the same extents, to manually move backups out of the SOBR, to a separate repo on the same servers in the SOBR to force backups to another as were down to 3% free on that extent. I agree this routine needs improvement.

[Gostev]

Is there a way to set the calculation to use % free space so that I end I have something roughly like the below so they are all proportionally filled the same.
1 x 10TB, 6TB free
1 x 11TB, 6.65TB free
1 x 22TB, 13.30 TB free

Could you clarify what are you trying to achieve with such [im]balance? In other words, what is the specific problem with every extent having about equal free disk space, that you're trying to solve here by changing the balance to be like this?

[RobMiller86]

Gostev for us at least, we end up with certain extents filling and getting dangerously close to full capacity on 1 extent. It would seem to me that when someone runs a rebalance and you have something like:

Extent 1: 50% free
Extent 2: 25% free
Extent 3: 8% free

And you kick off a rebalance, that it should immediately begin addressing the fullest extent first, by moving backups to the extent with the highest amount of free space. My most recent rebalance actually made it worse, and after 12 hours of running, my extent 3 went from 8%, to 3% free. It seems that perhaps that logic could be improved, as I would imagine most people are only running a rebalance when they must, as it interrupts backups, and that "must" is likely due to 1 extent getting close to full.

[Gostev]

Gostev for us at least, we end up with certain extents filling and getting dangerously close to full capacity on 1 extent.

That's exactly why I'm puzzled with what OP wants. It seems keeping each and every extent as far from overfilling is possible is what everyone would want from rebalance. Meaning the following is the perfect layout, yet somehow OP does NOT like this?

1 x 10TB, 8.7TB free
1 x 11TB, 8.9TB free
1 x 22 TB, 8.4TB free
Unfortunately, even after a rebalance the balance seems to optimize for giving the all the same amount of free SIZE (as seen above)

[RobMiller86]

As I understand it, Christine is looking for the same thing we are. We desire the rebalance to focus on free space %, and not simply raw free space. Using my example above:

Extent 1 150TB total: 50% free
Extent 2 80TB total: 25% free
Extent 3 50TB total: 8% free

It seems like the rebalance focuses on putting the same amount of storage in TB to all extents, rather than balancing based on percentage. That's likely why my extent 3 above ended up with less free space after the rebalance. I'm in the middle of a deploying a new SOBR with all 3 extents being identical to help mitigate that. But if you don't have equal extents, it gets hairy. This doesn't matter if all extents are equal. But that example above, after 12 hours, in terms of % I had more free space on extent 1, and even less on extent 3. The exact opposite of what I expected or needed to occur.

[Gostev]

To be honest I don't like the free space % based idea very much, because this means leaving smaller extents dangerously close to their full capacity. 10% from 150TB is 15TB, but from 50TB it's only 5TB. And some customers have a whole zoo of extents, both very big and very small.

[RobMiller86]

I understand that point, but at a minimum that should be configurable based on total free raw capacity or total free as a percentage. That SOBR rebalance in my example nearly crushed me. I had no choice but to do an emergency evac of backups completely out of the SOBR after the rebalance so that extent 3 didn't completely fill. Any SOBR operation should never make a situation worse than it already is.

In my opinion, the SOBR rebalance as-is, simply doesn't work as most people would need or expect, unless they are running identically sized extents.

What is Veeam recommendation then for this situation?

--------------------------
Scenario:
Extent 1 150TB total: 50% free
Extent 2 80TB total: 25% free
Extent 3 50TB total: 8% free

Extent 3 has more churn, and is slowly filling up and nearly full and losing 1% per day. We need to migrate some backups off of it to Extent 1. We have no way to manually select a particular backup and move it between extents. Our only option is rebalance or evac. Rebalance makes it worse as it moves even more from Extent 1 to Extent 3. Veeam has left us no choice but to find a large backup chain in Extent 3 manually, and then do a backup move to a completely different repo somewhere.
-------------------------

If I'm incorrect about that, please let me know. As of now, I don't know how to solve that situation other than making all extents equal so I'm not in that situation in the first place and can easily rebalance. Or just accepting that I will sometimes have to manually intervene and completely remove backups from the SOBR which has its own implications with SOBR offload to object storage. I don't see how rebalance really helps anyone unless all extents are equal, or you have tons of free space everywhere. Where people really need it, extents of different sizes and one of them is filling to capacity, it doesn't seem to be helpful with no other workarounds other than evac.

[RobMiller86]

I can see 2 ways to solve this.

1) Provide an option when selecting to rebalance to either use raw free or % free
2) Provide a method to select a backup chain and move it to a different extent

#1 would be best. If we only had #2, then we would need to never rebalance, and instead only manually move chains around which seems like extra work when it could very easily be automated with #1.

[ChristineAlexa]

Could you clarify what are you trying to achieve with such [im]balance? In other words, what is the specific problem with every extent having about equal free disk space, that you're trying to solve here by changing the balance to be like this?

Gostev,
Thanks for the reply,
Essentially, I have unbalanced SOBR. I would expect the balance take the size into account when balancing. Currently, it presumes they are all the same size, so balances them to the same free space percentage, which, in an example like I provided above, ends nearly only using the largest extent, and barely touching the two smaller extents.

I would expect each extent, to roughly fill up the same percent of free space, vs the same absolute GB of free space.

As it is now, one extent has nearly all of the I/O and storage, and the two smaller extents with nearly no data or I/O.
So I can't (with SOBR rebalance) spread my risk and I/O across the extents in a more even manner.

[ChristineAlexa]

I can see 2 ways to solve this.

1) Provide an option when selecting to rebalance to either use raw free or % free
2) Provide a method to select a backup chain and move it to a different extent

#1 would be best. If we only had #2, then we would need to never rebalance, and instead only manually move chains around which seems like extra work when it could very easily be automated with #1.

Agreed, #1 is what I'd love and where I was going with my original post
#2 would be an awesome "workaround" as I'd just stop using rebalance and just deal with it manually and/or write my own script to move them around via VBR power shell if it warrants additional automation.

[RobMiller86]

Yeah, the other issue is the new backup file placement. That follows the same logic as the rebalance. So you might be down to 10% free on your smallest extent, but then when you add a new job, it unfortunately chooses that smallest and fullest extent. So in my opinion, this should be a selectable option in the SOBR config. If the SOBR config allowed one to choose to balance/place-new based on raw or %, then all operations selecting which extent to place files would follow that, which would include new backups, or a rebalance. At present, I'm not really sure how it helps anyone with extents of differing sizes. It effectively limits you to the max capacity of your smallest extent.

SOBR Configuration: Select balancing metric based on raw or %
If I have identical capacity extents, no problem, I would use raw based capacity.
If I have differing capacity extents, no problem, I would use % based capacity.

Problem solved.

[Gostev]

when you add a new job, it unfortunately chooses that smallest and fullest extent

Hmm, why in the world would we have such a logic? @veremin please confirm: when all extents are available, a backup job will pick the one with most free space available (in absolute amount), correct? And without even looking at its capacity.

[ChristineAlexa]

it unfortunately chooses that smallest and fullest extent.

The only time I have seen this behavior, was with data locality on, it will try to keep the incrementals/partials with the full, so it can use block cloning effectively.
So the selection logic for backups, is a different issue than what I was trying to get resolved initially.

[RobMiller86]

Gostev I could be wrong with how the logic works as I haven't read any published information on the precise logic other than this https://helpcenter.veeam.com/docs/backu ... ml?ver=120. I am only basing this on what I have personally observed, and that's what the logic appeared to be doing. It was placing a new backup, for a new job, on our smallest extent, which also had the least amount of free space available, both in terms of raw and %. Could it be a bug? Yes, or it could be this is how it's designed to work. But I have witnessed this on one of our SOBRs with different sized extents. We always have data locality turned on as it seems to be more risky to not have that on. I understand that will keep the incrementals with the fulls, and Veeam has no way to predict the churn of that backup and how fast it will grow, so I imagine it must make a decision based on the estimated size of the first active full. That would then cause uneven growth, and a rebalance would be needed later, but that's expected. But even with data locality, I can't surmise why it would choose that smallest extent, using either metric for a new backup, or why an extent goes from 8% free to 3% free from a rebalance op with 2 others having more space using either metric. The only thing that made sense to me is that it's placing the new backup based on raw storage and trying to make that even everywhere.

Even this which is listed in the link above is hard to grasp on how it impacts the extent selection as it relates to storage: Load control settings — maximum number of tasks that the extent can process simultaneously.

If we could know precisely how it's intended to work, that would help us to align that with the behavior we tend to see, and then determine if it's working as designed or not, or if the design could use some additional optimizations or options. All I can really do is observe and try to ascertain what that logic might be.

For us it's not really going to be an issue anymore as we just bought all new equally sized servers for SOBR repos which we will be configuring soon, but it still would be nice to understand this and have options (if needed) for future needs.