SOBR initial balancing

[mdiver]

We are running (amongst other repos) a SOBR with 7 ReFS extents (36TB, 45TB, 36TB, 45TB, 17TB, 18TB, 10TB) as a target for one single Backup-Copy job with ~400VMs.

We are doing per-VM-chains and data-locality to benefit from ReFS savings on GFS and Fast-Clone.

On the first, inital run of the job we see only the 4 largest of the extents getting data ingested. Three extents stay completely empty.
In addition the Backup-Copy job is having to wait for repository-availability - so is hindered by having to few repositories to copy to.
This is though there are three more extents waiting for data.

To my understanding, with per-VM-chains we should see data being ingested to all extents at the same time to get the load somehow distributed.
Only when reaching capacity limits I would have expected an extent to be left out.

I opened case #03914575 to clarify, but was told that the distribution is not to be expected.

Thanks for any ideas,
Mike

[ejenner]

Are any of these repositories on the same server or are they all separate / NAS type things? I think it's better to join them together at the Windows level using a dynamic disk or if they're physical disks connected to a RAID controller you can put them all into the same logical volume. I've done this with our most recently implemented repository and it simply presents with 160TB straight to Veeam. The OS and RAID cards handle the distribution of the data.

[mdiver]

We're having three Windows servers here with 2+2+3 extents. We did this a lot already with other customers. Usually it just works.
Only in this case one of the servers with its three extents is fully left out of the game during data ingestion to the SOBR.

I personally also think it's better to have individual RAIDs to be presented as different extents to have Veeam distribute the load between them. It's mainly bottleneck target anyways. At least during the merges. So RAID is the limit.

[oleg.feoktistov]

How many concurrent tasks you configured in Load control settings for each of the SOBR extents?
Among other factors, VBR prioritize extent selection by slots for simultaneous tasks available and free space on the extents
as described here.

[mdiver]

We habe 6 slots on the each of 7 destination extents and also 6 slots on only 4 source extents.
Therefore we should never have to wait for destination slots. Backup copy is repo->repo. So no proxy involved.

The only thing that differs is the amount of free space on the destination extents as they derive from different server RAIDs (all between 20 and 40 TB).
4 of them have approx. double the size as the other three. But the user manual is not precise on how an extent is choosen by the amount of free space.
I'd have assumed that unless an extent nears its capacity limits, it will be used. I would like to spread the job to all 7 destination extents for performance reasons.

Thanks and by the way: happy holidays...

[oleg.feoktistov]

Hi,

VBR will choose SOBR extents based on load control settings (free slots) > free space.
So, in your case, among all the extents with equal amount of slots those with more free space will be chosen first.
Backups will be also distributed to 3 smaller extents when the larger ones will either run out of slots or reach similar to the smaller ones amount of free space.
I'd suggest to try lowering the amount of slots on larger extents to achieve your goal.

Merry Christmas and Best regards!
Oleg

[douglas.fernandes]

Hello all

I had a big problem at my environment, using SOBR with data locality Policy.

the SOBR put all files chain at the same extent and are save the files until the extent is completely full, but if this extents doesn’t have space, the job will failed with the error below.

“Unable to allocate processing resources. Error: Some extents storing previous backup file are offline”.

To resolve this error, you must add more space in the extension even if another extension has space.

I my opinion this is a strong error and problem, because what is the razon to use SOBR if the system not use correctly all disks put at the SOBR?

I did a test with SOBR with Performance Policy, where we informed that the files will be saved in all extensions, everything ok, except when necessary to merge the older incremental files, the task needs to be done during two or more days to finish, impossibility my Windows Job.

I hope that have some solution to solve it.

Regards

[oleg.feoktistov]

Hi Douglas,

With data locality policy a new backup chain can be put on different SOBR extent.
Also, if required extent is offline or cannot be reached, you can instruct VBR to trigger an active full backup on another extent:
Go to SOBR properties > Performance Tier > Advanced and tick "Perform full backup when required extent is offline" checkbox.

By the way, the info about free space on each extent is stored in cache and may not be timely updated with the real numbers if at least one task is targeted at a particular extent.
Check this KB.

Thanks and Happy holidays!
Oleg

[douglas.fernandes]

Hi Oleg,

This KB had already been applied here

About your comments

1 - My extension is not offline, but there was no more room to save file.

2 - If I add this option "Perform full backup ...", I understand that it will create a new chain in the other extension, but it will not be deleted from the old chain until this new chain has all the "* .vibs" points. Therefore, it will increase my disk space usage.

Thanks for Happy Holidays to you too"!!!

[mdiver]

I'd suggest to try lowering the amount of slots on larger extents to achieve your goal.

Happy new year everybody and thanks for this idea.

I tried to equalize by limiting the number of slots. As of now it seems to work.
A feature request would be some more sophisticated algorithm to distribute the backup chains between the extents. Currently it's putting the largest VM to the smallest extent.
Also an automatic balancing algorithm would be nice the relocates Independent chains to other extents.

Best regards,
Mike

[ejenner]

Dunno if you can use storage spaces direct or something like that? I would take a huge amount of convincing to try Veeam SOBR again. Got burnt once and prefer anything else. Just my opinion, I know it works really well for a lot of customers.

[foggy]

1 - My extension is not offline, but there was no more room to save file.

In that case, Veeam B&R should violate the policy and place the new backup on another available extent. If you see different behavior, I suggest asking support for assistance. Thanks!