Scale-out Repository not working as it should - 00464187

[prabhash.jena]

Hi,

Upgraded to version 9, 2 weeks ago and configured Scale-out Repository. I have 3 extents of different capacity - details as follows

Scale-out Rep. Capacity - 94.2 TB Free Space - 25.7 TB
Extent 1 - Capacity - 50.9 TB Free Space - 19.3 TB
Extent 2 - Capacity - 23.3 TB Free Space - 12.7 Gb
Extent 3 - Capacity - 20 TB Free Space - 6.5 TB

Though there is enough free space still jobs are failing with error
:: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.

I have re-scanned managed servers and repositories quite a few time however it still fails.
What could be wrong? Bug or my configurations?

Please provide inputs.

[PTide]

Hi,

Please describe you Backup File Placement policy - is it Data Locality of Performance? Also what's the size of your backup file?

Thank you.

[DaveWatkins]

As a workaround, if this is the same as what I saw, re-run the job manually and you'll get 3 or 4 more VM's that will work and complete with each run, keep doing that until all VM's in the job successfully backup once and it will work as normal after that

[prabhash.jena]

Hi,

Please describe you Backup File Placement policy - is it Data Locality of Performance? Also what's the size of your backup file?

Thank you.

I have opted for Performence. The size varies from few 100 Gbs to 3-4 TB.

[prabhash.jena]

As a workaround, if this is the same as what I saw, re-run the job manually and you'll get 3 or 4 more VM's that will work and complete with each run, keep doing that until all VM's in the job successfully backup once and it will work as normal after that

I will try your suggestion, infact during troubleshooting I did this on some jobs and left because of failures on subsequent runs. This needs to be addressed by Veeam on priority , this completely defies the whole purpose of setting up Scale-Out file repository. This was announced as a big feature in Veeam9. I am disappointed that it doesn't work as it was described.

[Gostev]

Do you have per-VM backup file chains enabled in the SOBR (Scale Out Backup Repository) settings?

[prabhash.jena]

Do you have per-VM backup file chains enabled in the SOBR (Scale Out Backup Repository) settings?

Indeed, I have enabled per-VM backup wherever I could find it or is available on Veeam console.

[Vitaliy S.]

Prabhash, what's your case ID for this problem? Both tickets (00450715, 00464187) mentioned in this thread and in our support system seems like refer to other things and are already closed.

[prabhash.jena]

I have not created a case yet.

[Vitaliy S.]

Ok, can you please let our support team take a look at your setup and let us know your case ID.

[SyNtAxx]

Hi,

Upgraded to version 9, 2 weeks ago and configured Scale-out Repository. I have 3 extents of different capacity - details as follows

Scale-out Rep. Capacity - 94.2 TB Free Space - 25.7 TB
Extent 1 - Capacity - 50.9 TB Free Space - 19.3 TB
Extent 2 - Capacity - 23.3 TB Free Space - 12.7 Gb
Extent 3 - Capacity - 20 TB Free Space - 6.5 TB

Though there is enough free space still jobs are failing with error
:: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.

I have re-scanned managed servers and repositories quite a few time however it still fails.
What could be wrong? Bug or my configurations?

Please provide inputs.

I've had a case open for this for a bit: Veeam Support - Case # 01691757


-NIck

[Gostev]

Nick, we have found the reason for your specific issue, hot fix is in the works.
Prabhash, please open a support case so that we can confirm you are having the same issue.

[prabhash.jena]

Case opened. Its 01699025. Awaiting for support engineers to collect some relevant logs. I provided logs from one job while I opened up the case.

[Gostev]

Thanks! You can also refer them to issue ID 69045 as there is a chance this may be the same issue. The hot fix for 69045 is currently with QC for validation.

[SyNtAxx]

Nick, we have found the reason for your specific issue, hot fix is in the works.

Thank you sir for the update. As always it is much appreciated.

-Nick

[SyNtAxx]

any updates?

[Vitaliy S.]

QA team is verifying it. Our support team will notify you once it is available!

[prabhash.jena]

Thanks! You can also refer them to issue ID 69045 as there is a chance this may be the same issue. The hot fix for 69045 is currently with QC for validation.

Thanks! I have shared the information with support.

My current situation is like this after I made manual changes to get the backups to work.
Extent1 – 50 TB Free space – 13 TB
Extent2 – 23 TB Free space – 02 TB
Extent3 – 20 TB Free space – 03 TB

Now the question is, when Extent2 or 3 becomes full will SOBR automatically redirects backup data to Extent1?
The extents are CIFS shares on 2 node Microsoft Scale-out File clusters.

[foggy]

Yes, data will be placed on extent where enough space is available.

[Gostev]

QA team is verifying it.

And they've been doing a really good job as emails from builder told me this hot fix was rebuilt 3 times already

[prabhash.jena]

Thanks! You can also refer them to issue ID 69045 as there is a chance this may be the same issue. The hot fix for 69045 is currently with QC for validation.

I provided the information to the support team. After few days I was provided with a fix. I have applied the fix.
It will be a week or so before I can conclude that the jobs are being redirected to another extent in case one of the extent goes full and runs out of space.
Thanks for your inputs

Prabhash

[SyNtAxx]

I seem to be running into the issue again, repo has 21TB free. The hotfix was applied a few weeks back.

-Nick

-

[Gostev]

At this time you should just get Update 1 from support, as it fixes all the related issues.

[prabhash.jena]

Hi Gostev,

I can confirm the scale out repository doesn't work as intended by Veeam atleast in my case. I experienced the same issue where 1 extent got full and the job failed instead of redirecting it to another extent which had more than 20TB free space.

[foggy]

Do you have Update 2 installed?

[prabhash.jena]

Yes. 9.0.0.1715. I am kind of early bird with application & system updates.

[foggy]

Then involving support is recommended to investigate the reasons of this behavior.

[pirx]

Running 9.5U1 here and see the same problems.

Error: No scale-out repository extents have sufficient disk space to store the backup file.

There are 58TB of 100TB free. Seems that SOBR is not working as expected at all.

[Gostev]

@Pirx what is the support case ID for this? All known issues around SOBR have been fixed in 9.5, so this must be something different.

[pirx]

#02059754

Code: Select all

Please note that we try to allocate the space on the extents at the very beginning of the job. The space is allocated according to the following criteria : full backup - 50% of the VM size, increment - 10%. Sure it could be plenty of free space but once all the jobs are started at the same time, it is possible that all the space is allocated and that's why all the jobs will fail. For such kind of situations we recommend two registry keys that will reduce the percentage of the space allocation : Name: SOBRFullCompressRate Type: REG_DWORD Value: 25 Name: SOBRIncrementCompressRate Type: REG_DWORD Value: 5 Both keys should be added at the veeam server in HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Please add both keys, restart the jobs and let me know about the results

All full jobs are finished now, we started them one after the other. I'll see if I can reproduce this with the given regkey settings. We use 14 primary restore points on this repository, but the backup copy repositories are SOBR too, so we should see the same effect (maybe later because our copy SOBR's are bigger because they must hold backups for 10 weeks).