Error: No scale-out repository extents have sufficient disk space to store the backup file

[toreml]

Hi.

I've registred a case with number 03321400 regarding the error message I get below.
I've had this before - and I was told that update 3a would probably fix this issue.
I've updated the system, but I still have the same problem.
It doesn't happen all the time - but it has now atleast happend twich in 7 days.

11.12.2018 08:41:31 :: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.


Regards
Tore ML

[Dima P.]

Hello Tore,

Can you please clarify the amount of free space on your Scale-out repository extents, backup placement policy and the estimated (or average) size of the restore point created by the backup job with Scale-out backup repository as a target. Thank you in advance.

[toreml]

Hi.
Free space is 33 TB on the scaleout.
Data locality and pr VM.

Last size of one of the jobs ha 23,9 GB transferred.
But this is None compression level and Local target ( 16TB+ backup files) since we use Cohesity as target - and Cohesity is compressing and deduplication the backup.

Tore

[toreml]

I got this reply from support - I also added the registry in the KB2282
I removed some old backup from the storage as well.
Will see tomorrow if these settings have some affect.

This issue is related to the logic of how Veeam calculating space available on SOBR extents. Veeam caches free space information from the database.
When resources for backup of a certain VM are assigned, we check the space it will take. For instance, VM is 10GB. Requested free space for its full backup will be 5GB (50% of VM size). Veeam now thinks 95GB is available on this extent. While there is at least 1 active task Veeam does not update for free space information. It uses its cache considering the size of VMs currently being backed up.
For instance, in 15 minutes we will try to assign resources for a new task, the meanwhile first one only created storage and started writing into it, having taken only 1GB for now. We may decide 99GB are available, but that's not correct. Available space is still 95GB.
Let's say, in half an hour new task is started, SOBR resource needs to be assigned. Active tasks are still running, so we still have 95GB free space.
New VM requested size 10GB. A resource is assigned, Veeam now thinks free space is 85GB (100-5-10). The actual status will be polled only if an extent is not used by any job.

In your case, we can see that there is a lot of space cached for the current jobs:
[11.12.2018 08:16:08] <23> Info [RTS] Updating extents statuses for 'Ulven Cohesity Scale-Out' SOBR
[11.12.2018 08:16:08] <23> Info [RTS] Extent name = 'Cohesity 01', id ='c934d112-b45c-4534-917b-0a359fc8849e' is available. Load: -5700.00%
[11.12.2018 08:16:08] <23> Info [RTS] Extent name = 'Cohesity 02', id ='9bcc76d2-5978-4029-a866-ccae8cf725b4' is available. Load: -900.00%
[11.12.2018 08:16:08] <23> Info [RTS] Extent name = 'Cohesity U03', id ='698e2edf-d7e9-480e-8583-080ccc3e8fea' is available. Load: -2800.00%

Since version 9.5 Update 2 it's possible to switch to the new logic. Using this logic Veeam recalculates available space each time when a task is finished. You can find how to enable this logic in this KB article: https://www.veeam.com/kb2282