Scale-out Repository not working as it should - 00464187

[Squish]

I have faced this problem as well. Bad is, there is no info about it in veeam documentation. So I was living in the idea that when I will be on low space I will simply add more later. And my jobs started to fail when my capacity was at 50% free.

Anyway adding both reg keys fixed this problem

[nunciate]

Does anyone know if this entire thing can be disabled by just entering zeros in the registry keys. I keep getting failures due to this issue. I went all the way down to 1 for each of these registry key values and still have the issue on a regular basis.

I have 5 repositories and some with a large amount of free space. 3,4 and 5 TB free on some yet I still get the message. It is a bit ridiculous. If it can check free space it should be smart enough to use an alternate drive with enough space.

[foggy]

Hi Alan, what Veeam B&R version are you running and what registry keys do you mean? Have you contacted support with the particular issue you're seeing?

[nunciate]

I am referring to the post below. I did this and it fixed most of my issues but some of my drives run a little lower on space than others. This issue still pops up from time to time.
I am on v9.5.0.823. I have not opened a ticket yet since I found this thread and for the most part I am fixed. It happened again a couple of days ago and a lot of backups failed. When I re-ran them individually they finished successfully.

Post earlier in this thread:

Please note that we try to allocate the space on the extents at the very beginning of the job. The space is allocated according to the following criteria : full backup - 50% of the VM size, increment - 10%. Sure it could be plenty of free space but once all the jobs are started at the same time, it is possible that all the space is allocated and that's why all the jobs will fail. For such kind of situations we recommend two registry keys that will reduce the percentage of the space allocation : Name: SOBRFullCompressRate Type: REG_DWORD Value: 25 Name: SOBRIncrementCompressRate Type: REG_DWORD Value: 5 Both keys should be added at the veeam server in HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Please add both keys, restart the jobs and let me know about the results

[foggy]

The quoted text does explain the issue and the fact that individual job runs were successful also confirms that. I'd try to spread the jobs in time to prevent this.

[nunciate]

Yea I will have to do that. I will limp along for a few months and then we are going to double our storage out there so it shouldn't be an issue. Just wondering if there was an easy way to just disabled the function temporarily

[foggy]

Yes, you can set the values to 0, however, this might result in filling up the repository and inability to save metadata, leading to even worse consequences.

[dries.vergote]

We have the same issue after or SOBR with 1 extent ran full.
We added 2TB to the disk and now have stille have 1,3TB free space but the backups are still failing if they run during the night.

Rerun of jobs fail almost immediatly if a job is running.

Case created: #02129726.

Is the only solution the regkey?
I'm also freeing up some diskspace by moving a job to another SOBR.

[dries.vergote]

Adding the keys resolved the problem, as creating more free diskspace.

[b.vanhaastrecht]

I've also added more space to the repository linked to an SOBR. The added space is detected, but jobs keep 'No scale-out repository extents have sufficient disk space to store the backup file.'

I've added the two registry keys (where I take the DWORD value is decimal?) and now the jobs run successfully.

Adding keys to allow a very common task, extending a repository, seems to be some kind of messy. Will this fix be applied more permenatly so I can remove the keys again?

[foggy]

The keys do not allow for repository extension, but disable space allocation checks. So they never be applied by default in the product.

[b.vanhaastrecht]

Ok, so could you explain why freshly added space to a REPO is not detected by SOBR and therefore jobs fail with insufficient space. All this while the Rep en SOBR display the newly, added disk space.

[foggy]

Probably space allocation mechanism still considered there was not enough space for all the started backups.

[b.vanhaastrecht]

I've run them three times in a time period of one hour. Did the reg keys right after a previous failed and instantly they go successfully. So that makes the regrets very highly the cause of the success.

[foggy]

Hard to tell via the forums, but it could be that both adding space and registry values modification did the trick.

[b.vanhaastrecht]

I agree. But the space was already added 3 test backup runs before, the jobs failed. Added the regkeys and eureka.

If I need to add space again, I will monitor the process and let you know.

[pirx]

Sorry that I've to bring up this old thread. After an absence of 3,5 years I'm back in my old job. Which means keep Veeam (10a) backups running. And after just one week I'm facing the same problem I've when this thread was started.

We regularly have problems with our SOBR. Yesterday a job did not run and retries failed too (3:00, 3:22, 3:46),. A manual run at 09:57 was successful. The SOBR had more than enough free space. It consists of 2 extends, each with 90TB. Currently there are 22TB available on each extend.

Code: Select all

Job log:
14.10.2020 03:22:06 :: Job started at 14.10.2020 03:22:04  
14.10.2020 03:22:06 :: Building list of machines to process  
14.10.2020 03:22:35 :: VM size: 1,8 TB (1,8 TB used)  
14.10.2020 03:22:36 :: Changed block tracking is enabled  
14.10.2020 03:27:53 :: Queued for processing at 14.10.2020 03:27:53  
14.10.2020 03:28:02 :: Required backup infrastructure resources have been assigned  
14.10.2020 03:28:03 :: All VMs have been prepared for storage snapshot  
14.10.2020 03:33:35 :: Creating storage snapshot  
14.10.2020 03:34:22 :: Waiting for backup infrastructure resources availability  
14.10.2020 03:34:37 :: Waiting for backup infrastructure resources availability  
14.10.2020 03:34:58 :: All VMs have been queued for processing  
14.10.2020 03:34:58 :: Deleting storage snapshot  
14.10.2020 03:36:33 :: Job finished with error at 14.10.2020 03:36:33

VM log:
14.10.2020 03:29:52 :: Creating VM snapshot  
14.10.2020 03:29:57 :: Collecting disk files location data  
14.10.2020 03:33:45 :: Removing VM snapshot  
14.10.2020 03:34:21 :: Queued for processing at 14.10.2020 03:34:22  
14.10.2020 03:34:37 :: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.  
14.10.2020 03:34:37 :: Processing finished with errors at 14.10.2020 03:34:37  

I found the 2 regkeys from this thread.

Code: Select all

SOBRFullCompressRate Type: REG_DWORD Value: 25 
SOBRIncrementCompressRate Type: REG_DWORD Value: 5 

Currently the values are 21 and 2. As I don't know the defaults, I'm not sure if I should change them to 25 and 5 or lower them.

I also found https://www.veeam.com/kb2282 (How to enable advanced space control for Scale-out Backup Repository)

Code: Select all

HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\SobrForceExtentSpaceUpdate

So where should I start? Just add the new key SobrForceExtentSpaceUpdate = 1 or change the other 2 keys too (which value)? I don't want to open an ticket right away, maybe there is standard solution.

[pirx]

I just noticed that there is also "SOBRSyntheticFullCompressRate". We see these "Error: No scale-out repository extents have sufficient disk space to store the backup file" for backup and syn full jobs. As far as I understand our value 22 for SOBRFullCompressRate is already very low (assuming 100 is the default). We do not have a SOBRSyntheticFullCompressRate regkey yet.

[pirx]

BTW: I created ticket 04442942

[pirx]

Update: I changed SobrForceExtentSpaceUpdate to 1 a while ago. This solved the problem with "No scale-out repository extents have sufficient disk space to store the backup file" but introduced a new problem: "Error: There is not enough space on the disk.". This happend when there was plenty of free space on the SOBR and it extents. The extents are CIFS shares on a NetApp FAS filer and build as Flexgroup with constituent volumes. So Veeam does only see a 100TB share, which the consists of 4 x 25TB. Currently it looks like one of the constituents was running full during synthetic operations (merges, syn full) as these files just grow in place the filer has no chance to place them on one of the other constituent volumes with more free space. An ONTAP update should solve this problem now. Maybe this helps if someone is running in tis problem, we could not find the reason for this "not enough space" for a long time as Veeam itself reported in all logs that the extent had plenty of free space...