Comprehensive data protection for all workloads
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Scale-out Repository not working as it should - 00464187

Post by prabhash.jena »

Hi,

Upgraded to version 9, 2 weeks ago and configured Scale-out Repository. I have 3 extents of different capacity - details as follows

Scale-out Rep. Capacity - 94.2 TB Free Space - 25.7 TB
Extent 1 - Capacity - 50.9 TB Free Space - 19.3 TB
Extent 2 - Capacity - 23.3 TB Free Space - 12.7 Gb
Extent 3 - Capacity - 20 TB Free Space - 6.5 TB

Though there is enough free space still jobs are failing with error
:: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.

I have re-scanned managed servers and repositories quite a few time however it still fails.
What could be wrong? Bug or my configurations?

Please provide inputs.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by PTide »

Hi,

Please describe you Backup File Placement policy - is it Data Locality of Performance? Also what's the size of your backup file?

Thank you.
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by DaveWatkins »

As a workaround, if this is the same as what I saw, re-run the job manually and you'll get 3 or 4 more VM's that will work and complete with each run, keep doing that until all VM's in the job successfully backup once and it will work as normal after that
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena »

PTide wrote:Hi,

Please describe you Backup File Placement policy - is it Data Locality of Performance? Also what's the size of your backup file?

Thank you.
I have opted for Performence. The size varies from few 100 Gbs to 3-4 TB.
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena »

DaveWatkins wrote:As a workaround, if this is the same as what I saw, re-run the job manually and you'll get 3 or 4 more VM's that will work and complete with each run, keep doing that until all VM's in the job successfully backup once and it will work as normal after that
I will try your suggestion, infact during troubleshooting I did this on some jobs and left because of failures on subsequent runs. This needs to be addressed by Veeam on priority , this completely defies the whole purpose of setting up Scale-Out file repository. This was announced as a big feature in Veeam9. I am disappointed that it doesn't work as it was described.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev »

Do you have per-VM backup file chains enabled in the SOBR (Scale Out Backup Repository) settings?
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena »

Gostev wrote:Do you have per-VM backup file chains enabled in the SOBR (Scale Out Backup Repository) settings?
Indeed, I have enabled per-VM backup wherever I could find it or is available on Veeam console.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Vitaliy S. »

Prabhash, what's your case ID for this problem? Both tickets (00450715, 00464187) mentioned in this thread and in our support system seems like refer to other things and are already closed.
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena »

I have not created a case yet.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Vitaliy S. »

Ok, can you please let our support team take a look at your setup and let us know your case ID.
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by SyNtAxx »

prabhash.jena wrote:Hi,

Upgraded to version 9, 2 weeks ago and configured Scale-out Repository. I have 3 extents of different capacity - details as follows

Scale-out Rep. Capacity - 94.2 TB Free Space - 25.7 TB
Extent 1 - Capacity - 50.9 TB Free Space - 19.3 TB
Extent 2 - Capacity - 23.3 TB Free Space - 12.7 Gb
Extent 3 - Capacity - 20 TB Free Space - 6.5 TB

Though there is enough free space still jobs are failing with error
:: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.

I have re-scanned managed servers and repositories quite a few time however it still fails.
What could be wrong? Bug or my configurations?

Please provide inputs.
I've had a case open for this for a bit: Veeam Support - Case # 01691757


-NIck
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev »

Nick, we have found the reason for your specific issue, hot fix is in the works.
Prabhash, please open a support case so that we can confirm you are having the same issue.
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena »

Case opened. Its 01699025. Awaiting for support engineers to collect some relevant logs. I provided logs from one job while I opened up the case.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev »

Thanks! You can also refer them to issue ID 69045 as there is a chance this may be the same issue. The hot fix for 69045 is currently with QC for validation.
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by SyNtAxx »

Gostev wrote:Nick, we have found the reason for your specific issue, hot fix is in the works.
Thank you sir for the update. As always it is much appreciated.

-Nick
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by SyNtAxx »

any updates?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Vitaliy S. »

QA team is verifying it. Our support team will notify you once it is available!
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena »

Gostev wrote:Thanks! You can also refer them to issue ID 69045 as there is a chance this may be the same issue. The hot fix for 69045 is currently with QC for validation.
Thanks! I have shared the information with support.

My current situation is like this after I made manual changes to get the backups to work.
Extent1 – 50 TB Free space – 13 TB
Extent2 – 23 TB Free space – 02 TB
Extent3 – 20 TB Free space – 03 TB

Now the question is, when Extent2 or 3 becomes full will SOBR automatically redirects backup data to Extent1?
The extents are CIFS shares on 2 node Microsoft Scale-out File clusters.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by foggy »

Yes, data will be placed on extent where enough space is available.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev » 1 person likes this post

Vitaliy S. wrote:QA team is verifying it.
And they've been doing a really good job as emails from builder told me this hot fix was rebuilt 3 times already ;)
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » 1 person likes this post

Gostev wrote:Thanks! You can also refer them to issue ID 69045 as there is a chance this may be the same issue. The hot fix for 69045 is currently with QC for validation.
I provided the information to the support team. After few days I was provided with a fix. I have applied the fix.
It will be a week or so before I can conclude that the jobs are being redirected to another extent in case one of the extent goes full and runs out of space.
Thanks for your inputs :)

Prabhash
SyNtAxx
Expert
Posts: 149
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by SyNtAxx »

I seem to be running into the issue again, repo has 21TB free. The hotfix was applied a few weeks back.

-Nick

-
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev »

At this time you should just get Update 1 from support, as it fixes all the related issues.
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena »

Hi Gostev,

I can confirm the scale out repository doesn't work as intended by Veeam atleast in my case. I experienced the same issue where 1 extent got full and the job failed instead of redirecting it to another extent which had more than 20TB free space.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by foggy »

Do you have Update 2 installed?
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena »

Yes. 9.0.0.1715. I am kind of early bird with application & system updates.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by foggy »

Then involving support is recommended to investigate the reasons of this behavior.
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by pirx »

Running 9.5U1 here and see the same problems.

Error: No scale-out repository extents have sufficient disk space to store the backup file.

There are 58TB of 100TB free. Seems that SOBR is not working as expected at all.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev »

@Pirx what is the support case ID for this? All known issues around SOBR have been fixed in 9.5, so this must be something different.
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by pirx »

#02059754

Code: Select all

Please note that we try to allocate the space on the extents at the very beginning of the job. The space is allocated according to the following criteria : full backup - 50% of the VM size, increment - 10%. Sure it could be plenty of free space but once all the jobs are started at the same time, it is possible that all the space is allocated and that's why all the jobs will fail. For such kind of situations we recommend two registry keys that will reduce the percentage of the space allocation : Name: SOBRFullCompressRate Type: REG_DWORD Value: 25 Name: SOBRIncrementCompressRate Type: REG_DWORD Value: 5 Both keys should be added at the veeam server in HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Please add both keys, restart the jobs and let me know about the results
All full jobs are finished now, we started them one after the other. I'll see if I can reproduce this with the given regkey settings. We use 14 primary restore points on this repository, but the backup copy repositories are SOBR too, so we should see the same effect (maybe later because our copy SOBR's are bigger because they must hold backups for 10 weeks).
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 196 guests