Discussions specific to the VMware vSphere hypervisor
prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » Feb 12, 2016 2:41 pm

Hi,

Upgraded to version 9, 2 weeks ago and configured Scale-out Repository. I have 3 extents of different capacity - details as follows

Scale-out Rep. Capacity - 94.2 TB Free Space - 25.7 TB
Extent 1 - Capacity - 50.9 TB Free Space - 19.3 TB
Extent 2 - Capacity - 23.3 TB Free Space - 12.7 Gb
Extent 3 - Capacity - 20 TB Free Space - 6.5 TB

Though there is enough free space still jobs are failing with error
:: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.

I have re-scanned managed servers and repositories quite a few time however it still fails.
What could be wrong? Bug or my configurations?

Please provide inputs.

P.Tide
Product Manager
Posts: 5315
Liked: 467 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by P.Tide » Feb 12, 2016 4:04 pm

Hi,

Please describe you Backup File Placement policy - is it Data Locality of Performance? Also what's the size of your backup file?

Thank you.

DaveWatkins
Expert
Posts: 349
Liked: 93 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by DaveWatkins » Feb 12, 2016 11:55 pm

As a workaround, if this is the same as what I saw, re-run the job manually and you'll get 3 or 4 more VM's that will work and complete with each run, keep doing that until all VM's in the job successfully backup once and it will work as normal after that

prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » Feb 14, 2016 7:38 pm

PTide wrote:Hi,

Please describe you Backup File Placement policy - is it Data Locality of Performance? Also what's the size of your backup file?

Thank you.
I have opted for Performence. The size varies from few 100 Gbs to 3-4 TB.

prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » Feb 14, 2016 7:43 pm

DaveWatkins wrote:As a workaround, if this is the same as what I saw, re-run the job manually and you'll get 3 or 4 more VM's that will work and complete with each run, keep doing that until all VM's in the job successfully backup once and it will work as normal after that
I will try your suggestion, infact during troubleshooting I did this on some jobs and left because of failures on subsequent runs. This needs to be addressed by Veeam on priority , this completely defies the whole purpose of setting up Scale-Out file repository. This was announced as a big feature in Veeam9. I am disappointed that it doesn't work as it was described.

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev » Feb 14, 2016 9:05 pm

Do you have per-VM backup file chains enabled in the SOBR (Scale Out Backup Repository) settings?

prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » Feb 15, 2016 10:02 am

Gostev wrote:Do you have per-VM backup file chains enabled in the SOBR (Scale Out Backup Repository) settings?
Indeed, I have enabled per-VM backup wherever I could find it or is available on Veeam console.

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Vitaliy S. » Feb 15, 2016 11:16 am

Prabhash, what's your case ID for this problem? Both tickets (00450715, 00464187) mentioned in this thread and in our support system seems like refer to other things and are already closed.

prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » Feb 15, 2016 11:41 am

I have not created a case yet.

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Vitaliy S. » Feb 15, 2016 11:49 am

Ok, can you please let our support team take a look at your setup and let us know your case ID.

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by SyNtAxx » Feb 15, 2016 6:49 pm

prabhash.jena wrote:Hi,

Upgraded to version 9, 2 weeks ago and configured Scale-out Repository. I have 3 extents of different capacity - details as follows

Scale-out Rep. Capacity - 94.2 TB Free Space - 25.7 TB
Extent 1 - Capacity - 50.9 TB Free Space - 19.3 TB
Extent 2 - Capacity - 23.3 TB Free Space - 12.7 Gb
Extent 3 - Capacity - 20 TB Free Space - 6.5 TB

Though there is enough free space still jobs are failing with error
:: Unable to allocate processing resources. Error: No scale-out repository extents have sufficient disk space to store the backup file.

I have re-scanned managed servers and repositories quite a few time however it still fails.
What could be wrong? Bug or my configurations?

Please provide inputs.
I've had a case open for this for a bit: Veeam Support - Case # 01691757


-NIck

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev » Feb 16, 2016 12:36 pm

Nick, we have found the reason for your specific issue, hot fix is in the works.
Prabhash, please open a support case so that we can confirm you are having the same issue.

prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » Feb 16, 2016 2:58 pm

Case opened. Its 01699025. Awaiting for support engineers to collect some relevant logs. I provided logs from one job while I opened up the case.

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev » Feb 16, 2016 3:33 pm

Thanks! You can also refer them to issue ID 69045 as there is a chance this may be the same issue. The hot fix for 69045 is currently with QC for validation.

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by SyNtAxx » Feb 16, 2016 4:31 pm

Gostev wrote:Nick, we have found the reason for your specific issue, hot fix is in the works.
Thank you sir for the update. As always it is much appreciated.

-Nick

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by SyNtAxx » Feb 17, 2016 4:16 pm

any updates?

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Vitaliy S. » Feb 17, 2016 5:00 pm

QA team is verifying it. Our support team will notify you once it is available!

prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » Feb 19, 2016 10:53 am

Gostev wrote:Thanks! You can also refer them to issue ID 69045 as there is a chance this may be the same issue. The hot fix for 69045 is currently with QC for validation.
Thanks! I have shared the information with support.

My current situation is like this after I made manual changes to get the backups to work.
Extent1 – 50 TB Free space – 13 TB
Extent2 – 23 TB Free space – 02 TB
Extent3 – 20 TB Free space – 03 TB

Now the question is, when Extent2 or 3 becomes full will SOBR automatically redirects backup data to Extent1?
The extents are CIFS shares on 2 node Microsoft Scale-out File clusters.

foggy
Veeam Software
Posts: 18366
Liked: 1576 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by foggy » Feb 19, 2016 10:58 am

Yes, data will be placed on extent where enough space is available.

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev » Feb 19, 2016 11:48 pm

Vitaliy S. wrote:QA team is verifying it.
And they've been doing a really good job as emails from builder told me this hot fix was rebuilt 3 times already ;)

prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » Feb 29, 2016 9:34 pm 1 person likes this post

Gostev wrote:Thanks! You can also refer them to issue ID 69045 as there is a chance this may be the same issue. The hot fix for 69045 is currently with QC for validation.
I provided the information to the support team. After few days I was provided with a fix. I have applied the fix.
It will be a week or so before I can conclude that the jobs are being redirected to another extent in case one of the extent goes full and runs out of space.
Thanks for your inputs :)

Prabhash

SyNtAxx
Expert
Posts: 148
Liked: 15 times
Joined: Jan 02, 2015 7:12 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by SyNtAxx » Mar 14, 2016 9:19 pm

I seem to be running into the issue again, repo has 21TB free. The hotfix was applied a few weeks back.

-Nick

-

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev » Mar 14, 2016 10:23 pm

At this time you should just get Update 1 from support, as it fixes all the related issues.

prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » Oct 06, 2016 12:32 pm

Hi Gostev,

I can confirm the scale out repository doesn't work as intended by Veeam atleast in my case. I experienced the same issue where 1 extent got full and the job failed instead of redirecting it to another extent which had more than 20TB free space.

foggy
Veeam Software
Posts: 18366
Liked: 1576 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by foggy » Oct 06, 2016 12:33 pm

Do you have Update 2 installed?

prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by prabhash.jena » Oct 06, 2016 1:38 pm

Yes. 9.0.0.1715. I am kind of early bird with application & system updates.

foggy
Veeam Software
Posts: 18366
Liked: 1576 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by foggy » Oct 06, 2016 1:40 pm

Then involving support is recommended to investigate the reasons of this behavior.

pirx
Enthusiast
Posts: 72
Liked: 9 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by pirx » Feb 04, 2017 9:46 am

Running 9.5U1 here and see the same problems.

Error: No scale-out repository extents have sufficient disk space to store the backup file.

There are 58TB of 100TB free. Seems that SOBR is not working as expected at all.

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by Gostev » Feb 04, 2017 5:40 pm

@Pirx what is the support case ID for this? All known issues around SOBR have been fixed in 9.5, so this must be something different.

pirx
Enthusiast
Posts: 72
Liked: 9 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: Scale-out Repository not working as it should - 00464187

Post by pirx » Feb 06, 2017 9:22 am

#02059754

Code: Select all

Please note that we try to allocate the space on the extents at the very beginning of the job. The space is allocated according to the following criteria : full backup - 50% of the VM size, increment - 10%. Sure it could be plenty of free space but once all the jobs are started at the same time, it is possible that all the space is allocated and that's why all the jobs will fail. For such kind of situations we recommend two registry keys that will reduce the percentage of the space allocation : Name: SOBRFullCompressRate Type: REG_DWORD Value: 25 Name: SOBRIncrementCompressRate Type: REG_DWORD Value: 5 Both keys should be added at the veeam server in HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ Please add both keys, restart the jobs and let me know about the results
All full jobs are finished now, we started them one after the other. I'll see if I can reproduce this with the given regkey settings. We use 14 primary restore points on this repository, but the backup copy repositories are SOBR too, so we should see the same effect (maybe later because our copy SOBR's are bigger because they must hold backups for 10 weeks).

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 9 guests