Comprehensive data protection for all workloads
Post Reply
christiankelly
Service Provider
Posts: 128
Liked: 11 times
Joined: May 06, 2012 6:22 pm
Full Name: Christian Kelly
Contact:

Anyone seen instability in a SOBR?

Post by christiankelly »

I've had a stable SOBR for about 6 months (4 50TB Dell servers) and a few nights ago some of the 40 backup copy jobs I have targeting the SOBR started getting the error "Unable to allocate processing resources. Error: No scale-out repository extents are available"

It seems somewhat random as jobs which have failed with the error sometimes then succeed on the following cycle, but overall the same jobs seem to be generally failing. I have rescanned the SOBR and it's seeing all the servers with no errors. I have also logged in to each server and they all seem stable.

I uploaded about a 1GB of logs to support, and it's early in the trouble shooting process but I was wondering if anyone has seen a similar issue.

Case: 01999793

PS: I opened this case under P1 and got a call back in about 30m saying that they couldn't keep the case at P1 as it was a backup copy job so it was moved to P2. While I understand that you have to prioritize, backup copy jobs which aren't running can be a very critical issue. In our case we have a datacenter with only a few days of local backups and then backup copy jobs with longer retention being pushed to another datacenter daily. While we're ok for another day or so this will quickly become critical for us.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Anyone seen instability in a SOBR?

Post by veremin »

Hi, Christian,

I still see the case as severity 1 ticket. However, I will double check that internally.

Thanks.
christiankelly
Service Provider
Posts: 128
Liked: 11 times
Joined: May 06, 2012 6:22 pm
Full Name: Christian Kelly
Contact:

Re: Anyone seen instability in a SOBR?

Post by christiankelly »

Thanks so much.

So far the recommendation from support has been to set the registry key DataMoverLocalFastPath to 1 as they saw this on the target repository logs:

Code: Select all

[09.12.2016 08:36:55] <   580> cli| Attempting to accept shared memory connection. Failed.
[09.12.2016 08:36:55] <   580> cli| WARN|The parameter is incorrect.
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Error code: 0x00000057
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Failed to open process [2824] handle.
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Failed to create shared memory IO device.
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Failed to accept shared memory connection.
The odd thing is I see this same error on jobs that are working without the error. Anyway I guess we'll see if it keeps happening.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Anyone seen instability in a SOBR?

Post by Gostev »

christiankelly wrote:Failed to open process [2824] handle.
Looks like interference of some antivirus? That's most generic Windows API function failing over there, so this cannot be a Veeam bug.
Sure, going over network stack via loopback interface instead will likely solve this, but this is obviously not great for performance...
christiankelly
Service Provider
Posts: 128
Liked: 11 times
Joined: May 06, 2012 6:22 pm
Full Name: Christian Kelly
Contact:

Re: Anyone seen instability in a SOBR?

Post by christiankelly »

That's the odd thing. These servers have Windows on them and nothing else. Nothings changed in on them for months. But I will look closely to see if something changed in the last week.
Filippo1975
Service Provider
Posts: 21
Liked: never
Joined: May 11, 2018 3:23 pm
Full Name: Filippo M
Contact:

Re: Anyone seen instability in a SOBR?

Post by Filippo1975 »

Hello there,
even this is an old post, we have find the same issue on ours infrastracture: today on 2021 January the 10, with both veeam cloud connect and veeam backup replication at version 10 update a, we get exactly the same issue.
We have just the veeam installed on the mount server and even on cloud connect server, there so antivirus installed..
the os installed on mount server is windows server 2019
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Anyone seen instability in a SOBR?

Post by Gostev »

Please open a support case. Thanks!
Filippo1975
Service Provider
Posts: 21
Liked: never
Joined: May 11, 2018 3:23 pm
Full Name: Filippo M
Contact:

Re: Anyone seen instability in a SOBR?

Post by Filippo1975 »

Hello Gostev!
we have already open the ticket.. it's the Case # 04581469... The support's guy wrote us it's could be due to insufficient resources, but we have already increase this,and even this the issue is still here..
We have disabled all featuers of the embedeed windows defender...
what could we do ?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Anyone seen instability in a SOBR?

Post by Gostev »

Filippo, as you can see from the age of this topic, this is not a known issue experienced by multiple customers, so it must be environment-specific. We cannot troubleshoot such issues over forum posts, so you will need to continue working with support. I expect your support engineer to give you the next troubleshooting steps based on the most recent logs, or escalate the case to a higher support tier (in case she can't think of anything else and needs help). Thanks!
Post Reply

Who is online

Users browsing this forum: Bing [Bot], ilisousou123 and 173 guests