Anyone seen instability in a SOBR?

Availability for the Always-On Enterprise

Anyone seen instability in a SOBR?

Veeam Logoby christiankelly » Fri Dec 09, 2016 1:20 am

I've had a stable SOBR for about 6 months (4 50TB Dell servers) and a few nights ago some of the 40 backup copy jobs I have targeting the SOBR started getting the error "Unable to allocate processing resources. Error: No scale-out repository extents are available"

It seems somewhat random as jobs which have failed with the error sometimes then succeed on the following cycle, but overall the same jobs seem to be generally failing. I have rescanned the SOBR and it's seeing all the servers with no errors. I have also logged in to each server and they all seem stable.

I uploaded about a 1GB of logs to support, and it's early in the trouble shooting process but I was wondering if anyone has seen a similar issue.

Case: 01999793

PS: I opened this case under P1 and got a call back in about 30m saying that they couldn't keep the case at P1 as it was a backup copy job so it was moved to P2. While I understand that you have to prioritize, backup copy jobs which aren't running can be a very critical issue. In our case we have a datacenter with only a few days of local backups and then backup copy jobs with longer retention being pushed to another datacenter daily. While we're ok for another day or so this will quickly become critical for us.
christiankelly
Service Provider
 
Posts: 115
Liked: 7 times
Joined: Sun May 06, 2012 6:22 pm
Full Name: Christian Kelly

Re: Anyone seen instability in a SOBR?

Veeam Logoby v.Eremin » Fri Dec 09, 2016 5:28 pm

Hi, Christian,

I still see the case as severity 1 ticket. However, I will double check that internally.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Anyone seen instability in a SOBR?

Veeam Logoby christiankelly » Fri Dec 09, 2016 6:25 pm

Thanks so much.

So far the recommendation from support has been to set the registry key DataMoverLocalFastPath to 1 as they saw this on the target repository logs:

Code: Select all
[09.12.2016 08:36:55] <   580> cli| Attempting to accept shared memory connection. Failed.
[09.12.2016 08:36:55] <   580> cli| WARN|The parameter is incorrect.
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Error code: 0x00000057
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Failed to open process [2824] handle.
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Failed to create shared memory IO device.
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Failed to accept shared memory connection.


The odd thing is I see this same error on jobs that are working without the error. Anyway I guess we'll see if it keeps happening.
christiankelly
Service Provider
 
Posts: 115
Liked: 7 times
Joined: Sun May 06, 2012 6:22 pm
Full Name: Christian Kelly

Re: Anyone seen instability in a SOBR?

Veeam Logoby Gostev » Sat Dec 10, 2016 6:44 pm

christiankelly wrote:Failed to open process [2824] handle.

Looks like interference of some antivirus? That's most generic Windows API function failing over there, so this cannot be a Veeam bug.
Sure, going over network stack via loopback interface instead will likely solve this, but this is obviously not great for performance...
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Anyone seen instability in a SOBR?

Veeam Logoby christiankelly » Sun Dec 11, 2016 9:05 am

That's the odd thing. These servers have Windows on them and nothing else. Nothings changed in on them for months. But I will look closely to see if something changed in the last week.
christiankelly
Service Provider
 
Posts: 115
Liked: 7 times
Joined: Sun May 06, 2012 6:22 pm
Full Name: Christian Kelly


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Yahoo [Bot] and 37 guests