Anyone seen instability in a SOBR?

[christiankelly]

I've had a stable SOBR for about 6 months (4 50TB Dell servers) and a few nights ago some of the 40 backup copy jobs I have targeting the SOBR started getting the error "Unable to allocate processing resources. Error: No scale-out repository extents are available"

It seems somewhat random as jobs which have failed with the error sometimes then succeed on the following cycle, but overall the same jobs seem to be generally failing. I have rescanned the SOBR and it's seeing all the servers with no errors. I have also logged in to each server and they all seem stable.

I uploaded about a 1GB of logs to support, and it's early in the trouble shooting process but I was wondering if anyone has seen a similar issue.

Case: 01999793

PS: I opened this case under P1 and got a call back in about 30m saying that they couldn't keep the case at P1 as it was a backup copy job so it was moved to P2. While I understand that you have to prioritize, backup copy jobs which aren't running can be a very critical issue. In our case we have a datacenter with only a few days of local backups and then backup copy jobs with longer retention being pushed to another datacenter daily. While we're ok for another day or so this will quickly become critical for us.

[veremin]

Hi, Christian,

I still see the case as severity 1 ticket. However, I will double check that internally.

Thanks.

[christiankelly]

Thanks so much.

So far the recommendation from support has been to set the registry key DataMoverLocalFastPath to 1 as they saw this on the target repository logs:

Code: Select all

[09.12.2016 08:36:55] <   580> cli| Attempting to accept shared memory connection. Failed.
[09.12.2016 08:36:55] <   580> cli| WARN|The parameter is incorrect.
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Error code: 0x00000057
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Failed to open process [2824] handle.
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Failed to create shared memory IO device.
[09.12.2016 08:36:55] <   580> cli| >>  |--tr:Failed to accept shared memory connection.

The odd thing is I see this same error on jobs that are working without the error. Anyway I guess we'll see if it keeps happening.

[Gostev]

Failed to open process [2824] handle.

Looks like interference of some antivirus? That's most generic Windows API function failing over there, so this cannot be a Veeam bug.
Sure, going over network stack via loopback interface instead will likely solve this, but this is obviously not great for performance...

[christiankelly]

That's the odd thing. These servers have Windows on them and nothing else. Nothings changed in on them for months. But I will look closely to see if something changed in the last week.

[Filippo1975]

Hello there,
even this is an old post, we have find the same issue on ours infrastracture: today on 2021 January the 10, with both veeam cloud connect and veeam backup replication at version 10 update a, we get exactly the same issue.
We have just the veeam installed on the mount server and even on cloud connect server, there so antivirus installed..
the os installed on mount server is windows server 2019

[Gostev]

Please open a support case. Thanks!

[Filippo1975]

Hello Gostev!
we have already open the ticket.. it's the Case # 04581469... The support's guy wrote us it's could be due to insufficient resources, but we have already increase this,and even this the issue is still here..
We have disabled all featuers of the embedeed windows defender...
what could we do ?

[Gostev]

Filippo, as you can see from the age of this topic, this is not a known issue experienced by multiple customers, so it must be environment-specific. We cannot troubleshoot such issues over forum posts, so you will need to continue working with support. I expect your support engineer to give you the next troubleshooting steps based on the most recent logs, or escalate the case to a higher support tier (in case she can't think of anything else and needs help). Thanks!