Failed to obtain resource availability

[UBC]

What does this error message mean? This is the server has only replication jobs running and performs no other functions other than running the veeam replication jobs but maybe once a week the jobs will start to fail with this message and restarting the server fixes it. The server that it says is the server running the replication jobs.

Code: Select all

Unable to allocate processing resources. Error: Failed to obtain resource availability: servername

[foggy]

Could you please describe your setup in a bit more detail? Do you mean this server is a Veeam B&R backup server itself that also plays the role of a proxy server? What proxy selection/transport mode settings do your jobs have? Have you contacted support with this issue? Thanks.

[UBC]

This server is a Veeam B&R server that just has replication and surebackup jobs running and this server is a proxy. The source and target proxy for the jobs are setup as automatic and no I haven't contacted support yet. We are just upgraded to version 9 but it has done this on the previous version and current version.

[foggy]

I recommend you to do that, job logs will tell the reason of the proxy unavailability.

[bdoe]

This is in reference to case 01842538. Veeam is installed on a physical Server 2012 machine, and Veeam is up to date. The server is a 16C/32T, 64GB system with 10GbE, and Veeam has access to several Linux repositories. While this has been going on, I haven't seen any indication of the repositories being offline or any services stopping.

For the past week or two, all of my B&R jobs have hung with the following error: "Unable to allocate processing resources. Error: Failed to obtain resource availability: veeam.company.com". Veeam.company.com is the physical system. The only jobs are Backup jobs, a mix of incremental and reverse incremental; there is no replication involved. Endpoint jobs start to pile up as well, although it seems they eventually trickle through, although they've often completed 12 hours later than normal. As far as I could see, this started two weeks ago, for no obvious reason. After a few reboots of the system, jobs began to run correctly again, until it started up again two days ago.

To go along with this, there are apparently database issues. My SureBackup job, which should run nightly, is not running on schedule, and weekend tape jobs are not either. I can manually start tape jobs, but the SureBackup job won't start and gives me an error.

I was wondering if anyone else had seen such an issue. I've been out this week which is making support a bit more difficult, unfortunately.

[foggy]

Bryan, is this physical machine the only proxy server in your environment? Job logs should contain more information about the reason of its unavailability, so please continue investigating with support.

[bdoe]

Yes, Veeam is only on one server, which is also where the only proxy is located.

[Daveyd]

Has this been resolved. I cam in this morning to see 15 Backup jobs are sitting in Pending status with Queued for processing. The majority of the jobs reference 1 proxy server - Failed to obtain resource availability. I have many proxy servers so I am not sure why all 15 jobs did not complete. They all have been completing successfully for a long time

[foggy]

Were all other proxies available during the job duration? What transport mode they typically use to retrieve VM data?

[Daveyd]

I have around 10 proxies and they were (and still are) available. It looks like the proxy mentioned in the error is showing as unavailable but not sure that should have caused all the other job issues. I typically have all the VM proxies set to Automatic selection under Transport mode with the exception of the Veeam server which has FC HBAs zoned into the storage fabric and also acts as the tape backup server with a FC HBA zoned into the FC Tape library

[foggy]

Dave, please contact technical support for the logs review. Logs should tell the actual reason for such proxy selection behavior and why Veeam B&R considered proxies unavailable.

[Daveyd]

I created a ticket shortly after I posted here

[foggy]

I appreciate you sharing its number here for further reference. Thanks!

[Daveyd]

Veeam Support - Case # 01976631

Interestingly enough it looks like "some" of the backup job are still processing - been over 15 hours! While the others are waiting around. The ones that are waiting around have some VMs in them that failed with the above error and others that are Pending. Odd.

[bgutwein]

Veeam Support - Case # 01976803

Same boat here. Came in one morning and all my jobs were failing waiting for infrastructure availability. No amount of rebooting seems to help. I have 4 virtual proxy servers and none of them work. Even tried manually assigning the job to the built in proxy on the Veeam server itself and no dice. I can't think of any updates that I installed recently that would have caused this behavior. Veeam doesn't update software in the background does it? It seems very strange that it all of a sudden started happening.

[Daveyd]

My "solution" was to reboot the Proxy that went unavailable in the Veeam console. Once that Proxy became available in Veeam's eyes all the jobs started SLOWLY processing again, which most had been "running" for 15 hours. This makes no sense since I have multiple Proxies and most of the SAN based jobs that only use the Veeam server as a proxy were failing as well. So far Support has really been no help saying that I may need to up the concurrent tasks to the max that the Proxy will take and consider upping the RAM the Proxy that was having issues? Huh?

[foggy]

I recommend escalating the case to investigate the actual reasons of the observed behavior.

[masonit]

We had the same problem. One of many proxy servers went offline during backup window. Left alot of jobs hanging. Cancelled all jobs and also rebooted vbr server. After that all jobs we tried said: Waiting for backup infrastructure resources availability. Even though none of the jobs are using the unavailalbe proxy. When proxy server was online again all jobs started working again..

Case: 01992017

\Masonit

[timgowen]

I'm testing Veeam and have noted that every few weeks I get a complete failure of my backups

Code: Select all

Extensis TestError7:00:14 PM8:30:35 PM0.0 B0.0 B0.0 B1:30:21 Unable to allocate processing resources. Error: Failed to obtain resource availability: Park 
LON-XIBO-HOSTError7:00:14 PM8:00:34 PM0.0 B0.0 B0.0 B1:00:20 Unable to allocate processing resources. Error: Failed to obtain resource availability: Dowding 
L-VM-00002Error7:00:13 PM7:15:35 PM0.0 B0.0 B0.0 B0:15:22 Unable to allocate processing resources. Error: Failed to obtain resource availability: Dowding 
New Virtual MachineError7:00:14 PM9:45:35 PM0.0 B0.0 B0.0 B2:45:21 Unable to allocate processing resources. Error: Failed to obtain resource availability: Park 
RAFM-DC-01Error7:00:14 PM8:15:32 PM0.0 B0.0 B0.0 B1:15:18 Unable to allocate processing resources. Error: Failed to obtain resource availability: Park 
RAFM-DC-02Error7:00:14 PM7:30:33 PM0.0 B0.0 B0.0 B0:30:19 Unable to allocate processing resources. Error: Failed to obtain resource availability: Dowding 
TRAPS Virtual MachineError7:00:14 PM8:45:33 PM0.0 B0.0 B0.0 B1:45:19 Unable to allocate processing resources. Error: Failed to obtain resource availability: Park 
TucanoError7:00:14 PM9:00:31 PM0.0 B0.0 B0.0 B2:00:17 Unable to allocate processing resources. Error: Failed to obtain resource availability: Park 
VERNON-TESTError7:00:14 PM7:45:31 PM0.0 B0.0 B0.0 B0:45:17 Unable to allocate processing resources. Error: Failed to obtain resource availability: Dowding 
Windows 10 Test ClientError7:00:14 PM9:30:32 PM0.0 B0.0 B0.0 B2:30:18 Unable to allocate processing resources. Error: Failed to obtain resource availability: Park 
Windows 7 Client for Vernon TestError7:00:14 PM9:15:34 PM0.0 B0.0 B0.0 B2:15:20 Unable to allocate processing resources. Error: Failed to obtain resource availability: Park 

I reboot the server (Windows Server 2012 R2) that's running Veeam, and re-run the backup - successfully.

Veeam Agent 'Backup Job JAGUAR' finished with Error and will be retried. These errors are logged for every failed server:

Code: Select all

Job details: Creating VSS snapshot Error: Failed to create snapshot: Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{477becb1-5133-408a-ab54-3e23e3ee0106}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].
Error: Failed to create snapshot: Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{477becb1-5133-408a-ab54-3e23e3ee0106}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].

I've seen other issues similar to this, but the fact that it recurs on more or less a monthly basis seems weird to me.

[PTide]

Hi,

Have you contacted our support team on the issue? If not then please do and post your case ID.

Thank you