Unable to allocate processing resources. Error: Failed to create production checkpoint.

[andreas2012]

Hi,

I have configured a veeam server for a new hyper-v cluster with 2 node. Windows Server 2022
When we try to backup a domain controller we get the error below, all other machines seems to work ok.
Suggestion to why this fails ? Application aware is reporting ok when we test.

09:17:31 Queued for processing at 30.11.2021 09:17:31
09:17:32 Required backup infrastructure resources have been assigned
09:17:36 VM processing started at 30.11.2021 09:17:36
09:17:36 VM size: 100 GB (22,3 GB used)
09:17:36 VM is now in the required state for processing 00:00
09:17:38 Preparing to create snapshot
09:17:43 Using guest interaction proxy servername (Same subnet)
09:17:49 Inventorying guest system 00:02
09:17:51 Subscribing to guest processing components 00:00
09:17:52 Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Failed to create VM (ID: 4ec250df-18c7-4d5b-8f23-9ed43dd0d5a8) recovery checkpoint. Job failed ('Checkpoint operation for 'domaincontroler2' failed. (Virtual machine ID 4EC250DF-18C7-4D5B-8F23-9ED43DD0D5A8) 'domaincontroller2' could not initiate a checkpoint operation: %%2147754996 (0x800423F4). (Virtual machine ID 4EC250DF-18C7-4D5B-8F23-9ED43DD0D5A8)'). Error code: '32768'. 00:26
09:18:19 Transaction logs will not be truncated 00:03
09:18:26 Retrying snapshot creation attempt (Failed to create production checkpoint.)
09:18:27 Task has been rescheduled
09:18:27 Queued for processing at 30.11.2021 09:18:27 00:11
09:18:41 Unable to allocate processing resources. Error: Failed to create production checkpoint.
09:18:41 Processing finished with errors at 30.11.2021 09:18:41


Thanks for any reply.


/R
Andy

[Mildur]

Hi Andreas

Please try to create a vm checkpoint by yourself in the hyperv manager.
Veeam is not responsible to create the checkpoint. Veeam gives the order to create the checkpoint to the hyperv server. If it‘s working in the hyperv console, please open a veeam support case. This is needed for any technical issue before posting it to the R&D forums. Thanks.

If the checkpoint is not working in the hyperv console, please check the vss writer state in the guest vm. If any vss writer is failed, you will not be able to create a checkpoint.

[andreas2012]

Hi,

Case: 05157792

We are trying to move a customer away from backup exec, so we have installed a trial version for demo, so I guess the support reply is slow.

What we have done....

I was able to create a checkpoint in hyper-v manager after we changed the vm checkpoints from Production to Standard. When I try from Veeam I can see that it starts to create a checkpoint in hyper-v manager but fails after 10-15% with the same error message I posted. We have tried to reboot the vm, also changed within the cm Shadow Copies > Settings Maximum size to No limit.

With vssadmin list writers I get this error
******************************************************************************************
Writer name: 'NTDS'
Writer Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Instance Id: {96ef30b5-acda-4b28-a2fa-1561c051eb93}
State: [11] Failed
Last error: Non-retryable error
******************************************************************************************

Then we did a vsstrace, it created a very long logfile that we are not able to debug. It contains a lot of customer data so will not publish it here.

From within the vm the VSS reports following
******************************************************************************************
Volume Shadow Copy Service error: Unexpected error calling routine RegOpenKeyExW(-2147483646,SYSTEM\CurrentControlSet\Services\VSS\Diag,...). hr = 0x80070005, Access is denied.
.

Operation:
Initializing Writer

Context:
Writer Class Id: {35e81631-13e1-48db-97fc-d5bc721bb18a}
Writer Name: NPS VSS Writer
Writer Instance ID: {b3410256-ebe4-49e5-9a36-e27c9d7dbdd6}
******************************************************************************************

And also the following
******************************************************************************************
A VSS writer has rejected an event with error 0x800423f4, The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

Operation:
PostSnapshot Event

Context:
Execution Context: Writer
Writer Class Id: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
Writer Name: NTDS
Writer Instance ID: {96ef30b5-acda-4b28-a2fa-1561c051eb93}
Command Line: C:\Windows\system32\lsass.exe
Process ID: 580
******************************************************************************************
The VSS writer NTDS failed with status 11 and writer specific failure code 0x800423F4.
******************************************************************************************

Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers I have the following
https://ibb.co/qYRVNd3
Dont know if i should try to delete some of them ?

Thanks for any advice, so we can have the customer buy Veeam instead of Backup Exec

/R
Andy

[PetrM]

Hi Andreas,

Have a look at this KB which tells about the similar error code 0x800423f4, probably some troubleshooting steps might be useful for your case as well. Also, did you check that another backup application also leverages VSS during backup of this VM?

Thanks!

[andreas2012]

Hi,

thanks for reply.
I have looked at the KB, and every step recommended, but the last step is VSS Trace and that log file is big and we would need help to debug it.

No other backup is running, I have also uninstalled Backup Exec, but was not sure about all the providers, ref url I provided.

........

I was no able to run the backup job successfully, and for that I had to follow these steps on the VM

1. Restart Com+
2. Restart Volume Shadow Copy
3. Restart hyper-v volume shadow copy requestor
4. Run the job = OK!

But this is not a solution, its a workaround.... comments ?

/R
Andy

[PetrM]

I believe that the result of this test will be very useful for our support engineers. Also, I'd suggest to monitor the job for a while, maybe the issue will not reoccur.

Since the job is running fine after restart of the mentioned services, I assume that the problem does not come from our code and perhaps it would make sense to involve Microsoft support as well. Anyway, let our support team to come up with an appropriate action plan for troubleshooting.

Thanks!

[andreas2012]

Hi,

The problem reoccur if we reboot the vm. Then we have to restart the services in the correct order.
I will wait for the support team. Thanks for answers.

[Stroytech]

Hi,
Have you received any solution from the MS Support Team?

[Mildur]

Hi Ilya

I suggest opening a case with our support team. It doesn't have to be the same cause in your case.
Can you create checkpoints manually in the HyperV console?

Please provide us with the case number as requested when posting a technical issue (forum rules). Without any case number, your comment maybe be deleted by an administrator.

Thank you
Fabian

[Stroytech]

Hi Fabian.
I didn't open a case because I have exactly the same problem. I couldn't create checkpoints manually in HyperV console but after doing as recomended here:
1. Restart Com+
2. Restart Volume Shadow Copy
3. Restart hyper-v volume shadow copy requestor
it's became possible to do that manually and automatically.
So it seems like the problem is in some MS stuff and I just wanted to ask if they gave the topicstarter some solution.

[Mildur]

Hi Ilya

Thanks for providing the solution for your case.
I understand that it is the same problem. But it doesn't have to be the same cause.
That's why we ask for a support case for such technical issues.

Thank you
Fabian

[HelpMe]

Did anyone got an solution for this?

[PetrM]

Hello,

Please refer to the post above to get a better idea of the error and open a support case for a detailed technical analysis. Also, please paste your support case ID over here for our reference.

Thanks!

[HelpMe]

Okay so i opend a new Case #05682858. Btw if i create a Checkpoint via the HyperV Manager it works absolutley fine. No problems and it only needs like in 10 seconds.