Failed to create VM recovery checkpoint

[aj_potc]

I'm running the free edition of Backup and Replication 9.5 update 3a on a Windows Server 2016 Hyper-V host. I'm trying to back up two guests (both also Windows Server 2016).

One of the two guests is intermittently throwing the following errors:

Code: Select all

Failed to create VM recovery checkpoint (mode: Hyper-V child partition snapshot) Details: Job failed ('Checkpoint operation for VM failed.

Code: Select all

Unable to allocate processing resources. Error: Job failed ('Checkpoint operation for 'VM' failed. 

Production checkpoints cannot be created for 'VM'. Error code: '32770'.
Failed to create VM recovery snapshot, VM ID '927335d9-05c4-4936-b25a-405fcb6fd0da'.  

In the Event Log of the Hyper-V host, I can also find the following entry when the backup process is attempted:

Code: Select all

Production checkpoints cannot be created for 'VM'. 

I've checked here on the forums and have found a number of reports of Hyper-V issues (mostly related to non-Server 2016 guests), but I haven't found any concrete solutions.

Does anyone know why I would have an issue with production checkpoints, and why this would cause Veeam to fail?

Thank you very much for any help.

[wishr]

Hi Aj_potc,

In certain circumstances, this issue can appear due to a bug in Hyper-V checkpoint creation processes. Could you please check if you are able to create a checkpoint for this machine using Hyper-V Manager? If this doesn't help I would recommend reaching out Microsoft support for assistance.

If you are be able to create a checkpoint manually using Hyper-V Manager, then here is a suggestion by one of the Veeam users that might be helpful, but I would definitely recommend you to raise a support case instead of digging into the issue on your own.

Thanks,
Fedor

[aj_potc]

Hi Fedor,

Thanks for your kind reply. I do indeed believe it to be a bug in Hyper-V -- I never really suspected Veeam was the problem.

Unfortunately I wasn't able to try your suggestion to create a checkpoint manually before I rebooted the VM. After the reboot, the backups now work correctly. While I'm happy that resolved the problem, I really hate that kind of "fix," because I have no idea where the real issue was, or if it will appear again. I thought that Windows Server 2016 and Hyper-V were very stable by now, but maybe not!

Thanks again for your help.

Best,

A.J.

[Mike Resseler]

A.J.

What I mostly see when this occurs, is that one of the VSS writers got stuck. A reboot then resolves it. It probably isn't even the Hyper-V writer but it could be the SQL writer or a 3rd party VSS writer. If this happens again, before rebooting, run vssadmin list writers (elevated command prompt) in the VM to see if one of the writers is stuck. Mostly that will give you a better view of what is causing issues

My 2 cents
Mike

[aj_potc]

Hi Mike and Fedor,

I've since had this issue come up again, and this time I've decided to dig into it more deeply. I've opened support case 03291211.

I have tried creating a checkpoint manually -- that works fine. I also tried Mike's suggestion of checking on the VSS writers in the VM itself. All of them report no errors.

I didn't try the suggestion in Fedor's link, as my guest is Server 2016, and this suggestion discusses taking files from a Server 2012 ISO.

Thanks for any further thoughts/ideas!

[wishr]

Hi AJ,

Thanks for replying.

This looks strange, though. Our support team will check the logs first and reach you back, we'll track it as well.

In the meanwhile, could you please make sure that your Guest OSs are updated to the latest version? Also, It might be useful to know which Integration Services version the failed VMs are running. There is a how-to guide on that by the aforementioned link.

Thanks

[aj_potc]

Thanks for your reply.

So far I haven't had much luck troubleshooting the problem.

I've checked the version of Integration Services on the Windows Server 2016 guest VM, and it shows:
10.0.14393.2007

I believe this should be the latest version, but it's not clear to me how that could be updated (maybe automatically via Windows Update?).

Thanks for any other help or feedback.

[wishr]

Hi AJ,

As far as I see the manual checkpoint creation operation suggested by our support team has failed with the same error message. In that case, we recommend reaching Microsoft Support for assistance - probably some specific reinstallations/reconfigurations will be required on the VM/Hyper-V end.

We will really appreciate if you could share the final solution by Microsoft that will be helpful, so everybody experiencing the same issue would be aware of it.

Thanks

[collinp]

Is the error 32768 and are you using smb?

[mark14]

Hi,

Currently we have Veeam Backup & Replication for Hyper-V and it is running as Virtual Machine on Hyper-V server. I want to backup linux virtual machine with MySQL database.

I just follow article:
https://helpcenter.veeam.com/docs/backu ... tml?ver=95

and create pre-freeze and post-thaw scripts:
https://bp.veeam.expert/applications/mysql

Backup Job has been created:
check enable applicaiton-aware processing
In Applications -> Scripts I added Pre-freeze and Post-thaw script
added guest os credentials (linux root credentials over 22 SSH)

But it is not working. Logs below:

machine ID A26BC399-7BEE-4172-A174-669AD0056C97) Production checkpoints cannot be created. Error code: '32770'. Failed to create VM recovery snapshot.
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Unable to allocate processing resources. Error: Job failed.Production checkpoints cannot be created.Error code: '32770'. Failed to create VM recovery snapshot.

[wishr]

Hi Mark,

Welcome to Veeam Community Forums and thank you for posting.

I've merged your post into a similar topic. Could you please take a quick look at it and let me know if you have any additional questions.

Thanks

[aj_potc]

I've made a little more progress researching the checkpoint error 32770.

On the Server 2016 guest VM, I noticed that there are a number of services related to Hyper-V. I suppose these are the Integration Services. All of them are set to Manual startup, and several were not running. One of these was:

Hyper-V Volume Shadow Copy Requestor

After I manually started the service, I could then create Hyper-V checkpoints and run Veeam backups as expected from the host system. Very interesting.

I suppose I'll have to investigate why this service was stopped, or how I can monitor it in the future.

Does anyone have any experience with this odd situation?

[wishr]

Hi AJ,

Thanks a LOT for sharing this. Personally, I haven't seen that before.

@Mark, could you please check your services state? Maybe your root cause is similar...

[mark14]

@wishr
As I said I am trying to backup Linux Virtual Machine (Debian 8.0 with MySQL). Do I need extra drivers/packages installed on machine? Also I can create now checkpoint Linux VM from Hyper-V

[wishr]

Hi Mark,

Hyper-V should embed all the required packages automatically in any guest OS. From Veeam perspective you do not require to install any additional packages either.

Could you please try to check the VM settings in Hyper-V and make sure that the checkbox called "Create standard checkpoints if it's not possible to create a production checkpoint" underneath the "Production checkpoints" option is unchecked (if not, please do so), then try to create a checkpoint manually one more time and let us know the result? Please do not forget to revert the setting back once done.

Thanks

[nmdange]

@wishr
As I said I am trying to backup Linux Virtual Machine (Debian 8.0 with MySQL). Do I need extra drivers/packages installed on machine? Also I can create now checkpoint Linux VM from Hyper-V

Debian 8.0 is listed as supported. There is a note around live VM backup about needing the hyperv-daemons package
https://docs.microsoft.com/en-us/window ... on-hyper-v

Starting with Debian 8.3 the manually-installed Debian package "hyperv-daemons" contains the key-value pair, fcopy, and VSS daemons. On Debian 7.x and 8.0-8.2 the hyperv-daemons package must come from Debian backports.

[aj_potc]

Just in case this helps others who may find this post in the future:

I've tracked back the failure of the service Hyper-V Volume Shadow Copy Requestor to a crash of svchost.exe related to ScDeviceEnum, which is Windows' smart card enumeration service. For reasons unknown, this service crashed and took out a number of other services. Since the Hyper-V Volume Shadow Copy Requestor is set to "Manual (Trigger Start)," it wasn't started back up after the crash. (By the way, does anybody know what the trigger should be for this service?)

Since I don't know what caused the crash of the smart card enumeration service and can't know whether it'll happen again, I've set the Hyper-V Volume Shadow Copy Requestor to Automatic start, which will hopefully restart it if svchost.exe goes down in the future.

[wishr]

Debian 8.0 is listed as supported. There is a note around live VM backup about needing the hyperv-daemons package
https://docs.microsoft.com/en-us/window ... on-hyper-v

Hi,

Not sure this can affect the checkpoint creation. The first thing everybody should give a try in order to narrow down the potential root cause area is to try creating a checkpoint manually following the procedure I mentioned in my post above. This will help to identify whether the root cause lies on the Hyper-V end or not.

Hope this makes sense. Thanks.

@AJ, thank you for sharing additional details btw. Appreciate your efforts.

[mark14]

Hi Mark,

Hyper-V should embed all the required packages automatically in any guest OS. From Veeam perspective you do not require to install any additional packages either.

Could you please try to check the VM settings in Hyper-V and make sure that the checkbox called "Create standard checkpoints if it's not possible to create a production checkpoint" underneath the "Production checkpoints" option is unchecked (if not, please do so), then try to create a checkpoint manually one more time and let us know the result? Please do not forget to revert the setting back once done.

Thanks

"Create standard checkpoints if it's not possible to create a production checkpoint" underneath the "Production checkpoints" is checked and I am able to create checkpoint manually from Hyper-V (RMB on Linux Machine -> Create Checkpoint).

I think the case is with veeam configuration because when I turn off Guest OS, backup is creating successfully.

[wishr]

Hi Mark,

As I mentioned above you should uncheck it and try the manual checkpoint creation. Please share your result once done.

Thanks,
Fedor

[mark14]

My mistake, sorry

I uncheck it and after this I cannot create manually checkpoint from Hyper-V server. Also Veeam job is failed:

Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed ('Checkpoint operation(Virtual machine ID 5E51D908-CF18-4397-9025-6356B691BB00) Production checkpoints cannot be created'. (Virtual machine ID 5E51D908-CF18-4397-9025-6356B691BB00)'). Error code: '32770'. Failed to create VM recovery snapshot, VM ID '5e51d908-cf18-4397-9025-6356b691bb00'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 03.12.2018 13:14:12
Unable to allocate processing resources. Error: Job failed ('Checkpoint operation failed. (Virtual machine ID 5E51D908-CF18-4397-9025-6356B691BB00) Production checkpoints cannot be created fo. (Virtual machine ID 5E51D908-CF18-4397-9025-6356B691BB00)'). Error code: '32770'. Failed to create VM recovery snapshot, VM ID '5e51d908-cf18-4397-9025-6356b691bb00'.

[mark14]

BTW. I edited job, and click "Test Now" In guest processing.

https://ibb.co/5LFxVmJ

It is really weird, because address of Linux Machine is reachable, and accessible.

[wishr]

Mark,

Thank you for the information.

The inability to create a production checkpoint manually (that is also utilized by B&R while legacy checkpoints are not) means that there is some issue in the Hyper-V/Guest OS. We recommend reaching out to Microsoft support for a comprehensive investigation of this issue. We will really appreciate if you could post a solution they will advise you in this thread.

As for the job error shown in your screenshot, I believe it's a separate issue. There are some existing topics about that issue (or kind of a similar). For example, this one. If the existing information does not help you may contact our support team and open a case. Please do not forget to post your case ID here once you'll have it.

Thanks,
Fedor

[mark14]

But it is problem when option "Create standard checkpoints if its not possible to create a production checkpoint" is unchecked (and in default configuration this option is checked).

https://ibb.co/L8PVmw6

Don't you think that is veeam configuration problem? Also when I disable Guest-OS, and select above option I am able to create backup via Veeam so technically checkpoint works.

[wishr]

Mark,

B&R only utilize production checkpoints and does not use the standard legacy checkpoints at all in Hyper-V 2016 and higher, regardless of the aforementioned checkbox.

If you tick the checkbox (to be clear, it's called "Create standard checkpoints if it's not possible to create a production checkpoint"), then if a production checkpoint creation has failed Hyper-V will utilize the legacy checkpoint creation mechanism and it will succeed. That's why you are able to create a checkpoint manually when the checkmark is set (allow the legacy mechanism), but not able to do so when it is not set (only use the modern mechanism, disallow the legacy one if the production has failed). Basically, in this case, the production checkpoint creation does not work at all on the Hyper-V end for some reason - we confirmed it using the simple procedure above.

It's not on our end, unfortunately, otherwise, we'll be able to help quickly.

I believe there is some underlying aspect in the Hyper-V and guest OS interaction mechanisms related to some driver/service/package being not up to date/broken/not installed or so, but this should be checked with Microsoft directly.

Hope this makes sense.

Thank you

[mark14]

You were right Problem has been solved. After reinstall "hyperv-daemons" package on linux virtual machine and reboot now I can create production checkpoint, and guest processing is working.

[chuck.coggins]

Support Case ID # 03327805

I am getting an error on 3 of my VM backups.
---------

Code: Select all

12/12/2018 11:30:39 PM :: Failed to create VM recovery checkpoint (mode: Hyper-V child partition snapshot) Details: Job failed ('Checkpoint operation for 'S2018HAProxy1' failed. (Virtual machine ID E21DAC9D-AAD0-4CFC-BAC1-83076DCA8AE7) Production checkpoints cannot be created for 'S2018HAProxy1'. (Virtual machine ID E21DAC9D-AAD0-4CFC-BAC1-83076DCA8AE7)'). Error code: '32770'. Failed to create VM recovery snapshot, VM ID 'e21dac9d-aad0-4cfc-bac1-83076dca8ae7'. 

--------
I have not been able to find anything by searching to fix my issue. I have 7 servers that all use HAProxy for load balancing of the 7 servers 3 of them will not backup my HAProxy Hyper-V VM (HAProxy is running FreeBSD). There are 2 differences between the 4 that run it and the 3 that do not.

The 4 that will create the backup:
Running Windows Server 2012 R2
Hyper-V: Version 6.3.96

The 3 that will not:
Windows Server 2016
Hyper-V: Version 10.0.14393

Any guidance on why this is not working on my 2016 server would be wonderful. Also, I should note on the 2016 servers Veeam is backing up like 9 other VM's I have running on them. It just will not back up the HAProxy VM's running FreeBSD. Thank you!.

[wishr]

Hi Charles,

Welcome to Veeam Community forums and thanks for posting!

I've merged your thread with an existing one, could you please take a quick look - it seems your circumstances are the same.

Thank you

[2mphtijlc]

Also, when the task fails to the original Windows VSS policy, when the VW Provider fails, the HW still does not work. At first, it made me think of an error that resists presence in Windows Server 2016 Hyper-V, where any direct transfer of any kind could lead to a RCT break and a whole new solution was needed to fix it. This bug has been fixed for a long time and there was no full new backup that did not solve anything here. Now there are many reasons why setting up a checkpoint will not work so we need to dive deeper. As always, an Event Viewer is your friend. what do we see? 3 events during backup are related to SQL Server.

[Mike Resseler]

Liam,

First: Welcome to the forums
Second: A good way to figure out whether it is Veeam or another issue is to take a manual Hyper-V checkpoint, but the newest one (production). If that fails, the underlying issue is most likely related to the application itself