Unexpected workaround to snapshot failure

[richw]

I have been struggling to get one VM backed up due to a common Hyper-V snapshot error, and found an unexpected way of getting the snapshot to work - sort of.
The backup job has NO application aware enabled, but the job constantly failed with errors below.
The job failed to create a checkpoint but Checkpoints can be created OK with Hyper-V manager.
The host VSS writer is in a failed state with Non-retryable error. The guest VSS writers are all stable no error. Event viewer on the guest has VSS event 12293 for each attempt.

I decided to enable application aware settings, with ignore errors option and truncate transaction logs. (the VM contains a SQL express DB)
With those settings, the Hyper-V checkpoint succeeds, a backup is created, a restore point is registered, the jobs finished with a warning status, but the exact same warnings are seen in the job log, saying unable to create snapshot.

This makes no sense: why is the guest VSS processing happening when not enabled in the job? why does enabling guest level VSS allow the host to create a checkpoint?

Guest VM event VSS 12293
Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details BeginPrepareSnapshot({ee69cf51-e893-4651-897a-eb0908e9d37a},{3c7d207a-e62f-4c70-bb99-e2bea8a62ad7},\\?\Volume{2925ed8b-dfcd-44b5-a128-4e0503e0c78a}\) [hr = 0x80070020, The process cannot access the file because it is being used by another process.
].

Operation:
Add a Volume to a Shadow Copy Set

Context:
Execution Context: Coordinator


Veeam backup job errors

Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Hyper-V child partition snapshot). Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.

Make sure VM does not have 'iSCSI Software Target Storage Provider' feature installed.

Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.

[PTide]

Hi,

Do you have support case opened? If not then kindly open one and post case ID here.

Thank you.

[alanbolte]

(mode: Hyper-V child partition snapshot)

If you had guest processing completely disabled you would see "mode: Crash consistent." "Hyper-V child partition snapshot" means you have the "guest quiescence" option enabled. I have not found any use cases for this feature on Windows VMs except for certain troubleshooting scenarios: AAIP is always the better option, and enabling AAIP overrides the "guest quiescence" option.

With your previous settings, there was no mechanism to failover to crash consistent backup. Your current settings are still failing to create an application-aware shadow, but are allowing failover to crash-consistent mode.

I would also like to clarify that while the ability to create a checkpoint is required for guest processing, it is not sufficient; error messages about a failure to create a checkpoint in such a backup usually indicate either inability of the VM to create a shadow copy of its own volumes, or a problem of communication between the Hyper-V host and the VM's VSS components.

[richw]

Support case 01082702
thanks

[richw]

Hi Alan
I do have Hyper-V guest quiescence enabled you are right. The purpose is to enable "Take crash consistent backup instead of suspending VM". I have had VM's suspend for a long time before, which causes a lot of customer dissatisfaction. Veeam support have recommended this option to prevent it.
There are 20 VM's in this job, and application aware is only enabled on the VM's that actually require it, to minimise the amount of VSS issues, which in my experience is often troublesome on Hyper-V.
I suppose I could remove this particular VM from the job and put it alone in a job with different settings.

[alanbolte]

You could, though in most circumstances that offers no clear advantage over the per-VM AAIP settings within a job. However, perhaps support team will be able to help you find the reason for AAIP failing. I see they've requested a log bundle; additional logs they may find useful are described here and here. Another useful isolation step is to try a manual shadow copy within the VM, such as with certain settings in Windows Server Backup, or using Diskshadow - this can help identify whether the problem is specific to the Hyper-V VSS provider or is general to in-guest VSS, and can also help identify whether there's a specific volume within the VM that's causing a problem (the error you posted would lead me to think the problem is specific to \\?\Volume{2925ed8b-dfcd-44b5-a128-4e0503e0c78a}\, whichever volume that might be).

My understanding of the 'instead of suspending' option was that it should only apply if the guest quiescence option is already enabled. The Hyper-V VSS writer is designed to try to use a saved state if true online backup isn't possible, but our AAIP is supposed to fail rather than suspend the VM. I believe there was a known issue with AAIP sometimes using saved state during replication resolved in v8 Update 2.