Unable to backup VM (VSS_WS_FAILED_AT_PREPARE_SNAPSHOT)

[turibbio]

Hi everyone,
I'm using Veeam B&R to backup VMs on a server. Specifically, I use Veeam cmdlets inside a script that execute Start-VBRZip such as the following command:

$result = Start-VBRZip -Entity $currentVmEntity -Compression 9 -Folder $destinationFolder -AutoDelete In3days

But sometimes, backup fail with the following error:
Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a transient error. If the backup process is retried,
the error may not reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.

If I re-execute manually the task by Veeam B&R IDE or using the previous command and adding -DisableQuiesce parameter, backups work.

Here my questions:
1)I've read something about this error, but I'm unable to find the root that cause backups to fail (also, I don't have iSCSI enabled). So, how can I really identify the main problem?

2)I'm not familiar with the quiesce concept (I'm basically a developer, not an ops guy!), but reading articles on Veeam site, if I'm not mistaken, disabling quiesce allow you to backup a VM without any downtime, but with data loss, because you don't backup any operation that your vm is executing. Am I correct? If so, when I should use the disable quiesce parameter?

Thanks in advance for your help.

[Mike Resseler]

Hi,

As you have guessed correctly, the root cause of the backups can be (a lot) of different things. What happens is that the VSS writer of Hyper-V does not succeed in creating a backup because one of the underlying components has failed. This could be a VSS component inside a VM, not enough space in that VM to take a snapshot and so on and so forth... So time to troubleshoot

1. Does this happen with every VM or some VMs? Try to run a backup for each VM separately to find the difficult VM
2. Look at event viewer, both on the host and on the VM that fails. You need to search for VSS related errors
3. Check if one of your VMs is in a hold state (which means there are snapshots left and it fails to "merge" the snapshot) This mostly can be resolved by restarting the service Hyper-V Virtual Machine Management
4. Run a VSSadmin list writers inside your VMs... It could be that your Hyper-V IC Software Shadow Copy Provider is broken or in a failed state or so... Rebooting does not always help so you sometimes need to re-initiate it (see below) to get it running. (The VSS related errors above should give you an idea if this is the case or not...)

About quiesce... Because you disable this, B&R will take a crash-consistant snapshot. This means it will backup the VM as is, without telling it first to "prepare for a backup". This might work but open files can cause an issue. For example, when there is a database such as SQL running on it, recovering it can mean the data is corrupt. VSS and quiescence means that you prepare the VM so it is backed up in an consistent state (Quiescence for file-level consistency, VSS additionally for application-aware backup by utilizing the inside VSS writers of the VM...)

Let us know what you find so we can see if we can try to resolve it. (PS: In most cases, VSS errors means that you are also missing updates / patches or sometimes that something is bad and MSFT support needs to get involved, but we will try our best to help where we can...)

Cheers
Mike

[Mike Resseler]

Sorry.... My bad, I should not be writing troubleshooting tips early Sunday morning without my coffee

I promised you the troubleshooting tip for the hold Hyper-V IC Software Shadow Copy Provider and forgot to add it at the end

If you find the VM that caused the issue and see the related errors in event log, you can try the following:

1. Do a vssadmin list writers. You should see the Hyper-V IC Software Shadow Copy Provider there. Copy the Writer instance ID of it, should be something like: {74600e39-7dc5-4567-a03b-f091d6c7b092}
2. Find the registry key for the ID and delete that registry key (but export first, just in case)... should be something like:

Code: Select all

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\{74600e39-7dc5-4567-a03b-f091d6c7b092}]
@="Hyper-V IC Software Shadow Copy Provider"
"Type"=dword:00000002
"Version"="1.0.0.0"
"VersionId"="{c797fff9-2c3e-453c-a3e9-cc7673a6c732}"

3. Restart the VM... It should reinitialize the shadow copy provider and (hopefully) fix the error

Mike