Discussions specific to the Microsoft Hyper-V hypervisor
Mike Resseler
Product Manager
Posts: 5446
Liked: 573 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Mike Resseler » Jan 11, 2018 6:22 am

Interested to learn what the result is Alex.

Yes, I agree it seems strange, but the fact that you have so many VMs (30%) immediately makes me think in the direction of the host.

M/

AlexLeadingEdge
Expert
Posts: 273
Liked: 32 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by AlexLeadingEdge » Jan 23, 2018 10:45 pm

Restarting the VSS writers on both the Host and the Guest 15 minutes before the backup has allowed the VSS writers to be picked up correctly. The problem is that it still errors, then at the retry it says the VSS writers are offline again :(

Code: Select all

23/01/2018 7:00:31 p.m. :: Queued for processing at 23/01/2018 7:00:31 p.m.  
23/01/2018 7:14:41 p.m. :: Preparing to create snapshot  
23/01/2018 7:15:53 p.m. :: Using guest interaction proxy BACKUPSVR (Same subnet)  
23/01/2018 7:15:56 p.m. :: Inventorying guest system  
23/01/2018 7:16:32 p.m. :: Subscribing to guest processing components  
23/01/2018 7:16:35 p.m. :: Creating snapshot Microsoft Software Shadow Copy provider 1.0 (mode: Veeam application-aware processing with failover)  
23/01/2018 7:19:51 p.m. :: Required backup infrastructure resources have been assigned  
23/01/2018 7:19:54 p.m. :: VM processing started at 23/01/2018 7:19:54 p.m.  
23/01/2018 7:19:54 p.m. :: VM size: 61.0 GB (52.8 GB used)  
23/01/2018 7:19:55 p.m. :: Truncating SQL server transaction logs  
[color=#FF0000]23/01/2018 7:19:59 p.m. :: Retrying snapshot creation attempt (Failed to create snapshot for vm b0bf5146-1e0c-4ba9-8e88-e606c2280ced)  [/color]
23/01/2018 7:19:59 p.m. :: Required backup infrastructure resources have been assigned  
23/01/2018 7:19:59 p.m. :: Required backup infrastructure resources have been assigned  
23/01/2018 7:25:28 p.m. :: Closing snapshot  
23/01/2018 7:47:17 p.m. :: VM processing started at 23/01/2018 7:47:17 p.m.  
23/01/2018 7:47:17 p.m. :: VM size: 61.0 GB (52.8 GB used)  
23/01/2018 7:47:24 p.m. :: Using guest interaction proxy BACKUPSVR (Same subnet)  
[color=#FF0000]23/01/2018 7:47:32 p.m. :: Failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing with failover) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
23/01/2018 7:50:07 p.m. :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)  
23/01/2018 7:50:21 p.m. :: Guest processing skipped (check guest OS VSS state and hypervisor integration components version)[/color]  
23/01/2018 7:50:26 p.m. :: Creating snapshot System Provider (mode: Crash consistent)  
23/01/2018 7:50:49 p.m. :: Using source proxy Hyper1.domain.local (onhost)  
23/01/2018 7:50:50 p.m. :: Getting list of guest file system local users  
23/01/2018 7:50:51 p.m. :: Saving B0BF5146-1E0C-4BA9-8E88-E606C2280CED.xml  
23/01/2018 7:50:51 p.m. :: Updating auxiliary data backup_doc  
23/01/2018 7:50:52 p.m. :: Updating auxiliary data writer_doc  
23/01/2018 7:50:54 p.m. :: Hard disk 1 (61.0 GB) 11.4 GB read at 445 MB/s [CBT] 
23/01/2018 7:51:37 p.m. :: Closing snapshot  
23/01/2018 7:51:41 p.m. :: Saving GuestMembers.xml  
23/01/2018 7:51:45 p.m. :: Finalizing  
23/01/2018 7:51:52 p.m. :: Swap file blocks skipped: 370.0 MB  
23/01/2018 7:51:52 p.m. :: Busy: Source 86% > Proxy 12% > Network 90% > Target 1%  
23/01/2018 7:51:52 p.m. :: Primary bottleneck: Network  
23/01/2018 7:51:52 p.m. :: Network traffic verification detected no corrupted blocks  
23/01/2018 7:51:52 p.m. :: Processing finished with warnings at 23/01/2018 7:51:52 p.m.

Mike Resseler
Product Manager
Posts: 5446
Liked: 573 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Mike Resseler » Jan 24, 2018 6:33 am

Alex,

So restarting the writers gives you a successful backup, but the second backup fails again. Did I understand that correctly?

What I propose is that you restart the VSS writers again, do a backup and then do a backup again (let it fail). Then you can go to the host, open command prompt as administrator and run vssadmin list writers. In that output you should be able to find out what writer is in error. I kind of fear that it is the WMI writer which would mean that you have a WMI corrupted database.

Let me know

andersgustaf
Influencer
Posts: 11
Liked: never
Joined: Oct 15, 2015 11:08 am
Full Name: Anders
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by andersgustaf » Jan 24, 2018 10:43 am

I have seen the same problem and this is (for us atleast) usualy the VM Guest that causes the error. The Guest OS often got multiple applications that registers with VSS.. but somehow it doesnt report any errors when listing all VSS inside the Guest OS.

My workaround is to just uncheck "Backup" in SCVMM and check it again... but with powershell :)

Code: Select all

# Veeam console and SCVMM Console needs to be installed to run the cmdlets
# Change SERVERNAME to Veeam Backupserver
Add-PSSnapin VeeamPSSnapin
Connect-VBRServer -Server SERVERNAME

foreach ($Job in $jobs){

    #Get latest job for every job that failed
    $FailedLastSession = $Job.FindLastSession() | Where-Object {$_.Result -eq "Failed"}


    #If failed jobs
    if($FailedLastSession){

            #Extact all vms from last failed job
            $vms = $FailedLastSession.GetTaskSessions() | Where-Object {$_.Status -eq "Failed"}
            Write-Host = "Failed VMs are:" $vms

                
                foreach ($vm in $vms){
                    # You need to disable backup and then enable backup on a failed VM in order for backup to work
                        Set-SCVirtualMachine $vm.Name -EnableBackup $false -ErrorAction Continue | Out-Null
                        Write-Host $VM.Name "- Deactivate Backup IC"
                        start-sleep -s 2
                        Set-SCVirtualMachine $vm.Name -EnableBackup $true -ErrorAction Continue | Out-Null
                        Write-Host $VM.Name "- Activate Backup IC"
                }
            
            #Start the job that will perform a backup on all failed VMs
            {
                Start-VBRJob -Job $FailedLastSession.JobName -RetryBackup -RunAsync -ErrorAction Continue
                Write-Host = "Restarting job" $FailedLastSession.JobName
            }
            Clear-Variable FailedLastSession
        }
}
Best regards,
Anders

AlexLeadingEdge
Expert
Posts: 273
Liked: 32 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by AlexLeadingEdge » Jan 24, 2018 8:11 pm

Mike Resseler wrote:Alex,

So restarting the writers gives you a successful backup, but the second backup fails again. Did I understand that correctly?
The backup is successful, but I'm getting the same error as before. Going by the log above (if I am reading it correctly) it says the VSS writers are fine, but then says it failed to use them successfully, and when it retries it says the VSS writers are no longer available?

Mike Resseler
Product Manager
Posts: 5446
Liked: 573 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Mike Resseler » Jan 24, 2018 8:17 pm

Go for the vssadmin list writers on host and guest so we can see which one fails.

AlexLeadingEdge
Expert
Posts: 273
Liked: 32 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by AlexLeadingEdge » Feb 19, 2018 12:41 am

Mike Resseler wrote:Go for the vssadmin list writers on host and guest so we can see which one fails.
No writers display on the Guest VM:

Code: Select all

Microsoft Windows [Version 6.3.9600]
(c) 2013 Microsoft Corporation. All rights reserved.

[b]C:\Windows\system32>vssadmin list writers[/b]
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

C:\Windows\system32>
Microsoft Hyper-V VSS Writer failed on Host:

Code: Select all

C:\Windows\system32>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Writer name: 'Task Scheduler Writer'
   Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
   Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
   State: [1] Stable
   Last error: No error

Writer name: 'VSS Metadata Store Writer'
   Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
   Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
   State: [1] Stable
   Last error: No error

Writer name: 'Performance Counters Writer'
   Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
   Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
   State: [1] Stable
   Last error: No error

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {e652c672-a61b-4d60-8664-57889edad8ff}
   State: [1] Stable
   Last error: No error

[b]Writer name: 'Microsoft Hyper-V VSS Writer'
   Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
   Writer Instance Id: {8db22e25-984f-4378-8a01-7a962046da31}
   State: [8] Failed
   Last error: Non-retryable error[/b]

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {0c375489-2460-496c-9d6b-583757881b1b}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {f9c51e6b-a5a8-43e3-9344-b2980d5b65ff}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {f5c93f9f-4524-407b-b392-42e91b948d71}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Config Writer'
   Writer Id: {2a40fd15-dfca-4aa8-a654-1f8c654603f6}
   Writer Instance Id: {93ad0fa8-a025-4e4a-9ee4-feaa7a6cd1d8}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {979e655f-bd30-4c8c-a7ee-e89b850c4ed2}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {5ee246dd-f950-4161-ac01-d5b044bc7aab}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {773b01f8-f000-44f1-9a0d-375d1f63b8c1}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   Writer Instance Id: {fe6ff903-5dfc-42ff-942f-4c1e6a3190b7}
   State: [1] Stable
   Last error: No error

Mike Resseler
Product Manager
Posts: 5446
Liked: 573 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Mike Resseler » Feb 19, 2018 6:45 am

So my guess was correct. The problem is with the host. Best you can do now is search in the event viewer. Search in Windows Logs, Application and System.  Look for Disk, VSS, and VolSnap entries around the time that you take a backup. And since it is about Hyper-V, also look at Applications and Services Logs -> Microsoft -> Windows -> Hyper-V-.... for specific errors around the time of backup.

Bennon Maina
Novice
Posts: 7
Liked: 4 times
Joined: Mar 07, 2018 3:23 pm
Full Name: Bennon Maina
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Bennon Maina » Mar 07, 2018 3:35 pm 2 people like this post

I remember spending about 5 months with both Microsoft and Veeam Support with backup failures to due 'prepare snapshot' failures to no avail.

The issue (as described in this article https://blog.workinghardinit.work/2015/ ... l-machine/) turned out to be orphaned snapshot files (.AVHDX) in the VM disk storage location (a two node 2K12 R2 Hyper-V Cluster). I was able to delete some but others had been locked by 'System'. I stopped the Hyper-V VMM Service and Started after a couple of minutes (note not restarting). This will start a merge process for the orphaned snapshot files (files locked by 'System').

Once merged (Check Hyper-V Manager) for the progress. (You may need to stop and start the Hyper-V VMM Service severally until the merge is complete to 100%).
If they are still a few AVHDX files still left in the storage location, you can manually delete them. After re-running the backup, it completed successfully.

It might also be worth noting backing up WK28 R2 Exchange 2010 DAG ( 1 HT, 2 DB [Active - Passive]) with AAIP enabled) all running from one job, fails on occasion but do manage to rerun the job in case it fails (and do occasionally check for these orphaned snapshot files). I Am in the process of upgrading these VMs to 2012 - 2016.

Hope this helps your situation :D .

AlexLeadingEdge
Expert
Posts: 273
Liked: 32 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by AlexLeadingEdge » Mar 13, 2018 2:06 am 1 person likes this post

Bennon, I think you may have just solved the mystery! The three VMs that we keep having issues with each have a VM_NAME-AutoRecovery.avhdx file in the VM folder dated from 2016! :shock:

AlexLeadingEdge
Expert
Posts: 273
Liked: 32 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by AlexLeadingEdge » Mar 13, 2018 3:13 am

AlexLeadingEdge wrote:Bennon, I think you may have just solved the mystery! The three VMs that we keep having issues with each have a VM_NAME-AutoRecovery.avhdx file in the VM folder dated from 2016! :shock:
I have deleted these three orphaned files and will see how the backups go tonight.

Mike Resseler
Product Manager
Posts: 5446
Liked: 573 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Mike Resseler » Mar 13, 2018 6:05 am

Cool. I am hoping that it is solved. One thing I am interested to know about. (Both Alex and Bennon), were those avhdx files not visible in the hyper-v manager? And what was the location of those? Same directory as the VHDX files?

Bennon Maina
Novice
Posts: 7
Liked: 4 times
Joined: Mar 07, 2018 3:23 pm
Full Name: Bennon Maina
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Bennon Maina » Mar 13, 2018 9:30 am 1 person likes this post

Hi Alex, it would be interesting to know if it works.
Mike, no they are not visible in Hyper-V Manager. Unless of course you manually create a snapshot/checkpoint. The snapshot/checkpoint file will have .AVHDX file extension in the VM disk storage folder/location.

Another way around this, although wouldn't recommend it on backups jobs for VMs with high I/O that are failing to create at preparing a snapshot/checkpoint; create one manually from Hyper-V Manager or SCVMM.
Once completed, delete the snapshot. All AVHDX files associated with the VMs will be merged but do check the storage location in case any are left. You should be able to delete them manually and run a backup job.

I haven't had a backup failure since.

Mike Resseler
Product Manager
Posts: 5446
Liked: 573 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Mike Resseler » Mar 13, 2018 9:34 am

Yeah, I don't think it is a good workaround to be honest but if it works now, great. Doesn't mean we should look at it for a future view. By the way, do you have a case number?

AlexLeadingEdge
Expert
Posts: 273
Liked: 32 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by AlexLeadingEdge » Mar 15, 2018 9:27 pm

Great news! Backups are working perfectly now! 8)
Mike Resseler wrote:Cool. I am hoping that it is solved. One thing I am interested to know about. (Both Alex and Bennon), were those avhdx files not visible in the hyper-v manager?
No, the Hyper-V Manager doesn't show them :shock:
Mike Resseler wrote:And what was the location of those? Same directory as the VHDX files?
Yes, in the same directory as the VHDX files. I never would have suspected looking in there! See image. I had to blank out details for security reasons, this image will delete itself in 31 days time.

Image

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests