The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem..

Discussions specific to the Microsoft Hyper-V hypervisor

The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem..

Veeam Logoby mooseracing » Mon Aug 21, 2017 2:02 pm

We are getting this error on two of the five VM's on a Hyper V 2012 host. One guest is Server 08R2 and the other Centos7.

I've made sure Application aware is unchecked, restarted the Hyper V Mgmt Service, VM's have enough free space (plenty but not sure on a specific number required).

We are trialing the software so I will admit limited knowledge here. What would be the next places to look at for errors? I'm going to try and kick off a Windows server backup and see if it pukes as well.
mooseracing
Novice
 
Posts: 5
Liked: never
Joined: Tue Jul 18, 2017 5:51 pm

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby nmdange » Mon Aug 21, 2017 2:23 pm

Check the event log inside the VM. This is usually an issue with VSS within the guest OS. You can try using Windows Server Backup within the VM as a test. Also make sure you've updated the Hyper-V Integration Services to the latest version.

For Linux VMs, to do "VSS" backups you need either the Linux Integration Services or the Hyper-V backup daemon. This page has some info on that: https://docs.microsoft.com/en-us/window ... on-hyper-v

Also, if you are on Hyper-V 2012 R1, and not Hyper-V 2012 R2, you really should upgrade your host as the backups are more reliable on 2012 R2. 2016 also brings some significant improvements as well.
nmdange
Expert
 
Posts: 320
Liked: 79 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby mooseracing » Mon Aug 21, 2017 5:45 pm

The host server is 2012R2, will be 2016 with some time as we have moved some of the other hosts to 2016.

The Windows VM is running the latest integration services. I restarted the services for the timed out VSS writers, they get stuck timing out when running a windows backup as well. Looking further into that.
mooseracing
Novice
 
Posts: 5
Liked: never
Joined: Tue Jul 18, 2017 5:51 pm

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby mooseracing » Tue Aug 22, 2017 11:55 am

Turning off Backup IS in the VM options on Hyper V allows the backup to complete. Turning them back on fails again.
mooseracing
Novice
 
Posts: 5
Liked: never
Joined: Tue Jul 18, 2017 5:51 pm

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby foggy » Tue Aug 22, 2017 4:48 pm

Please contact technical support and share your case ID here for reference.
foggy
Veeam Software
 
Posts: 16494
Liked: 1320 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby jazzoberoi » Wed Dec 13, 2017 10:17 pm

Hi Foggy,

We need urgent help with this issue.

My Case # 02418428.

Hyper-V Cluster Hosts and the Guest FileServer VM both are Win 2012 R2 with latest updates.

Code: Select all
14/12/2017 8:09:37 AM :: Queued for processing at 14/12/2017 8:09:37 AM 
14/12/2017 8:09:38 AM :: Required backup infrastructure resources have been assigned 
14/12/2017 8:09:42 AM :: VM processing started at 14/12/2017 8:09:42 AM 
14/12/2017 8:09:42 AM :: VM size: 10.4 TB (2.7 TB used) 
14/12/2017 8:09:44 AM :: Preparing to create snapshot 
14/12/2017 8:09:46 AM :: Failed to create snapshot (HP 3PAR VSS Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks. 
14/12/2017 8:22:00 AM :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.) 
14/12/2017 8:22:02 AM :: Task has been rescheduled 
14/12/2017 8:22:14 AM :: Failed to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks. 
14/12/2017 8:32:56 AM :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.) 
14/12/2017 8:33:01 AM :: Queued for processing at 14/12/2017 8:33:01 AM 
14/12/2017 8:34:03 AM :: Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks. 
14/12/2017 8:34:03 AM :: Network traffic verification detected no corrupted blocks 
14/12/2017 8:34:03 AM :: Processing finished with errors at 14/12/2017 8:34:03 AM 


Manually running the VSS creation command from within the Guest VM seems to be working.

Image

Please help !
jazzoberoi
Enthusiast
 
Posts: 79
Liked: 16 times
Joined: Wed Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby Mike Resseler » Thu Dec 14, 2017 8:54 am

Hi Jazz,

First: It is good that you have a support case. Keep working with our engineers as they will be able to find much more valuable information in the logs then we can see here. From what I can see in the log, it is not a problem inside the VM (as you have shown also) but there is a problem with the CSV VSS writer (which will be on the host).

Cheers
Mike
Mike Resseler
Veeam Software
 
Posts: 4523
Liked: 486 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby jazzoberoi » Mon Dec 18, 2017 11:47 pm

Hi Mike,

I managed to resolve the issue by creating a new Windows 2016 Server VM for the fileserver and migrating all the data to it.

This new VM runs from the same host and the same 3par LUN but yet it backs-up perfectly fine.

This indicates that the issue has always been within the VM itself and not the host or the 3par storage array.

I've attached the logs for the backup job for the last 7 days to my case for your reference as well :)
jazzoberoi
Enthusiast
 
Posts: 79
Liked: 16 times
Joined: Wed Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby AlexLeadingEdge » Thu Dec 21, 2017 1:20 am

We are having the same issue, seeing this on about 30% of all VMs. I opened case # 02378199 and the Veeam technician said it was a Microsoft issue and couldn't help "as it is outside the scope of Veeam technical support" :(
AlexLeadingEdge
Expert
 
Posts: 226
Liked: 28 times
Joined: Mon Dec 14, 2015 9:42 pm

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby Mike Resseler » Thu Dec 21, 2017 6:26 am

Alex,

30 percent is a lot, really a lot. Are these all the same OS? And are these on the same host? Can you verify what the writers on the host are saying? (VSSadmin list writers)
Mike Resseler
Veeam Software
 
Posts: 4523
Liked: 486 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby Rumple » Fri Dec 29, 2017 6:33 pm

In order to get my machines to backup with this error, I had to shutdown the VM and disable the Backup (volume snapshot) integration in the hardware settings.
Even though I was not using application consistent in the backup job, it still seemed to be trying to call the vss provider or something and failing.

once I disabled that setting, then I was able to run the backup with only the warning (mode:crash consistent) being present.
Rumple
Enthusiast
 
Posts: 51
Liked: never
Joined: Wed Mar 10, 2010 7:50 pm
Full Name: Mark Hodges

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby PTide » Mon Jan 01, 2018 4:54 pm

Even though I was not using application consistent in the backup job, it still seemed to be trying to call the vss provider
Correct. During a Hyper-V VM backup, Hyper-V VSS writer calls the Hyper-V Volume Shadow Copy Requestor inside the guest OS. The requestor triggers all the VSS writers inside the guest to quiesce the system for the Hyper-V snapshot.
PTide
Veeam Software
 
Posts: 4138
Liked: 341 times
Joined: Tue May 19, 2015 1:46 pm

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby AlexLeadingEdge » Tue Jan 09, 2018 10:13 pm

Mike Resseler wrote:Alex, 30 percent is a lot, really a lot. Are these all the same OS? And are these on the same host? Can you verify what the writers on the host are saying? (VSSadmin list writers)


Different Hyper-V Hosts (but same OS version), three different Guest VM OS's. Both the Host and the Guests have the Microsoft Hyper-V VSS Writer in a failed state. Getting the services on the Guest to restart 15 minutes before the backup starts allows the backups to work correctly (I have a batch file running in Task Scheduler), but the cause of the issue is still unknown to me. I haven't bothered to reboot the two Hyper-V Host's Microsoft Hyper-V VSS Writer service, mainly because I'm not sure what that might do to all the running VMs.

J Host 1 = Microsoft Windows Server 2012 R2 Standard Edition, 64-bit (build 9600)
C Server = Microsoft Windows Server 2008 R2 Standard Edition, 64-bit Service Pack 1 (build 7601)
P Server = Microsoft Windows Server 2012 Standard Edition, 64-bit (build 9200)
Exchange Server = Microsoft Windows Server 2012 R2 Standard Edition, 64-bit (build 9600)

S Host 2 = Microsoft Windows Server 2012 R2 Standard Edition, 64-bit (build 9600)
M2 Server = Microsoft Windows Server 2008 R2 Standard Edition, 64-bit Service Pack 1 (build 7601)


I have upgraded all of the Guest VMs on J Host 1 to the latest Hyper-V Integration Services but I can't do S Host 2's M2 Server as it has to be planned days or even weeks in advance as the install requires a reboot.

The batch file contains:

net stop VSS
net stop Winmgmt
net stop SQLWriter
net start VSS
net start Winmgmt
net start SQLWriter


The Exchange Server also has this in the batch file, and has Winmgmt removed:

net stop MSExchangeIS
net start MSExchangeIS


The VSS Writer error on the Hosts and Guest VMs is:

Writer name: 'Microsoft Hyper-V VSS Writer'
Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
Writer Instance Id: {07434c75-70ef-4057-b4b1-3f1a7ffc486b}
State: [8] Failed
Last error: Non-retryable error


The error message on Backup and Replication is:

9/01/2018 7:23:19 p.m. :: Failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing with failover) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.

9/01/2018 7:26:04 p.m. :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)

9/01/2018 7:26:18 p.m. :: Guest processing skipped (check guest OS VSS state and hypervisor integration components version)
AlexLeadingEdge
Expert
 
Posts: 226
Liked: 28 times
Joined: Mon Dec 14, 2015 9:42 pm

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby Mike Resseler » Wed Jan 10, 2018 8:18 am

Alex,

Did you look at this article: http://www.insidetechnologies.eu/en/blo ... ter-error/

This might be your problem
M/
Mike Resseler
Veeam Software
 
Posts: 4523
Liked: 486 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Veeam Logoby AlexLeadingEdge » Wed Jan 10, 2018 11:18 pm

Mike Resseler wrote:Did you look at this article: http://www.insidetechnologies.eu/en/blo ... ter-error/

This might be your problem
Hi Mike,

Interesting article, it seems to imply it is a Host issue, not a VM issue. It talks about a "Hold State" and in Figure 2 it says "Backing up...", which I have never seen on our Hosts.

I have restarted the Hyper-V Virtual Machine Management service on the J Host and will test.
AlexLeadingEdge
Expert
 
Posts: 226
Liked: 28 times
Joined: Mon Dec 14, 2015 9:42 pm

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 17 guests