Discussions specific to the Microsoft Hyper-V hypervisor
mooseracing
Novice
Posts: 5
Liked: never
Joined: Jul 18, 2017 5:51 pm
Contact:

The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem..

Post by mooseracing » Aug 21, 2017 2:02 pm

We are getting this error on two of the five VM's on a Hyper V 2012 host. One guest is Server 08R2 and the other Centos7.

I've made sure Application aware is unchecked, restarted the Hyper V Mgmt Service, VM's have enough free space (plenty but not sure on a specific number required).

We are trialing the software so I will admit limited knowledge here. What would be the next places to look at for errors? I'm going to try and kick off a Windows server backup and see if it pukes as well.

nmdange
Expert
Posts: 341
Liked: 82 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by nmdange » Aug 21, 2017 2:23 pm

Check the event log inside the VM. This is usually an issue with VSS within the guest OS. You can try using Windows Server Backup within the VM as a test. Also make sure you've updated the Hyper-V Integration Services to the latest version.

For Linux VMs, to do "VSS" backups you need either the Linux Integration Services or the Hyper-V backup daemon. This page has some info on that: https://docs.microsoft.com/en-us/window ... on-hyper-v

Also, if you are on Hyper-V 2012 R1, and not Hyper-V 2012 R2, you really should upgrade your host as the backups are more reliable on 2012 R2. 2016 also brings some significant improvements as well.

mooseracing
Novice
Posts: 5
Liked: never
Joined: Jul 18, 2017 5:51 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by mooseracing » Aug 21, 2017 5:45 pm

The host server is 2012R2, will be 2016 with some time as we have moved some of the other hosts to 2016.

The Windows VM is running the latest integration services. I restarted the services for the timed out VSS writers, they get stuck timing out when running a windows backup as well. Looking further into that.

mooseracing
Novice
Posts: 5
Liked: never
Joined: Jul 18, 2017 5:51 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by mooseracing » Aug 22, 2017 11:55 am

Turning off Backup IS in the VM options on Hyper V allows the backup to complete. Turning them back on fails again.

foggy
Veeam Software
Posts: 16832
Liked: 1361 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by foggy » Aug 22, 2017 4:48 pm

Please contact technical support and share your case ID here for reference.

jazzoberoi
Enthusiast
Posts: 79
Liked: 16 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by jazzoberoi » Dec 13, 2017 10:17 pm

Hi Foggy,

We need urgent help with this issue.

My Case # 02418428.

Hyper-V Cluster Hosts and the Guest FileServer VM both are Win 2012 R2 with latest updates.

Code: Select all

14/12/2017 8:09:37 AM :: Queued for processing at 14/12/2017 8:09:37 AM  
14/12/2017 8:09:38 AM :: Required backup infrastructure resources have been assigned  
14/12/2017 8:09:42 AM :: VM processing started at 14/12/2017 8:09:42 AM  
14/12/2017 8:09:42 AM :: VM size: 10.4 TB (2.7 TB used)  
14/12/2017 8:09:44 AM :: Preparing to create snapshot  
14/12/2017 8:09:46 AM :: Failed to create snapshot (HP 3PAR VSS Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
14/12/2017 8:22:00 AM :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)  
14/12/2017 8:22:02 AM :: Task has been rescheduled  
14/12/2017 8:22:14 AM :: Failed to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
14/12/2017 8:32:56 AM :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)  
14/12/2017 8:33:01 AM :: Queued for processing at 14/12/2017 8:33:01 AM  
14/12/2017 8:34:03 AM :: Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error.  If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.  
14/12/2017 8:34:03 AM :: Network traffic verification detected no corrupted blocks  
14/12/2017 8:34:03 AM :: Processing finished with errors at 14/12/2017 8:34:03 AM  
Manually running the VSS creation command from within the Guest VM seems to be working.

Image

Please help !

Mike Resseler
Veeam Software
Posts: 4825
Liked: 509 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Mike Resseler » Dec 14, 2017 8:54 am

Hi Jazz,

First: It is good that you have a support case. Keep working with our engineers as they will be able to find much more valuable information in the logs then we can see here. From what I can see in the log, it is not a problem inside the VM (as you have shown also) but there is a problem with the CSV VSS writer (which will be on the host).

Cheers
Mike

jazzoberoi
Enthusiast
Posts: 79
Liked: 16 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by jazzoberoi » Dec 18, 2017 11:47 pm

Hi Mike,

I managed to resolve the issue by creating a new Windows 2016 Server VM for the fileserver and migrating all the data to it.

This new VM runs from the same host and the same 3par LUN but yet it backs-up perfectly fine.

This indicates that the issue has always been within the VM itself and not the host or the 3par storage array.

I've attached the logs for the backup job for the last 7 days to my case for your reference as well :)

AlexLeadingEdge
Expert
Posts: 251
Liked: 29 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by AlexLeadingEdge » Dec 21, 2017 1:20 am

We are having the same issue, seeing this on about 30% of all VMs. I opened case # 02378199 and the Veeam technician said it was a Microsoft issue and couldn't help "as it is outside the scope of Veeam technical support" :(

Mike Resseler
Veeam Software
Posts: 4825
Liked: 509 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Mike Resseler » Dec 21, 2017 6:26 am

Alex,

30 percent is a lot, really a lot. Are these all the same OS? And are these on the same host? Can you verify what the writers on the host are saying? (VSSadmin list writers)

Rumple
Service Provider
Posts: 53
Liked: 1 time
Joined: Mar 10, 2010 7:50 pm
Full Name: Mark Hodges
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Rumple » Dec 29, 2017 6:33 pm

In order to get my machines to backup with this error, I had to shutdown the VM and disable the Backup (volume snapshot) integration in the hardware settings.
Even though I was not using application consistent in the backup job, it still seemed to be trying to call the vss provider or something and failing.

once I disabled that setting, then I was able to run the backup with only the warning (mode:crash consistent) being present.

PTide
Veeam Software
Posts: 4316
Liked: 355 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by PTide » Jan 01, 2018 4:54 pm

Even though I was not using application consistent in the backup job, it still seemed to be trying to call the vss provider
Correct. During a Hyper-V VM backup, Hyper-V VSS writer calls the Hyper-V Volume Shadow Copy Requestor inside the guest OS. The requestor triggers all the VSS writers inside the guest to quiesce the system for the Hyper-V snapshot.

AlexLeadingEdge
Expert
Posts: 251
Liked: 29 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by AlexLeadingEdge » Jan 09, 2018 10:13 pm

Mike Resseler wrote:Alex, 30 percent is a lot, really a lot. Are these all the same OS? And are these on the same host? Can you verify what the writers on the host are saying? (VSSadmin list writers)
Different Hyper-V Hosts (but same OS version), three different Guest VM OS's. Both the Host and the Guests have the Microsoft Hyper-V VSS Writer in a failed state. Getting the services on the Guest to restart 15 minutes before the backup starts allows the backups to work correctly (I have a batch file running in Task Scheduler), but the cause of the issue is still unknown to me. I haven't bothered to reboot the two Hyper-V Host's Microsoft Hyper-V VSS Writer service, mainly because I'm not sure what that might do to all the running VMs.

J Host 1 = Microsoft Windows Server 2012 R2 Standard Edition, 64-bit (build 9600)
C Server = Microsoft Windows Server 2008 R2 Standard Edition, 64-bit Service Pack 1 (build 7601)
P Server = Microsoft Windows Server 2012 Standard Edition, 64-bit (build 9200)
Exchange Server = Microsoft Windows Server 2012 R2 Standard Edition, 64-bit (build 9600)

S Host 2 = Microsoft Windows Server 2012 R2 Standard Edition, 64-bit (build 9600)
M2 Server = Microsoft Windows Server 2008 R2 Standard Edition, 64-bit Service Pack 1 (build 7601)


I have upgraded all of the Guest VMs on J Host 1 to the latest Hyper-V Integration Services but I can't do S Host 2's M2 Server as it has to be planned days or even weeks in advance as the install requires a reboot.

The batch file contains:
net stop VSS
net stop Winmgmt
net stop SQLWriter
net start VSS
net start Winmgmt
net start SQLWriter
The Exchange Server also has this in the batch file, and has Winmgmt removed:
net stop MSExchangeIS
net start MSExchangeIS
The VSS Writer error on the Hosts and Guest VMs is:
Writer name: 'Microsoft Hyper-V VSS Writer'
Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
Writer Instance Id: {07434c75-70ef-4057-b4b1-3f1a7ffc486b}
State: [8] Failed
Last error: Non-retryable error
The error message on Backup and Replication is:
9/01/2018 7:23:19 p.m. :: Failed to create snapshot (Microsoft Software Shadow Copy provider 1.0) (mode: Veeam application-aware processing with failover) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.

9/01/2018 7:26:04 p.m. :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)

9/01/2018 7:26:18 p.m. :: Guest processing skipped (check guest OS VSS state and hypervisor integration components version)

Mike Resseler
Veeam Software
Posts: 4825
Liked: 509 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by Mike Resseler » Jan 10, 2018 8:18 am

Alex,

Did you look at this article: http://www.insidetechnologies.eu/en/blo ... ter-error/

This might be your problem
M/

AlexLeadingEdge
Expert
Posts: 251
Liked: 29 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: The infamous 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT' problem

Post by AlexLeadingEdge » Jan 10, 2018 11:18 pm

Mike Resseler wrote:Did you look at this article: http://www.insidetechnologies.eu/en/blo ... ter-error/

This might be your problem
Hi Mike,

Interesting article, it seems to imply it is a Host issue, not a VM issue. It talks about a "Hold State" and in Figure 2 it says "Backing up...", which I have never seen on our Hosts.

I have restarted the Hyper-V Virtual Machine Management service on the J Host and will test.

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests