Host-based backup of Microsoft Hyper-V VMs.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by foggy »

Unfortunately, I cannot make sense out of this case, since it is in Italian. ;) However, it is still open, so I suggest you opening your own case for investigation. Thanks.
Link State
Veeam Legend
Posts: 33
Liked: 5 times
Joined: Jan 06, 2014 9:58 am
Full Name: Link State
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Link State »

Hi guys , whit Veeam Engineer we did several tests, using the tools snapshot.hta we have excluded all Writer on the partitionCSV (Hyperv Host win 2012 R2) expect of the provider HyperV, unfortunately did not solve the problem.

Code: Select all

-> set verbose on
-> set context persistent
-> set metadata c:\temp\diskshadow_02-06_11-51.cab
-> 
-> #Volumes:
-> #Adding volume c:\clusterstorage\csv-servicecl\ provider microsoft csv shadow copy provider
-> add volume \\?\volume{f9544437-ca59-4fb8-887d-973b6b6dee98}\ provider {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff} alias vol1
-> 
-> #Exclude writers:
-> #Exclude asr writer:
-> writer exclude {be000cbe-11fe-4426-9c58-531aa6355fc4}
-> #Exclude bits writer:
-> writer exclude {4969d978-be47-48b0-b100-f328f07ac1e0}
-> #Exclude cluster database:
-> writer exclude {41e12264-35d8-479b-8e5c-9b23d1dad37e}
-> #Exclude cluster shared volume vss writer:
-> writer exclude {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
-> #Exclude com+ regdb writer:
-> writer exclude {542da469-d3e1-473c-9f4f-7847f01fc64f}
-> #Exclude performance counters writer:
-> writer exclude {0bada1de-01a9-4625-8278-69e735f39dd2}
-> #Exclude registry writer:
-> writer exclude {afbab4a2-367d-4d15-a586-71dbb18f8485}
-> #Exclude shadow copy optimization writer:
-> writer exclude {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
-> #Exclude system writer:
-> writer exclude {e8132975-6f93-4464-a53e-1050253ae220}
-> #Exclude task scheduler writer:
-> writer exclude {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
-> #Exclude vss metadata store writer:
-> writer exclude {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
-> #Exclude wmi writer:
-> writer exclude {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
-> 
-> #Process components:
-> #Exclude microsoft hyper-v vss writer:host component:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\host component"
-> #Exclude microsoft hyper-v vss writer:online\Server01:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\fbdc5740-70d4-499f-8a41-55fc89fe2255"
-> #Exclude microsoft hyper-v vss writer:online\pkappmsmcthv01:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\26f0762f-82be-4c09-bbba-4e4b6a8b4e87"
-> #Exclude microsoft hyper-v vss writer:online\server02:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\44236a18-bb52-4f39-aea5-8cfc2d69899d"
-> #Include microsoft hyper-v vss writer:online\server02:
-> writer verify "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\005ca1c5-b0e5-409b-b6e3-3cd86b387c54"
-> #Exclude microsoft hyper-v vss writer:online\server02:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\c0fe6904-0bd9-4401-a657-72bb97bf7269"
-> #Exclude microsoft hyper-v vss writer:online\server02:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\be81c44f-202c-4165-8543-e06b56848941"
-> 
-> create
Component "\26F0762F-82BE-4C09-BBBA-4E4B6A8B4E87" from writer "Microsoft Hyper-V VSS Writer" is explicitly excluded.
Component "\44236A18-BB52-4F39-AEA5-8CFC2D69899D" from writer "Microsoft Hyper-V VSS Writer" is explicitly excluded.
Component "\BE81C44F-202C-4165-8543-E06B56848941" from writer "Microsoft Hyper-V VSS Writer" is explicitly excluded.
Component "\FBDC5740-70D4-499F-8A41-55FC89FE2255" from writer "Microsoft Hyper-V VSS Writer" is explicitly excluded.
Component "\Host Component" from writer "Microsoft Hyper-V VSS Writer" is explicitly excluded.
Component "\005CA1C5-B0E5-409B-B6E3-3CD86B387C54" from writer "Microsoft Hyper-V VSS Writer" is selected.

* Including writer "Microsoft Hyper-V VSS Writer":
	+ Adding component: \005CA1C5-B0E5-409B-B6E3-3CD86B387C54
	+ Adding component: \CDACA7ED-10BD-4433-9F6B-92BE7A6C53D3
Veeam: Veeam Legends 2021&2022 | VMCE 2020 | VMCA 2022 | VMXP - Microsoft: MCITP | MCP| MCSA | 2008 R2 | 2012R2 | 2016 | MCSE Infrastructure | MCSE Cloud Platform - Azure: AZ900 | AZ104 - Cisco: CCNA-
kdovale
Lurker
Posts: 2
Liked: never
Joined: Apr 14, 2014 7:37 pm
Full Name: Keith
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by kdovale »

We are having a similar issue but with windows 2008 r2 vm on a hyper-v 2012 r2 host, if we reboot I can back it up once, but if I back it up using a job it fails with the same error you are seeing, I have checked all the recommended fixes, ie integration services, etc
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by foggy »

Keith, have you contacted technical support already?
kdovale
Lurker
Posts: 2
Liked: never
Joined: Apr 14, 2014 7:37 pm
Full Name: Keith
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by kdovale »

I did a bit of a search on google came across a post where someone was having the same issue, the resolution was to delete the job and re-create it and the problem was resolved, haven't had the issue again since. Seems that the job actually becomes the issue, weird, but its a fix.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Vitaliy S. »

Yes, that's a bit unexpected to re-create the job for this issue, VSS writer operation cannot be affected by job creation/removal. If you see it once again, I would appreciate if you could let our technical team take a look a the debug logs. Thanks!
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Shralok »

I am having the same issue as described in this thread.

Windows 2008 R2 VM which is a file server (approximately 900 GB)
I am in the middle of migrating from Hyper-V 2008 R2 to Hyper-V 2012 R2 and therefore I have moved the VM in question from a 2008 R2 host to a 2012 R2 host.
I have done this with other VMs and Veeam experienced no errors - I only had to reconfigure the job to tell Veeam that the VM was now on a different cluster.
However for this particular VM the job fails 80% of the time. This is why I find it so odd, because it will work sometimes but the majority of the time it fails with this same VSS snapshot error.

Same result after upgrading to version 7.0.0.871

Veeam Support - Case # 00612524
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by foggy »

Thanks for providing the case ID. Do you consistently see any related messages in the Windows Event Log on this VM? Have you tried all the suggestions given above?
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Shralok »

foggy wrote:Thanks for providing the case ID. Do you consistently see any related messages in the Windows Event Log on this VM? Have you tried all the suggestions given above?
I have found that it might be something in Windows as this VM has several times seen a pop-up of the following:

(Keep in mind there is no "N:" on this VM...)

Image

Also I am seeing these in the Application Log repeatedly.

EDIT: [Logs have been removed by moderator]
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by foggy »

Let's wait for what our support engineers can suggest after reviewing the log files.
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Shralok »

I have a second VM where this same error is occurring. This second VM has also been migrated from HV 2008 R2 to HV 2012 R2.

My process for moving the VM is to shut down the VM and move the LUN on the SAN from one storage group (2008 R2) to another (2012 R2). Then I add that LUN to my Cluster Shared Volumes in 2012 R2 cluster so I can see the VHDs. Then I create a new VM configuration in 2012 R2 and attach the already-existing VHDs.

Doing this for most VMs (approximately 8 others) has not been a problem, so I am not sure why these two have issues. I am leaning towards file shares possibly being an issue here but I am not able to prove it.

I have uploaded a significant amount of logs to my case with also a screenshot of the "ghost Shares" in Server Manager that are appearing when Veeam tries to back up.

Turning off Quiescence and Application-Aware Image Processing has enabled the first VM to back up successfully. For this second VM I am not sure if I want to disable the features that have made my first VM back up correctly. This is because the second VM is a SQL VM and I do not think it is wise to turn off Quiescence and AAIP. Please let me know if this is not the case.

Thank you
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by foggy »

Typically, SQL Server is able to successfully recover from crash-consistent backup, so you can try disabling AAIP and VMware quiescence. If this works, then you'll at least have crash-consistent backup, better than no backup at all.

That said, please continue investigating the issue with our support.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Vitaliy S. »

Shralok wrote:My process for moving the VM is to shut down the VM and move the LUN on the SAN from one storage group (2008 R2) to another (2012 R2). Then I add that LUN to my Cluster Shared Volumes in 2012 R2 cluster so I can see the VHDs. Then I create a new VM configuration in 2012 R2 and attach the already-existing VHDs.
Just on a side note - why not to use Veeam replication jobs to migrate the VMs from 2008 R2 to 2012 R2 hosts? Since you have a second machine with this issue I would try to use replication jobs instead of your existing migration process, this would allow to isolate the reason for this strange behavior.
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Shralok »

Vitaliy S. wrote: Just on a side note - why not to use Veeam replication jobs to migrate the VMs from 2008 R2 to 2012 R2 hosts? Since you have a second machine with this issue I would try to use replication jobs instead of your existing migration process, this would allow to isolate the reason for this strange behavior.
So in this case I would shut down the source VM, run a replication job (in order to have the VMs 100% synchronized) and then do a permanent failover to the destination?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by foggy »

Correct.
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Shralok »

foggy wrote:Correct.
On the Replication job I am getting a Warning of "Failed to update VM registration in cluster Details: The server threw an exception. (Exception from HRESULT: 0x80010105 (RPC_E_SERVERFAULT))" so I will have to open a separate case.

:cry:
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Shralok »

Well I am unable to edit my previous post but I found that this article was very helpful in understanding the migration from 2008 R2 Hyper-V to 2012 R2 Hyper-V:
http://www.veeam.com/blog/3-east-steps- ... 12-r2.html
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by foggy »

So were you able to successfully migrate?
a32guy
Lurker
Posts: 2
Liked: 1 time
Joined: Jan 23, 2015 3:40 pm
Full Name: CS
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by a32guy » 1 person likes this post

I know its an older thread but I had the same issue with one particular Server 2008 R2 VM. I was able to resolve and wanted to share my fix. I did a combination of:

1. Disabling backup from integration components, rebooting, then reenabling.
2. Enabling shadow copies on system and data volumes with no size limit.
3. Finally reregistering VSS writers using this script and rebooting (test on lab guest first!):

Code: Select all

REM : Register VSS componant for Windows Serve r2008
REM :   You will need to run the batch file as Administrator.
REM : This is only for Windows 2008 not Windows 2003.
REM : After running the bat file, you will need to reboot the server to bring all of the writers into a stable state.


net stop "System Event Notification Service"
net stop "Background Intelligent Transfer Service"
net stop "COM+ Event System"
net stop "Microsoft Software Shadow Copy Provider"
net stop "Volume Shadow Copy"
cd /d %windir%\system32
net stop vss
net stop swprv
regsvr32 /s ATL.DLL
regsvr32 /s comsvcs.DLL
regsvr32 /s credui.DLL
regsvr32 /s CRYPTNET.DLL
regsvr32 /s CRYPTUI.DLL
regsvr32 /s dhcpqec.DLL
regsvr32 /s dssenh.DLL
regsvr32 /s eapqec.DLL
regsvr32 /s esscli.DLL
regsvr32 /s FastProx.DLL
regsvr32 /s FirewallAPI.DLL
regsvr32 /s kmsvc.DLL
regsvr32 /s lsmproxy.DLL
regsvr32 /s MSCTF.DLL
regsvr32 /s msi.DLL
regsvr32 /s msxml3.DLL
regsvr32 /s ncprov.DLL
regsvr32 /s ole32.DLL
regsvr32 /s OLEACC.DLL
regsvr32 /s OLEAUT32.DLL
regsvr32 /s PROPSYS.DLL
regsvr32 /s QAgent.DLL
regsvr32 /s qagentrt.DLL
regsvr32 /s QUtil.DLL
regsvr32 /s raschap.DLL
regsvr32 /s RASQEC.DLL
regsvr32 /s rastls.DLL
regsvr32 /s repdrvfs.DLL
regsvr32 /s RPCRT4.DLL
regsvr32 /s rsaenh.DLL
regsvr32 /s SHELL32.DLL
regsvr32 /s shsvcs.DLL
regsvr32 /s /i swprv.DLL
regsvr32 /s tschannel.DLL
regsvr32 /s USERENV.DLL
regsvr32 /s vss_ps.DLL
regsvr32 /s wbemcons.DLL
regsvr32 /s wbemcore.DLL
regsvr32 /s wbemess.DLL
regsvr32 /s wbemsvc.DLL
regsvr32 /s WINHTTP.DLL
regsvr32 /s WINTRUST.DLL
regsvr32 /s wmiprvsd.DLL
regsvr32 /s wmisvc.DLL
regsvr32 /s wmiutils.DLL
regsvr32 /s wuaueng.DLL
sfc /SCANFILE=%windir%\system32\catsrv.DLL
sfc /SCANFILE=%windir%\system32\catsrvut.DLL
sfc /SCANFILE=%windir%\system32\CLBCatQ.DLL
net start "COM+ Event System"
bcscholl
Enthusiast
Posts: 31
Liked: 3 times
Joined: Dec 04, 2013 3:56 pm
Full Name: Brian Scholl
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by bcscholl » 2 people like this post

I had a similar issue fixed by creating a script to restart the Hyper-V Volume Shadow Copy Requestor [vmicvss]. We run the script once a day for all VMs in the backup job.
LinusH
Service Provider
Posts: 8
Liked: 2 times
Joined: Feb 09, 2015 8:32 am
Full Name: Linus Hellsing
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by LinusH »

Hi,
Just for information, we also have the error VSS_WS_FAILED_AT_PREPARE_SNAPSHOT and in our case it's on our Exchange running on vmware. I will create a support ticket soon but wanted you to know that you're not alone...

Br Linus Hellsing
marcinbojko
Enthusiast
Posts: 40
Liked: 12 times
Joined: Mar 06, 2014 10:40 am
Full Name: Marcin Bojko
Location: Lublin/Poland
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by marcinbojko »

I am interested - is there any general solution from Veeam engineer? From my point of view - restarting VM helps, but in production environment it is not possible to restart working SQL server every 24 or 12h.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by foggy »

Marcin, you can open your own case for direct faster communication with our engineers.
marcinbojko
Enthusiast
Posts: 40
Liked: 12 times
Joined: Mar 06, 2014 10:40 am
Full Name: Marcin Bojko
Location: Lublin/Poland
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by marcinbojko »

Foggy,
I was hoping for general solution, but in case of recurring issue, I will be forced to.
Tibo
Influencer
Posts: 10
Liked: never
Joined: May 20, 2011 8:15 am
Full Name: Tibo
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Tibo »

Same problem here on Hyper-V 2012 R2 with a 2008 R2 VM and a 2012 R2 VM ...

I still have no official fix from Veeam support (Case opened #00914558 for more than two weeks and tons of logs / tests have been made :-/ )
JoshRountree
Enthusiast
Posts: 34
Liked: 2 times
Joined: Feb 23, 2012 5:49 pm
Full Name: Josh Rountree
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by JoshRountree »

Did anyone ever find a resolution to this that didn't involve disabling crash consistent backups?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by foggy »

Josh, please contact technical support, since reasons of the issue in your particular case can be different.
1971merlin
Influencer
Posts: 19
Liked: never
Joined: Jun 01, 2014 6:03 am
Full Name: Craig Hart
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by 1971merlin »

For us, issuing the command DISKPART AUTOMOUNT SCRUB from the commandline of the VM fixed the issue.

My Theory is that VEEAM somehow reads the automount data (Registry?) and tries to mount all the fixed disk volumes in order to back them up. Disks such as iSCSI are considered local (i.e. not removable) therefore fail the mount (because they no longer exist), in turn this fails the VSS snapshot and also generates the erronous "format" dialog on the console.

I discovered this because every server I've had this issue on, has previously backed up using Windows server Backup, to an iSCSI volume - that was the common denominator that pointed to the issue for us.
SamiKarhula
Lurker
Posts: 1
Liked: never
Joined: Nov 04, 2013 7:53 am
Full Name: Sami Karhula
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by SamiKarhula »

we had same problem one customer who wants testing Veeam and we cannot get that works with Veeam support. I made demo lab to my home with server 2016 Cluster shared volume and i
get same error in home what i get on customers server. I think first there is on 2008r2 cluster something old crap from backup exec (sorry for dirty word:D)
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Post by Mike Resseler »

Hi Sami,

I am not going to use the word ;-) You state that you created a demo lab at home with server 2016 and that you received the same error. Is that with Veeam B&R 9.5 (released yesterday)? Because that is the only version that is supported for Windows Server 2016. Also, the VM you are trying to backup, is that the same one as the one with the customer? Or a brand new installed VM

Thanks
Mike
Post Reply

Who is online

Users browsing this forum: No registered users and 21 guests