Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAPSHOT

[foggy]

Unfortunately, I cannot make sense out of this case, since it is in Italian. However, it is still open, so I suggest you opening your own case for investigation. Thanks.

[Link State]

Hi guys , whit Veeam Engineer we did several tests, using the tools snapshot.hta we have excluded all Writer on the partitionCSV (Hyperv Host win 2012 R2) expect of the provider HyperV, unfortunately did not solve the problem.

Code: Select all

-> set verbose on
-> set context persistent
-> set metadata c:\temp\diskshadow_02-06_11-51.cab
-> 
-> #Volumes:
-> #Adding volume c:\clusterstorage\csv-servicecl\ provider microsoft csv shadow copy provider
-> add volume \\?\volume{f9544437-ca59-4fb8-887d-973b6b6dee98}\ provider {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff} alias vol1
-> 
-> #Exclude writers:
-> #Exclude asr writer:
-> writer exclude {be000cbe-11fe-4426-9c58-531aa6355fc4}
-> #Exclude bits writer:
-> writer exclude {4969d978-be47-48b0-b100-f328f07ac1e0}
-> #Exclude cluster database:
-> writer exclude {41e12264-35d8-479b-8e5c-9b23d1dad37e}
-> #Exclude cluster shared volume vss writer:
-> writer exclude {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
-> #Exclude com+ regdb writer:
-> writer exclude {542da469-d3e1-473c-9f4f-7847f01fc64f}
-> #Exclude performance counters writer:
-> writer exclude {0bada1de-01a9-4625-8278-69e735f39dd2}
-> #Exclude registry writer:
-> writer exclude {afbab4a2-367d-4d15-a586-71dbb18f8485}
-> #Exclude shadow copy optimization writer:
-> writer exclude {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
-> #Exclude system writer:
-> writer exclude {e8132975-6f93-4464-a53e-1050253ae220}
-> #Exclude task scheduler writer:
-> writer exclude {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
-> #Exclude vss metadata store writer:
-> writer exclude {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
-> #Exclude wmi writer:
-> writer exclude {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
-> 
-> #Process components:
-> #Exclude microsoft hyper-v vss writer:host component:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\host component"
-> #Exclude microsoft hyper-v vss writer:online\Server01:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\fbdc5740-70d4-499f-8a41-55fc89fe2255"
-> #Exclude microsoft hyper-v vss writer:online\pkappmsmcthv01:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\26f0762f-82be-4c09-bbba-4e4b6a8b4e87"
-> #Exclude microsoft hyper-v vss writer:online\server02:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\44236a18-bb52-4f39-aea5-8cfc2d69899d"
-> #Include microsoft hyper-v vss writer:online\server02:
-> writer verify "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\005ca1c5-b0e5-409b-b6e3-3cd86b387c54"
-> #Exclude microsoft hyper-v vss writer:online\server02:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\c0fe6904-0bd9-4401-a657-72bb97bf7269"
-> #Exclude microsoft hyper-v vss writer:online\server02:
-> writer exclude "{66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}:\be81c44f-202c-4165-8543-e06b56848941"
-> 
-> create
Component "\26F0762F-82BE-4C09-BBBA-4E4B6A8B4E87" from writer "Microsoft Hyper-V VSS Writer" is explicitly excluded.
Component "\44236A18-BB52-4F39-AEA5-8CFC2D69899D" from writer "Microsoft Hyper-V VSS Writer" is explicitly excluded.
Component "\BE81C44F-202C-4165-8543-E06B56848941" from writer "Microsoft Hyper-V VSS Writer" is explicitly excluded.
Component "\FBDC5740-70D4-499F-8A41-55FC89FE2255" from writer "Microsoft Hyper-V VSS Writer" is explicitly excluded.
Component "\Host Component" from writer "Microsoft Hyper-V VSS Writer" is explicitly excluded.
Component "\005CA1C5-B0E5-409B-B6E3-3CD86B387C54" from writer "Microsoft Hyper-V VSS Writer" is selected.

* Including writer "Microsoft Hyper-V VSS Writer":
	+ Adding component: \005CA1C5-B0E5-409B-B6E3-3CD86B387C54
	+ Adding component: \CDACA7ED-10BD-4433-9F6B-92BE7A6C53D3

[kdovale]

We are having a similar issue but with windows 2008 r2 vm on a hyper-v 2012 r2 host, if we reboot I can back it up once, but if I back it up using a job it fails with the same error you are seeing, I have checked all the recommended fixes, ie integration services, etc

[foggy]

Keith, have you contacted technical support already?

[kdovale]

I did a bit of a search on google came across a post where someone was having the same issue, the resolution was to delete the job and re-create it and the problem was resolved, haven't had the issue again since. Seems that the job actually becomes the issue, weird, but its a fix.

[Vitaliy S.]

Yes, that's a bit unexpected to re-create the job for this issue, VSS writer operation cannot be affected by job creation/removal. If you see it once again, I would appreciate if you could let our technical team take a look a the debug logs. Thanks!

[Shralok]

I am having the same issue as described in this thread.

Windows 2008 R2 VM which is a file server (approximately 900 GB)
I am in the middle of migrating from Hyper-V 2008 R2 to Hyper-V 2012 R2 and therefore I have moved the VM in question from a 2008 R2 host to a 2012 R2 host.
I have done this with other VMs and Veeam experienced no errors - I only had to reconfigure the job to tell Veeam that the VM was now on a different cluster.
However for this particular VM the job fails 80% of the time. This is why I find it so odd, because it will work sometimes but the majority of the time it fails with this same VSS snapshot error.

Same result after upgrading to version 7.0.0.871

Veeam Support - Case # 00612524

[foggy]

Thanks for providing the case ID. Do you consistently see any related messages in the Windows Event Log on this VM? Have you tried all the suggestions given above?

[Shralok]

Thanks for providing the case ID. Do you consistently see any related messages in the Windows Event Log on this VM? Have you tried all the suggestions given above?

I have found that it might be something in Windows as this VM has several times seen a pop-up of the following:

(Keep in mind there is no "N:" on this VM...)



Also I am seeing these in the Application Log repeatedly.

EDIT: [Logs have been removed by moderator]

[foggy]

Let's wait for what our support engineers can suggest after reviewing the log files.

[Shralok]

I have a second VM where this same error is occurring. This second VM has also been migrated from HV 2008 R2 to HV 2012 R2.

My process for moving the VM is to shut down the VM and move the LUN on the SAN from one storage group (2008 R2) to another (2012 R2). Then I add that LUN to my Cluster Shared Volumes in 2012 R2 cluster so I can see the VHDs. Then I create a new VM configuration in 2012 R2 and attach the already-existing VHDs.

Doing this for most VMs (approximately 8 others) has not been a problem, so I am not sure why these two have issues. I am leaning towards file shares possibly being an issue here but I am not able to prove it.

I have uploaded a significant amount of logs to my case with also a screenshot of the "ghost Shares" in Server Manager that are appearing when Veeam tries to back up.

Turning off Quiescence and Application-Aware Image Processing has enabled the first VM to back up successfully. For this second VM I am not sure if I want to disable the features that have made my first VM back up correctly. This is because the second VM is a SQL VM and I do not think it is wise to turn off Quiescence and AAIP. Please let me know if this is not the case.

Thank you

[foggy]

Typically, SQL Server is able to successfully recover from crash-consistent backup, so you can try disabling AAIP and VMware quiescence. If this works, then you'll at least have crash-consistent backup, better than no backup at all.

That said, please continue investigating the issue with our support.

[Vitaliy S.]

My process for moving the VM is to shut down the VM and move the LUN on the SAN from one storage group (2008 R2) to another (2012 R2). Then I add that LUN to my Cluster Shared Volumes in 2012 R2 cluster so I can see the VHDs. Then I create a new VM configuration in 2012 R2 and attach the already-existing VHDs.

Just on a side note - why not to use Veeam replication jobs to migrate the VMs from 2008 R2 to 2012 R2 hosts? Since you have a second machine with this issue I would try to use replication jobs instead of your existing migration process, this would allow to isolate the reason for this strange behavior.

[Shralok]

Just on a side note - why not to use Veeam replication jobs to migrate the VMs from 2008 R2 to 2012 R2 hosts? Since you have a second machine with this issue I would try to use replication jobs instead of your existing migration process, this would allow to isolate the reason for this strange behavior.

So in this case I would shut down the source VM, run a replication job (in order to have the VMs 100% synchronized) and then do a permanent failover to the destination?

[foggy]

Correct.

[Shralok]

Correct.

On the Replication job I am getting a Warning of "Failed to update VM registration in cluster Details: The server threw an exception. (Exception from HRESULT: 0x80010105 (RPC_E_SERVERFAULT))" so I will have to open a separate case.

[Shralok]

Well I am unable to edit my previous post but I found that this article was very helpful in understanding the migration from 2008 R2 Hyper-V to 2012 R2 Hyper-V:
http://www.veeam.com/blog/3-east-steps- ... 12-r2.html

[foggy]

So were you able to successfully migrate?

[a32guy]

I know its an older thread but I had the same issue with one particular Server 2008 R2 VM. I was able to resolve and wanted to share my fix. I did a combination of:

1. Disabling backup from integration components, rebooting, then reenabling.
2. Enabling shadow copies on system and data volumes with no size limit.
3. Finally reregistering VSS writers using this script and rebooting (test on lab guest first!):

Code: Select all

REM : Register VSS componant for Windows Serve r2008
REM :   You will need to run the batch file as Administrator.
REM : This is only for Windows 2008 not Windows 2003.
REM : After running the bat file, you will need to reboot the server to bring all of the writers into a stable state.


net stop "System Event Notification Service"
net stop "Background Intelligent Transfer Service"
net stop "COM+ Event System"
net stop "Microsoft Software Shadow Copy Provider"
net stop "Volume Shadow Copy"
cd /d %windir%\system32
net stop vss
net stop swprv
regsvr32 /s ATL.DLL
regsvr32 /s comsvcs.DLL
regsvr32 /s credui.DLL
regsvr32 /s CRYPTNET.DLL
regsvr32 /s CRYPTUI.DLL
regsvr32 /s dhcpqec.DLL
regsvr32 /s dssenh.DLL
regsvr32 /s eapqec.DLL
regsvr32 /s esscli.DLL
regsvr32 /s FastProx.DLL
regsvr32 /s FirewallAPI.DLL
regsvr32 /s kmsvc.DLL
regsvr32 /s lsmproxy.DLL
regsvr32 /s MSCTF.DLL
regsvr32 /s msi.DLL
regsvr32 /s msxml3.DLL
regsvr32 /s ncprov.DLL
regsvr32 /s ole32.DLL
regsvr32 /s OLEACC.DLL
regsvr32 /s OLEAUT32.DLL
regsvr32 /s PROPSYS.DLL
regsvr32 /s QAgent.DLL
regsvr32 /s qagentrt.DLL
regsvr32 /s QUtil.DLL
regsvr32 /s raschap.DLL
regsvr32 /s RASQEC.DLL
regsvr32 /s rastls.DLL
regsvr32 /s repdrvfs.DLL
regsvr32 /s RPCRT4.DLL
regsvr32 /s rsaenh.DLL
regsvr32 /s SHELL32.DLL
regsvr32 /s shsvcs.DLL
regsvr32 /s /i swprv.DLL
regsvr32 /s tschannel.DLL
regsvr32 /s USERENV.DLL
regsvr32 /s vss_ps.DLL
regsvr32 /s wbemcons.DLL
regsvr32 /s wbemcore.DLL
regsvr32 /s wbemess.DLL
regsvr32 /s wbemsvc.DLL
regsvr32 /s WINHTTP.DLL
regsvr32 /s WINTRUST.DLL
regsvr32 /s wmiprvsd.DLL
regsvr32 /s wmisvc.DLL
regsvr32 /s wmiutils.DLL
regsvr32 /s wuaueng.DLL
sfc /SCANFILE=%windir%\system32\catsrv.DLL
sfc /SCANFILE=%windir%\system32\catsrvut.DLL
sfc /SCANFILE=%windir%\system32\CLBCatQ.DLL
net start "COM+ Event System"

[bcscholl]

I had a similar issue fixed by creating a script to restart the Hyper-V Volume Shadow Copy Requestor [vmicvss]. We run the script once a day for all VMs in the backup job.

[LinusH]

Hi,
Just for information, we also have the error VSS_WS_FAILED_AT_PREPARE_SNAPSHOT and in our case it's on our Exchange running on vmware. I will create a support ticket soon but wanted you to know that you're not alone...

Br Linus Hellsing

[marcinbojko]

I am interested - is there any general solution from Veeam engineer? From my point of view - restarting VM helps, but in production environment it is not possible to restart working SQL server every 24 or 12h.

[foggy]

Marcin, you can open your own case for direct faster communication with our engineers.

[marcinbojko]

Foggy,
I was hoping for general solution, but in case of recurring issue, I will be forced to.

[Tibo]

Same problem here on Hyper-V 2012 R2 with a 2008 R2 VM and a 2012 R2 VM ...

I still have no official fix from Veeam support (Case opened #00914558 for more than two weeks and tons of logs / tests have been made :-/ )

[JoshRountree]

Did anyone ever find a resolution to this that didn't involve disabling crash consistent backups?

[foggy]

Josh, please contact technical support, since reasons of the issue in your particular case can be different.

[1971merlin]

For us, issuing the command DISKPART AUTOMOUNT SCRUB from the commandline of the VM fixed the issue.

My Theory is that VEEAM somehow reads the automount data (Registry?) and tries to mount all the fixed disk volumes in order to back them up. Disks such as iSCSI are considered local (i.e. not removable) therefore fail the mount (because they no longer exist), in turn this fails the VSS snapshot and also generates the erronous "format" dialog on the console.

I discovered this because every server I've had this issue on, has previously backed up using Windows server Backup, to an iSCSI volume - that was the common denominator that pointed to the issue for us.

[SamiKarhula]

we had same problem one customer who wants testing Veeam and we cannot get that works with Veeam support. I made demo lab to my home with server 2016 Cluster shared volume and i
get same error in home what i get on customers server. I think first there is on 2008r2 cluster something old crap from backup exec (sorry for dirty word:D)

[Mike Resseler]

Hi Sami,

I am not going to use the word You state that you created a demo lab at home with server 2016 and that you received the same error. Is that with Veeam B&R 9.5 (released yesterday)? Because that is the only version that is supported for Windows Server 2016. Also, the VM you are trying to backup, is that the same one as the one with the customer? Or a brand new installed VM

Thanks
Mike