Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAPSHOT

Hyper-V specific discussions

Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAPSHOT

Veeam Logoby Link State » Mon Jan 06, 2014 11:04 am

Hi guys, i got problem whit my Application Backup VM , i have already open case: ID#00498719

Veeam version in 7.0.0.771, the configuration is OnHostbackup installed on Windows 2012 Hyperv - ISCSI storage attached p2000 HP. i have Create application group for backup all Vm on cluster Windows 2012 R2, i got this error on 2 VM.


Code: Select all
 Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Veeam application-aware processing). Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. --tr:Failed to verify writers state. --tr:Failed to create VSS snapshot. --tr:Failed to perform pre-backup tasks. Make sure VM does not have 'iSCSI Software Target Storage Provider' feature installed. Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. --tr:Failed to verify writers state. --tr:Failed to create VSS snapshot. --tr:Failed to perform pre-backup tasks.
MCSA 2008R2 - 2012 R2 - MCSE 2008R2 - 2012R2 - MCSE: Cloud Platform and Infrastructure
Link State
Enthusiast
 
Posts: 26
Liked: 3 times
Joined: Mon Jan 06, 2014 9:58 am
Full Name: Prodigy

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby foggy » Mon Jan 06, 2014 1:22 pm

What kind of VM it is (what applications does it run)? Do you see any related messages in the Windows Event Log on this VM?
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby matt306 » Mon Jan 06, 2014 2:33 pm 1 person likes this post

I've had similar errors on both a 2012 and 2012R2 cluster, if this is only happening on a couple of VM's (and other VM's backup onhost OK) then try one or more of the following:

    Check free disk space (under 1Gb can throw up errors that aren't obvious)
    Rebooting the affected VM
    Ensure the iSCSI service is disabled (if not required)
    Drain the node, reboot and move the VM's back.
matt306
Influencer
 
Posts: 19
Liked: 11 times
Joined: Mon Jan 06, 2014 2:29 pm
Full Name: Matthew Brown

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby Link State » Mon Jan 06, 2014 4:27 pm

foggy wrote:What kind of VM it is (what applications does it run)? Do you see any related messages in the Windows Event Log on this VM?

All VM Windows based, 2 VM fails backup it is Enterprise Edition w2k3 Sp2, Running instance of Bussines Object XI Realese 2.

EventLog pick from VM Fail backup:

Event Type:Error
Event Source:VSS
Event Category:None
Event ID:12293
Date:05/01/2014
Time:4.50.48
User:N/A
Volume Shadow Copy Service error: Error calling a routine on the Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details BeginPrepareSnapshot({c40b599f-c15b-4167-88b0-38feb9c1b2d1},{af575fa2-8818-44b7-b007-282f05832859},\\?\Volume{34d7db38-6bb4-11e3-ab1b-806e6f6e6963}\) [hr = 0x80070020].
MCSA 2008R2 - 2012 R2 - MCSE 2008R2 - 2012R2 - MCSE: Cloud Platform and Infrastructure
Link State
Enthusiast
 
Posts: 26
Liked: 3 times
Joined: Mon Jan 06, 2014 9:58 am
Full Name: Prodigy

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby Link State » Mon Jan 06, 2014 4:49 pm

matt306 wrote:I've had similar errors on both a 2012 and 2012R2 cluster, if this is only happening on a couple of VM's (and other VM's backup onhost OK) then try one or more of the following:

    Check free disk space (under 1Gb can throw up errors that aren't obvious)
    Rebooting the affected VM
    Ensure the iSCSI service is disabled (if not required)
    Drain the node, reboot and move the VM's back.


Disck Space its ok,
iSCI in not enabled
i'll try Drain node and need reboot node where hosted VM?
MCSA 2008R2 - 2012 R2 - MCSE 2008R2 - 2012R2 - MCSE: Cloud Platform and Infrastructure
Link State
Enthusiast
 
Posts: 26
Liked: 3 times
Joined: Mon Jan 06, 2014 9:58 am
Full Name: Prodigy

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby matt306 » Mon Jan 06, 2014 7:26 pm

Sorry, I made the assumption you were running a cluster, if you've only got the one host, then you will need to schedule downtime for your VM's.
matt306
Influencer
 
Posts: 19
Liked: 11 times
Joined: Mon Jan 06, 2014 2:29 pm
Full Name: Matthew Brown

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby Link State » Tue Jan 07, 2014 12:18 pm

matt306 wrote:Sorry, I made the assumption you were running a cluster, if you've only got the one host, then you will need to schedule downtime for your VM's.


No mate, i running a cluster, but the 2 VM failed backup are one for each node.
i can switch all VM running on first node and reboot second node Host hyperv and viceversa.
right? :)
MCSA 2008R2 - 2012 R2 - MCSE 2008R2 - 2012R2 - MCSE: Cloud Platform and Infrastructure
Link State
Enthusiast
 
Posts: 26
Liked: 3 times
Joined: Mon Jan 06, 2014 9:58 am
Full Name: Prodigy

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby matt306 » Tue Jan 07, 2014 12:50 pm

Yep in Cluster manager right click the node and drain roles, once you've checked that they've moved and any disks/cluster resources have been moved to the other node, reboot it. When it comes back up right click on the node and click resume and fail roles back. Repeat for the other node and you're done.
matt306
Influencer
 
Posts: 19
Liked: 11 times
Joined: Mon Jan 06, 2014 2:29 pm
Full Name: Matthew Brown

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby Link State » Wed Jan 08, 2014 9:17 am

Ok guys, i have reboot all 2 node cluster:
il rerun job, Now all VM failed backup

Code: Select all
[08.01.2014 09.05.39] <10> Info     [ServerX] Connection to the proxy service is established. Endpoint: '1x.6x.x.x:6163'
[08.01.2014 09.05.39] <10> Error    [ServerX]  Guest processing skipped (check guest OS VSS state and integration components version)
[08.01.2014 09.05.39] <10> Error    [ServerX]  Guest processing skipped (check guest OS VSS state and integration components version)
[08.01.2014 09.05.39] <10> Warning  [RTS] [ServerX]  Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Crash consistent).
[08.01.2014 09.05.39] <10> Warning  [ServerX]  Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Crash consistent).
[08.01.2014 09.05.39] <10> Error    Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvVmGuest.ApplyGuestSkippedMessage(Boolean& guestSkippedMessageApplied, CTaskLogger log, String message, EMessageSeverity severity)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvVmGuest.InitList(CHvSnapshotAttemptCtx snapGroup, List`1 guestsList)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvOnHostSnapshot..ctor(CHvSnapshotAttemptCtx snapGroup)
[08.01.2014 09.05.39] <10> Error    Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvVmGuest.ApplyGuestSkippedMessage(Boolean& guestSkippedMessageApplied, CTaskLogger log, String message, EMessageSeverity severity)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvVmGuest.InitList(CHvSnapshotAttemptCtx snapGroup, List`1 guestsList)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvOnHostSnapshot..ctor(CHvSnapshotAttemptCtx snapGroup)
[08.01.2014 09.05.39] <10> Warning  [RTS] Marking snapshot '00000000-0000-0000-0000-000000000000' for deletion: snapshot was not completed
[08.01.2014 09.05.39] <10> Warning  [ServerX]  Marking snapshot '00000000-0000-0000-0000-000000000000' for deletion: snapshot was not completed
[08.01.2014 09.05.39] <10> Info             [hv-ServerX'] Closing snapshot '00000000-0000-0000-0000-000000000000'.
[08.01.2014 09:05:39] <48> Error    Snapshot with id '{00000000-0000-0000-0000-000000000000}' was not found.
[08.01.2014 09:05:39] <48> Error    --tr:Failed to close VSS snapshot '{00000000-0000-0000-0000-000000000000}'.
[08.01.2014 09.05.40] <10> Error    Failed to call RPC function 'HvCloseSnapshot'.
[08.01.2014 09.05.40] <10> Error    Snapshot with id '{00000000-0000-0000-0000-000000000000}' was not found.\n--tr:Failed to close VSS snapshot '{00000000-0000-0000-0000-000000000000}'. (System.Runtime.InteropServices.COMException)
[08.01.2014 09.05.40] <10> Error       at VeeamProxyClientDllLib.VeeamProxyInvokerClass.DoRpc(String cmdName, String inXml, String& outXml)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.ProxyProvider.CRpcInvoker.Call(String methodName, CInvokeInputArg inputArgs)
[08.01.2014 09.05.40] <10> Error    [RTS] Failed to close current snapshot.
[08.01.2014 09.05.40] <10> Error    Snapshot with id '{00000000-0000-0000-0000-000000000000}' was not found.\nArguments:  (Veeam.Backup.Common.CAppException)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.ProxyProvider.CRpcInvoker.Call(String methodName, CInvokeInputArg inputArgs)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.ProxyProvider.CVeeamHvOnhostVss.CloseSnapshot(Guid snapshotID)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvOnHostSnapshot.SafeCloseCurrentSnapshot()
[08.01.2014 09.05.40] <10> Error    [ServerX]  Failed to close current snapshot.
[08.01.2014 09.05.40] <10> Error    Snapshot with id '{00000000-0000-0000-0000-000000000000}' was not found.\nArguments:  (Veeam.Backup.Common.CAppException)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.ProxyProvider.CRpcInvoker.Call(String methodName, CInvokeInputArg inputArgs)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.ProxyProvider.CVeeamHvOnhostVss.CloseSnapshot(Guid snapshotID)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvOnHostSnapshot.SafeCloseCurrentSnapshot()


error on all VM in the cluster:
Code: Select all
Guest processing skipped (check guest OS VSS state and integration components version)
Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Crash consistent). Details: Guest processing skipped (check guest OS VSS state and integration components version)
Error: Guest processing skipped (check guest OS VSS state and integration components version)


Any tips?
MCSA 2008R2 - 2012 R2 - MCSE 2008R2 - 2012R2 - MCSE: Cloud Platform and Infrastructure
Link State
Enthusiast
 
Posts: 26
Liked: 3 times
Joined: Mon Jan 06, 2014 9:58 am
Full Name: Prodigy

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby Link State » Wed Jan 08, 2014 12:32 pm 2 people like this post

ok Guys, i have fixed this problem.

My cluster run 2 node Windows 2012 R2 Host Hyperv , storage is iSCSI p2000 target on Hosts.
I have Drained node and reboot Hyperv hosts (no downitme for guest VM), i have rebbot only affected VM not backuped from the job.

1) I have deleted job and related backup file on storage.

2) Recreated job, in Storage Wizard(Advanced Setting) enabled HyperV Guest-quiescence and take crash consistent backup.

Image

3) in Guest processing Wizard i have unchecked Enable application-aware image processing.

Image

4) in Guest processing Wizard Advanced Option i have setting up Administrativr Account for guest VM.

Image

I have run job, now backing up at 92%, total backup all Vm on cluster is 762 Gb.
Now backingup 2 affected VM and successfully backuped. :mrgreen:

A question, 2 Windows 2k8 R2 VM there are 2 instances of Oracle, with this backup method i'm sure to be able to sucessfully restore the two windows server runs Oracle in good consistency state?

Edit: Hyper-V Backup job: PkService (Full) Success!!!!

This problem is related "guest OS VSS state and integration components" version problem in Win 2012 R2 ?

Thanks
MCSA 2008R2 - 2012 R2 - MCSE 2008R2 - 2012R2 - MCSE: Cloud Platform and Infrastructure
Link State
Enthusiast
 
Posts: 26
Liked: 3 times
Joined: Mon Jan 06, 2014 9:58 am
Full Name: Prodigy

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby Vitaliy S. » Wed Jan 08, 2014 1:00 pm

Thanks for sharing the steps that have helped you.
Link State wrote:A question, 2 Windows 2k8 R2 VM there are 2 instances of Oracle, with this backup method i'm sure to be able to sucessfully restore the two windows server runs Oracle in good consistency state?

I believe so, as all transaction based applications and especially Oracle should handle crash consistent backups.
Link State wrote:This problem is related "guest OS VSS state and integration components" version problem in Win 2012 R2 ?

It's hard to say without seeing full log files and comparing both backup job runs. I would still suggest asking your support engineer to review the logs and determine why application-aware image processing causes these kind of errors.

On a side note, can you please tell me if you can create a VSS snapshot (using Windows built-in backup functionality) on the affected VMs?
Vitaliy S.
Veeam Software
 
Posts: 19539
Liked: 1097 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby Link State » Wed Jan 08, 2014 1:28 pm

Vitaliy S. wrote:It's hard to say without seeing full log files and comparing both backup job runs. I would still suggest asking your support engineer to review the logs and determine why application-aware image processing causes these kind of errors.

On a side note, can you please tell me if you can create a VSS snapshot (using Windows built-in backup functionality) on the affected VMs?


When I opened the case, I've uploaded full log backup job failed for Veeam team support, if need i can upload full log successfuly backup.

The 2 affected VM is win 2k3 SP2, for test VSS snapshot (using Windows built-in backup functionality) i hope need use local ntbackup from windows 2003 Sp2, right?

Thanks Veaam for Backup Accademy, Following your online training, I am certified Backup Academy, this has allowed me to learn and improve my skills on virtualization and backup enviroment.
MCSA 2008R2 - 2012 R2 - MCSE 2008R2 - 2012R2 - MCSE: Cloud Platform and Infrastructure
Link State
Enthusiast
 
Posts: 26
Liked: 3 times
Joined: Mon Jan 06, 2014 9:58 am
Full Name: Prodigy

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby Vitaliy S. » Wed Jan 08, 2014 1:46 pm

Link State wrote:When I opened the case, I've uploaded full log backup job failed for Veeam team support, if need i can upload full log successfuly backup.

Yes, please do it and ask your engineer for assistance with reviewing both log files.
Link State wrote:The 2 affected VM is win 2k3 SP2, for test VSS snapshot (using Windows built-in backup functionality) i hope need use local ntbackup from windows 2003 Sp2, right?

Correct. Alternatively, you can follow this blog post to run your tests > http://axelilly.wordpress.com/2009/01/1 ... 03-server/
Link State wrote:Thanks Veaam for Backup Accademy, Following your online training, I am certified Backup Academy, this has allowed me to learn and improve my skills on virtualization and backup enviroment.

Awesome feedback! Glad that you have found this training useful.
Vitaliy S.
Veeam Software
 
Posts: 19539
Liked: 1097 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby Link State » Wed Jan 08, 2014 4:46 pm

Hey Vitaly, i have best idea, i can create Application Group whit all my previously backuped production enviroment + one Domain controller for AD, DNS functionality, and setting-up Virtual Lab for testing backuped production enviroment, i'll create one Test Enviroment for many test [test VSS snapshot (using Windows built-in backup functionality)], check if Vm windows\Orcale DB and NLB server for my web application work good.
Stay Tuned
Regards
MCSA 2008R2 - 2012 R2 - MCSE 2008R2 - 2012R2 - MCSE: Cloud Platform and Infrastructure
Link State
Enthusiast
 
Posts: 26
Liked: 3 times
Joined: Mon Jan 06, 2014 9:58 am
Full Name: Prodigy

Re: Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAP

Veeam Logoby Vitaliy S. » Wed Jan 08, 2014 7:40 pm

Indeed, a much better idea! :)
Vitaliy S.
Veeam Software
 
Posts: 19539
Liked: 1097 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 8 guests