Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAPSHOT

[Link State]

Hi guys, i got problem whit my Application Backup VM , i have already open case: ID#00498719

Veeam version in 7.0.0.771, the configuration is OnHostbackup installed on Windows 2012 Hyperv - ISCSI storage attached p2000 HP. i have Create application group for backup all Vm on cluster Windows 2012 R2, i got this error on 2 VM.

Code: Select all

 Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Veeam application-aware processing). Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. --tr:Failed to verify writers state. --tr:Failed to create VSS snapshot. --tr:Failed to perform pre-backup tasks. Make sure VM does not have 'iSCSI Software Target Storage Provider' feature installed. Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. --tr:Failed to verify writers state. --tr:Failed to create VSS snapshot. --tr:Failed to perform pre-backup tasks. 

[foggy]

What kind of VM it is (what applications does it run)? Do you see any related messages in the Windows Event Log on this VM?

[matt306]

I've had similar errors on both a 2012 and 2012R2 cluster, if this is only happening on a couple of VM's (and other VM's backup onhost OK) then try one or more of the following:

• Check free disk space (under 1Gb can throw up errors that aren't obvious)
  Rebooting the affected VM
  Ensure the iSCSI service is disabled (if not required)
  Drain the node, reboot and move the VM's back.

[Link State]

What kind of VM it is (what applications does it run)? Do you see any related messages in the Windows Event Log on this VM?

All VM Windows based, 2 VM fails backup it is Enterprise Edition w2k3 Sp2, Running instance of Bussines Object XI Realese 2.

EventLog pick from VM Fail backup:

Event Type: Error
Event Source: VSS
Event Category: None
Event ID: 12293
Date: 05/01/2014
Time: 4.50.48
User: N/A

Volume Shadow Copy Service error: Error calling a routine on the Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details BeginPrepareSnapshot({c40b599f-c15b-4167-88b0-38feb9c1b2d1},{af575fa2-8818-44b7-b007-282f05832859},\\?\Volume{34d7db38-6bb4-11e3-ab1b-806e6f6e6963}\) [hr = 0x80070020].

[Link State]

I've had similar errors on both a 2012 and 2012R2 cluster, if this is only happening on a couple of VM's (and other VM's backup onhost OK) then try one or more of the following:

• Check free disk space (under 1Gb can throw up errors that aren't obvious)
  Rebooting the affected VM
  Ensure the iSCSI service is disabled (if not required)
  Drain the node, reboot and move the VM's back.

Disck Space its ok,
iSCI in not enabled
i'll try Drain node and need reboot node where hosted VM?

[matt306]

Sorry, I made the assumption you were running a cluster, if you've only got the one host, then you will need to schedule downtime for your VM's.

[Link State]

Sorry, I made the assumption you were running a cluster, if you've only got the one host, then you will need to schedule downtime for your VM's.

No mate, i running a cluster, but the 2 VM failed backup are one for each node.
i can switch all VM running on first node and reboot second node Host hyperv and viceversa.
right?

[matt306]

Yep in Cluster manager right click the node and drain roles, once you've checked that they've moved and any disks/cluster resources have been moved to the other node, reboot it. When it comes back up right click on the node and click resume and fail roles back. Repeat for the other node and you're done.

[Link State]

Ok guys, i have reboot all 2 node cluster:
il rerun job, Now all VM failed backup

Code: Select all

[08.01.2014 09.05.39] <10> Info     [ServerX] Connection to the proxy service is established. Endpoint: '1x.6x.x.x:6163'
[08.01.2014 09.05.39] <10> Error    [ServerX]  Guest processing skipped (check guest OS VSS state and integration components version)
[08.01.2014 09.05.39] <10> Error    [ServerX]  Guest processing skipped (check guest OS VSS state and integration components version)
[08.01.2014 09.05.39] <10> Warning  [RTS] [ServerX]  Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Crash consistent).
[08.01.2014 09.05.39] <10> Warning  [ServerX]  Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Crash consistent).
[08.01.2014 09.05.39] <10> Error    Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvVmGuest.ApplyGuestSkippedMessage(Boolean& guestSkippedMessageApplied, CTaskLogger log, String message, EMessageSeverity severity)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvVmGuest.InitList(CHvSnapshotAttemptCtx snapGroup, List`1 guestsList)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvOnHostSnapshot..ctor(CHvSnapshotAttemptCtx snapGroup)
[08.01.2014 09.05.39] <10> Error    Guest processing skipped (check guest OS VSS state and integration components version) (System.Exception)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvVmGuest.ApplyGuestSkippedMessage(Boolean& guestSkippedMessageApplied, CTaskLogger log, String message, EMessageSeverity severity)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvVmGuest.InitList(CHvSnapshotAttemptCtx snapGroup, List`1 guestsList)
[08.01.2014 09.05.39] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvOnHostSnapshot..ctor(CHvSnapshotAttemptCtx snapGroup)
[08.01.2014 09.05.39] <10> Warning  [RTS] Marking snapshot '00000000-0000-0000-0000-000000000000' for deletion: snapshot was not completed
[08.01.2014 09.05.39] <10> Warning  [ServerX]  Marking snapshot '00000000-0000-0000-0000-000000000000' for deletion: snapshot was not completed
[08.01.2014 09.05.39] <10> Info             [hv-ServerX'] Closing snapshot '00000000-0000-0000-0000-000000000000'.
[08.01.2014 09:05:39] <48> Error    Snapshot with id '{00000000-0000-0000-0000-000000000000}' was not found.
[08.01.2014 09:05:39] <48> Error    --tr:Failed to close VSS snapshot '{00000000-0000-0000-0000-000000000000}'.
[08.01.2014 09.05.40] <10> Error    Failed to call RPC function 'HvCloseSnapshot'.
[08.01.2014 09.05.40] <10> Error    Snapshot with id '{00000000-0000-0000-0000-000000000000}' was not found.\n--tr:Failed to close VSS snapshot '{00000000-0000-0000-0000-000000000000}'. (System.Runtime.InteropServices.COMException)
[08.01.2014 09.05.40] <10> Error       at VeeamProxyClientDllLib.VeeamProxyInvokerClass.DoRpc(String cmdName, String inXml, String& outXml)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.ProxyProvider.CRpcInvoker.Call(String methodName, CInvokeInputArg inputArgs)
[08.01.2014 09.05.40] <10> Error    [RTS] Failed to close current snapshot.
[08.01.2014 09.05.40] <10> Error    Snapshot with id '{00000000-0000-0000-0000-000000000000}' was not found.\nArguments:  (Veeam.Backup.Common.CAppException)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.ProxyProvider.CRpcInvoker.Call(String methodName, CInvokeInputArg inputArgs)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.ProxyProvider.CVeeamHvOnhostVss.CloseSnapshot(Guid snapshotID)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvOnHostSnapshot.SafeCloseCurrentSnapshot()
[08.01.2014 09.05.40] <10> Error    [ServerX]  Failed to close current snapshot.
[08.01.2014 09.05.40] <10> Error    Snapshot with id '{00000000-0000-0000-0000-000000000000}' was not found.\nArguments:  (Veeam.Backup.Common.CAppException)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.ProxyProvider.CRpcInvoker.Call(String methodName, CInvokeInputArg inputArgs)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.ProxyProvider.CVeeamHvOnhostVss.CloseSnapshot(Guid snapshotID)
[08.01.2014 09.05.40] <10> Error       at Veeam.Backup.Core.ResourceScheduler.CHvOnHostSnapshot.SafeCloseCurrentSnapshot()

error on all VM in the cluster:

Code: Select all

Guest processing skipped (check guest OS VSS state and integration components version)
Unable to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Crash consistent). Details: Guest processing skipped (check guest OS VSS state and integration components version)
Error: Guest processing skipped (check guest OS VSS state and integration components version)

Any tips?

[Link State]

ok Guys, i have fixed this problem.

My cluster run 2 node Windows 2012 R2 Host Hyperv , storage is iSCSI p2000 target on Hosts.
I have Drained node and reboot Hyperv hosts (no downitme for guest VM), i have rebbot only affected VM not backuped from the job.

1) I have deleted job and related backup file on storage.

2) Recreated job, in Storage Wizard(Advanced Setting) enabled HyperV Guest-quiescence and take crash consistent backup.



3) in Guest processing Wizard i have unchecked Enable application-aware image processing.



4) in Guest processing Wizard Advanced Option i have setting up Administrativr Account for guest VM.



I have run job, now backing up at 92%, total backup all Vm on cluster is 762 Gb.
Now backingup 2 affected VM and successfully backuped.

A question, 2 Windows 2k8 R2 VM there are 2 instances of Oracle, with this backup method i'm sure to be able to sucessfully restore the two windows server runs Oracle in good consistency state?

Edit: Hyper-V Backup job: PkService (Full) Success!!!!

This problem is related "guest OS VSS state and integration components" version problem in Win 2012 R2 ?

Thanks

[Vitaliy S.]

Thanks for sharing the steps that have helped you.

A question, 2 Windows 2k8 R2 VM there are 2 instances of Oracle, with this backup method i'm sure to be able to sucessfully restore the two windows server runs Oracle in good consistency state?

I believe so, as all transaction based applications and especially Oracle should handle crash consistent backups.

This problem is related "guest OS VSS state and integration components" version problem in Win 2012 R2 ?

It's hard to say without seeing full log files and comparing both backup job runs. I would still suggest asking your support engineer to review the logs and determine why application-aware image processing causes these kind of errors.

On a side note, can you please tell me if you can create a VSS snapshot (using Windows built-in backup functionality) on the affected VMs?

[Link State]

It's hard to say without seeing full log files and comparing both backup job runs. I would still suggest asking your support engineer to review the logs and determine why application-aware image processing causes these kind of errors.

On a side note, can you please tell me if you can create a VSS snapshot (using Windows built-in backup functionality) on the affected VMs?

When I opened the case, I've uploaded full log backup job failed for Veeam team support, if need i can upload full log successfuly backup.

The 2 affected VM is win 2k3 SP2, for test VSS snapshot (using Windows built-in backup functionality) i hope need use local ntbackup from windows 2003 Sp2, right?

Thanks Veaam for Backup Accademy, Following your online training, I am certified Backup Academy, this has allowed me to learn and improve my skills on virtualization and backup enviroment.

[Vitaliy S.]

When I opened the case, I've uploaded full log backup job failed for Veeam team support, if need i can upload full log successfuly backup.

Yes, please do it and ask your engineer for assistance with reviewing both log files.

The 2 affected VM is win 2k3 SP2, for test VSS snapshot (using Windows built-in backup functionality) i hope need use local ntbackup from windows 2003 Sp2, right?

Correct. Alternatively, you can follow this blog post to run your tests > http://axelilly.wordpress.com/2009/01/1 ... 03-server/

Thanks Veaam for Backup Accademy, Following your online training, I am certified Backup Academy, this has allowed me to learn and improve my skills on virtualization and backup enviroment.

Awesome feedback! Glad that you have found this training useful.

[Link State]

Hey Vitaly, i have best idea, i can create Application Group whit all my previously backuped production enviroment + one Domain controller for AD, DNS functionality, and setting-up Virtual Lab for testing backuped production enviroment, i'll create one Test Enviroment for many test [test VSS snapshot (using Windows built-in backup functionality)], check if Vm windows\Orcale DB and NLB server for my web application work good.
Stay Tuned
Regards

[Vitaliy S.]

Indeed, a much better idea!

[Link State]

ok Guys can edit thread in [Fixed]Win 2012 R2 Cluster Failed VSS_WS_FAILED_AT_PREPARE_SNAPSHOT ?

[Vitaliy S.]

Yes, sure, but how did you manage to fix VSS snapshot errors? Can you please share?

[Link State]

Yes, sure, but how did you manage to fix VSS snapshot errors? Can you please share?

For this related error VSS_WS_FAILED_AT_PREPARE_SNAP on cluster win2012 R2 work around is:
1) job, in Storage Wizard(Advanced Setting) enabled HyperV Guest-quiescence and take crash consistent backup.
2) in Guest processing Wizard i have unchecked Enable application-aware image processing.
3) in Guest processing Wizard Advanced Option i have setting up Administrativr Account for guest VM.

Scheduled backup Job run this night!!!

[veremin]

But the VSS snapshot issue isn't fixed yet and you're still using non-recommended crash-consistent mode, right? Thanks.

[Link State]

But the VSS snapshot issue isn't fixed yet and you're still using non-recommended crash-consistent mode, right? Thanks.

Veeam engineers should compare logs of the failed job with the job completed successfully backup, I wrote to the support team if you want the log succesffully backup, veeam team already counts with the previous log backup failed with image process-aware enabled, but i have not had an answer ...
Ill make test VSS snapshot whit ntbackup on affected VM, when im setting-up my Test production Virtual Lab.

[veremin]

Ill make test VSS snapshot whit ntbackup on affected VM, when im setting-up my Test production Virtual Lab.

Kindly, keep us updated about the result you get. Thanks.

[Link State]

Kindly, keep us updated about the result you get. Thanks.

So i have tested VSS snapshot whit vshadow.exe http://axelilly.wordpress.com/2009/01/1 ... 03-server/

1) Created shapshot on affected VM Windows 2003 SP2 Ent Edition hosted on Windows 2012 R2 Cluster HyperV.

C:\Program Files\Microsoft\VSSSDK72\Tools\VSSReports>vshadow.exe -p c:

VSHADOW.EXE 2.2 - Volume Shadow Copy sample client
Copyright (C) 2005 Microsoft Corporation. All rights reserved.


(Option: Persistent shadow copy)
(Option: Create shadow copy set)
- Setting the VSS context to: 0x00000009
(Gathering writer metadata...)
(Waiting for the asynchronous operation to finish...)
Initialize writer metadata ...
Discover directly excluded components ...
- Excluding writer 'MSDEWriter' since it has no selected components for restore.

- Excluding writer 'BITS Writer' since it has no selected components for restore
.
Discover components that reside outside the shadow set ...
Discover all excluded components ...
Discover excluded writers ...
Discover explicitly included components ...
Verifying explicitly specified writers/components ...
Select explicitly included components ...
* Writer 'System Writer':
- Add component \System Files
* Writer 'COM+ REGDB Writer':
- Add component \COM+ REGDB
* Writer 'Registry Writer':
- Add component \Registry
* Writer 'Event Log Writer':
- Add component \Event Logs
* Writer 'WMI Writer':
- Add component \WMI
Creating shadow set {3bcd475f-3933-40ef-ae31-c6d25df0f035} ...
- Adding volume \\?\Volume{c8d52eb4-51f4-11e3-bfbd-806e6f6e6963}\ [C:\] to the s
hadow set...
Preparing for backup ...
(Waiting for the asynchronous operation to finish...)
(Waiting for the asynchronous operation to finish...)
Creating the shadow (DoSnapshotSet) ...
(Waiting for the asynchronous operation to finish...)
(Waiting for the asynchronous operation to finish...)
Shadow copy set succesfully created.

List of created shadow copies:


Querying all shadow copies with the SnapshotSetID {3bcd475f-3933-40ef-ae31-c6d25df0f035} ...

* SNAPSHOT ID = {2e80e2a9-8668-44e6-9c78-c277befce08f} ...
- Shadow copy Set: {3bcd475f-3933-40ef-ae31-c6d25df0f035}
- Original count of shadow copies = 1
- Original Volume name: \\?\Volume{c8d52eb4-51f4-11e3-bfbd-806e6f6e6963}\ [C:\]
- Creation Time: 09/01/2014 16.03.46
- Shadow copy device name: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
- Originating machine: serverx.domain
- Service machine: serverx.domain
- Not Exposed
- Provider id: {b5946137-7b9f-4925-af80-51abd60b20d5}
- Attributes: No_Auto_Release Persistent Differential

- Mark all writers as succesfully backed up...
Completing the backup (BackupComplete) ...
(Waiting for the asynchronous operation to finish...)
(Waiting for the asynchronous operation to finish...)

2) Verified snapshot Created vshadow.exe -q :


3) mounting snapshot: vshadow.exe -el={2e80e2a9-8668-44e6-9c78-c277befce08f},X:


4) bowser snapshot mounted all ok:


5) deleting snapshot:


My opinion is compatibility problem whit integration service whit windows 2012 R2 and Veeam is possible?
Is possible Veeam dont detect version of the integration service on VM windows 2003?
I have create separate job only for windows 2003 whit disabled Enable application-aware image processing.

I hope that this test will help you.

[Vitaliy S.]

My opinion is compatibility problem whit integration service whit windows 2012 R2 and Veeam is possible?

Windows Server 2012 R2 is fully supported, so let's wait for the update from our technical team.

Is possible Veeam dont detect version of the integration service on VM windows 2003?

That should be stated in the backup job logs. To make sure that everything is ok with integration services, you may want yo re-deploy them on these VMs.

[Link State]

Windows Server 2012 R2 is fully supported, so let's wait for the update from our technical team. That should be stated in the backup job logs. To make sure that everything is ok with integration services, you may want yo re-deploy them on these VMs.

ok waiting your engineer

But in failed job on Veeam there is this message:

Code: Select all

09/01/2014 17.39.06 :: Processing 'SERVERX' Error: Guest processing skipped (check guest OS VSS state and integration components version)09/01/2014 17.42.34 :: Make sure VM does not have 'iSCSI Software Target Storage Provider' feature installed.

[Vitaliy S.]

Seems like there is an existing topic discussing a similar problem, please check it out > Job warnings after upgrade to v7

[Link State]

Nothing news from your Veeam engineer for my case ID#00498719

[Vitaliy S.]

Your case is in open state, just checked, so I believe our engineer will reach out to you soon.

[Link State]

veeam support conctact me!

[veremin]

I've just checked the case, and it seems that you've scheduled a session with our engineer for tomorrow. So, let's see how well it goes. Thanks.

[C.Lane]

Was there any resolution to this case? I am experiencing the same issue with a Win2K3 failing to create a snapshot. I can create an NT backup ok, have plenty of disk space. Can't create a backup using the suggested job changes, though.