SQL VSS unfreeze error

[myFist]

Hello

We've got following VSS error on multiple VMware VMs with SQL Express:

Code: Select all

22.03.2018 19:28:59   1356  ERR                                     The state of a backup critical writer is [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2]. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{11070b29-877c-41d7-a481-99c2d06febb6}].
22.03.2018 19:28:59   1356                                              Cancelling changes made to prepare an application for a subsequent restore operation. Application: [NTDS].
22.03.2018 19:28:59   1356                                                  There is no changes related to the NDIS service or changes were already cancelled.
22.03.2018 19:28:59   1356                                              Cancelling changes made to prepare an application for a subsequent restore operation. Application: [NTDS].. Ok.
22.03.2018 19:28:59   1356                                              Cancelling changes (if exists) that were made before the freeze stage in order to support Exchange server writer.
22.03.2018 19:28:59   1356                                                  Cancellation of changes made to support Exchange Server writer is not required.
22.03.2018 19:28:59   1356                                              Cancelling changes (if exists) that were made before the freeze stage in order to support Exchange server writer.. Ok.
22.03.2018 19:28:59   1356                                      Creating a shadow copy of the volumes containing writer's data. Backup type: 'VSS_BT_FULL'.. Failed.
22.03.2018 19:28:59   1356                              Performing backup steps. Backup type: [VSS_BT_FULL].. Failed.
22.03.2018 19:28:59   1356  INFO                        Terminating the snapshot set. Snapshot set ID: [{d6c12572-b31d-4a9e-a09f-d667858dd7eb}].
22.03.2018 19:28:59   1356  INFO                        Terminating the snapshot set. Snapshot set ID: [{d6c12572-b31d-4a9e-a09f-d667858dd7eb}].. Ok.
22.03.2018 19:28:59   1356  ERR                     Freeze job failed.
22.03.2018 19:28:59   1356  ERR                         Backup job failed.
22.03.2018 19:28:59   1356  ERR                         Cannot create a shadow copy of the volumes containing writer's data.
22.03.2018 19:28:59   1356  ERR                         A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{11070b29-877c-41d7-a481-99c2d06febb6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].
22.03.2018 19:28:59   3612                          UnfreezeErrorString: [Backup job failed.
22.03.2018 19:28:59   3612                          Cannot create a shadow copy of the volumes containing writer's data.
22.03.2018 19:28:59   3612                          A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{11070b29-877c-41d7-a481-99c2d06febb6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].]

This error occures with different up to date Microsoft SQL instances (Express 2008 / 2012 / 2014 / 2016). And the first retry of the Veeam Job works without any problems.
Already opened a support call and they told me to contact microsoft support, but maybe you have any other ideas...

Best regards
Robin

[Vitaliy S.]

Hello Robin,

I would suggest to open a Windows Event log and try to find any Microsoft KBs regarding VSS errors you will find there. Also, check that all writers are in the stable state.

Thank you!

[myFist]

Hello Vitaly,

we've done that already:

Code: Select all

- Source: SQLVDI - ID: 1
-> SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=4964. Thread=6068. Server. Instance=MSSQLSERVER. VD=Global\{AEC0E2A8-DF61-4A39-9DC1-27DC6565F107}1_SQLVDIMemoryName_0. 

Code: Select all

- Source: MSSQLSERVER - ID 18210
-> BackupVirtualDeviceFile::SendFileInfoBegin:  failure on backup device '{AEC0E2A8-DF61-4A39-9DC1-27DC6565F107}3'. Operating system error 995(failed to retrieve text for this error. Reason: 15105).

Code: Select all

- Source: VSS - ID: 8229
Von einem VSS Writer wurde ein Ereignis mit dem Fehler "0x800423f2, Das Zeitlimit des Generators für den Zeitraum zwischen dem Freeze- und dem Thaw-Ereignis wurde überschritten.
" zurückgewiesen. Änderungen, die während der Verarbeitung des Ereignisses an den Writerkomponenten vorgenommen wurden, sind für den Anforderer nicht verfügbar. Zugehörige Ereignisse der Hostanwendung des VSS Writer finden Sie im Ereignisprotokoll. 

Vorgang:
   Thaw-Ereignis

Kontext:
   Ausführungskontext: Writer
   Generatorklassen-ID: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Generatorname: Registry Writer
   Generatorinstanz-ID: {3b13e1ad-e5dc-4a47-b01d-39ecea567391}
   Befehlszeile: C:\Windows\system32\vssvc.exe
   Prozess-ID: 1032

The Writers are ok after the first retry.

Best regards
Robin

[Vitaliy S.]

Quick google search on your error lead me to this MSDN article, please review it as it should contain some tips on how to resolve that. This issue seems to relate to lack of threads or virtual memory for the SQL server. Hope this helps!

[mkeo@jmbm.com]

I'm having the issue. Been troubleshooting with Veeam Support for almost two weeks now. Veeam is able to backup successfully if I reboot the SQL server, but the next backup fails. Still no resolution. I hope they don't ask me to call Microsoft.

Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{d3b43e21-4146-4c5f-85d6-f9b54e1c1404}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].].

[Vitaliy S.]

Moni, the error message refers to VSS writer/provider problem, so the best bet would be to investigate Windows Event log and if possible install all latest updates to the SQL Server application.

[myFist]

we investigated that problem (my original post) with Microsoft support and we found the cause now: Sophos AntiVirus... all veeam and sql exclusion are set but we will check the exclusions again

[myFist]

Additonal to my post above:
- Workaround + official statement from Sophos: https://community.sophos.com/products/e ... s-utilized