[SOLUTION] the strange case of VSS panthom error

[lucaliga]

Hello,

First of all sorry for my bad english. I want to share my experience to avoid someone of you to spend many days in debugging, troubleshooting, reading tech docs, many coffes and cigarettes and headaches.

The story: our company acquire one little company with one ESXi host V.6.0 without a professional/business backup.
We have enought VUL and corporate IT manager asks my team to add this new host into corporate Veeam Backup & Replication 10.0.0.4461 enviroment. Seems so simple. There is only 1 VM: Windows 2012 R2 AD DC with installed Microsoft SQL 2008R2 one instance with 3 tiny db (sum is less than 2 GB).
I started creating in the ESXi host a second VM Windows 2012 R2 for hosting Veeam services then add to the backup infrastructure: Esxi host, the new VM as backup proxy and one existing little NAS as repository.

I created a new backup job, selecting the two VM, scheduled at night with usual Application-Aware Processing enabled.

Next morning the nightmare starts. Veeam backup job successfully backup the VM proxy but reports for the VM AD and SQL:

Code: Select all

[02.07.2020 22:46:52] <28> Info     VSSControl: Frozen in 8sec
[02.07.2020 22:46:52] <28> Info     [VimApi] Create snapshot, ref "40", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[02.07.2020 22:46:55] <28> Info     [Soap] Snapshot VM '40' was created. Ref: '40-snapshot-111'
[02.07.2020 22:46:55] <28> Info     VSSControl: Unfreezing
[02.07.2020 22:47:01] <28> Error    Unable to release guest. Error: Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{f80b2378-e9f8-4ce1-b934-41dc84a7a471}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f0].].

Veeam logs don't show any another information. There are no error nor warning in Windows event viewer of the VM during backup job running time. We made any kind of troubleshooting during fours days: no results. Since the beginning my idea was Veeam product has nothing to do with the issue but it only suffers of some other component problem. Google search about VSS SNAPSHOT ERROR returns many results but none exactly related to the issue we suffer.
To make long story short thanks also to https://www.veeam.com/kb1980 I test VSS using diskshadow utility and it reports:

Component <xxxxxxxxx> from writer "SqlServerWriter" is excluded from backup because it requires a volume that cannot be found on this system.

Where <xxxxxxxxx> is server\instance\dbname. Again in diskshadow but triggering the specific VSS writer "SqlServerWriter" we found (not so evident nor highlighted):

WRITER "SqlServerWriter"
.......<omitted>.....
- Volumes affected by this component:
- \\SERVER01\BUSDIR\ [Does not exist]

You have to render and construe in you mind both the statements "is excluded from backup because it requires a volume that cannot be found on this system" and "[Does not exist]" because are unclear and incorrect. The share EXIST and is on the server himself: is a local shared folder.
These little warning cause is another and originated by an unsupported configuration made in Microsoft SQ but allowed by the application.
Executing SQL query: 'SELECT DB_NAME(database_id) AS DatabaseName, Name AS Logical_Name, type_desc AS Type, Physical_Name AS Location, state_desc AS OnlineStatus FROM sys.master_files' we got that one of three SQL db has Physical_Name inside folder \\SERVER01\BUSDIR\. This is allowed by SQL server but not by the SQL Server's writer used to support Microsoft Volume Shadow Copy Service ("SqlServerWriter") because it support only local volumes, NOT external volumes like UNC path, even if the UNC path point to the local system himself.

The error Veeam B&R job shows, he got from OS and now we can translate this way: sorry cant Snapshot VM using VSS because one of the triggered VSS writer [SqlServerWriter] returns he cant VSS one db due to unsupported location type.

I Hope this help someone.

Thanks for attention.
Ciao

[Gostev]

Thanks for sharing!

[rjv1971]

Thanks for this write-up. We had this exact case, tickets with Veeam and Microsoft..... removing UNC paths fixed it.

[DWRAFTER]

Case ID 05120807

I am getting this error message when run a backup job on one of our Domain Controllers running Veeam Backup & Replication 10.

Processing MMSS-VDC19 Error: Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{0d6e410a-4be0-44fe-a6a2-1d734909a31c}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].].

Steps taken to resolve the problem to date are:

Restarted the server hosting the Veeam Backup & Replication 10 software.
Restarting the Domain Controller.
Restarting the SQL Server VSS WRITER service with elevated privileges.
Recreating the Backup job from scratch. Rerunning the backup job once recreated.

Backup server is running Veeam Backup & Replication 10. Microsoft SQL Server 2014, running on Windows Server 2008 R2 standard.

Domain Controller is running Windows Server 2019 Standard.

All other backup jobs are running successfully on all other servers.

[Natalia Lupacheva]

Hi David,

Thank you for sharing your case ID and the detailed description!
I've moved your post to the existing thread with a similar issue. Maybe it's not exactly the same, but might give you a hint.

Thanks!

[TinchoB]

Hello All,
I want to share with you my experience. I had the same error as reported by @Lucaglia in the first post. The problem was exactly the same, with the same resolution.

A couple of weeks ago one of our Backup JOBs started to fail with the infamous error:
Unable to release guest. Error: Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{77ff0021-b8c9-48ed-b8ac-9556127e5364}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f0].].
Error: Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{77ff0021-b8c9-48ed-b8ac-9556127e5364}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f0].].

The Virtual Machine has Windows Server 2012 with SQL Server. For this reasong JOB was set up to do a Guest Processing in SQL Copy Mode. For this example hostname=ServerName (name not real but valid for explanatory purposes).

After a week of researching and researching again, we open a case in support. They instructed me to follow the guidelines of the following KB to produce a manual DISKSHADOW in "CopyMode":
https://www.veeam.com/kb1980?ad=in-text-link

They determined it was not a VEEAM issue and request us to check with Microsoft support.


After some digging, I found this post.

Then I decided to analyzed the OUTPUT.TXT file produced as a result of the manual DISKSAHDOW process of the VEEAM KB.

Inside it I found the following warning:

Component "Server_Name\SBCG" from writer "SqlServerWriter" is excluded from backup because it requires a volume that cannot be found on this system.

Then boom, we found the issue !!
In Management Studio we discovered on the SQL databases (SBCG one) was attached to the SQL Server but NOT as Local (C:\SHARE\SBCG) in fact a UNC path (\\ServerName\SHARE\SBCG.mdf...)... the same database as the DISKSHADOWN OUTPUT.TXT warning....

For some unknown reason a guy from Application Support decided to mount a small template database (SBCG), but for another unknown reason decided to
put the template DB in a share of the same Server, and then mounted it from UNC path.... ok SQL liked it... no problem... but SqlServerWriter not !!!!

The explanation we found:
When DISKSHADOW is invoked by VEEAM, Windows needs to produce a Shadow Copy with 100% integrity. For this purpose it calls different VSS Writers... for SQL Server it uses SqlServerWriter... As he needs to get a 100% integrity, none warning or errors are expected... if one warning happens (as the one we got in OUTPUT.TXT) all the processes is cancel.

After detaching the DB and re-attaching it but as a Local path, VEEAM JOB got happy and started to works as usual without any problems !!

So avoid attaching databases with UNC path !!!

Thank you very Much !!

[PetrM]

Hi Martin,

Good research, it is very helpful for the Community! Your feedback is highly appreciated!

Thanks!

[Sysmaint]

Solutions for me.
Running vmware vSphere Client version 7.0.3.01700

Select Computer.
Select Updates
VMware Tools show- Up to Date.
But..
VM Hardware Compatibility was NOT in shape.
Run Upgrade to match host.