Error Backing up MS SQLAG Node VSS Snapshot Error

[rciscon]

I've been working with the Veeam support team on this on and off for about 6 weeks with zero luck. Case Number: 07072471 - Here's the situation:

I have a Cisco UCS B2000-M5 server blade running Windows Server 2019 and Microsoft SQL 2019 Always On with version 5 of the Veeam Agent for Microsoft Windows on it.

We have three nodes on this Always On group, and Nodes 1 and 3 fail at every attempt at a backup. Node 2 is able to backup successfully. Node 3 is at a different geographical location with similar hardware.

I've been doing the testing on the 3rd Node because it's the farthest removed from our Production activity.

Here's the error messages I've been receiving:

Code: Select all

1/9/2024 4:33:30 PM :: Creating VSS snapshot Error: Failed to create snapshot: Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{2131e63e-f1fc-444b-a789-c23b388e1362}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f0].  

1/9/2024 4:34:07 PM :: Error: Failed to create snapshot: Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{2131e63e-f1fc-444b-a789-c23b388e1362}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f0].  

Has anyone out there had any similar issues? Please let me know if there's anymore information you need.

[Mildur]

Hi Raymond

I'll ping support management to ask for an update of the case.
MSSQL VSS writer are managed and run by the Microsoft SQL application. Have you considered to open a case with Microsoft?

Best,
Fabian

[rciscon]

Fabian,

Thank you for the update.

At one point in time the Veeam support tech was talking about bringing Microsoft into this issue, but nothing ever came of this.

Bringing Microsoft in from our side doesn't really make sense because the SQL server itself is functioning normally, it's the Veeam Agent for Windows that is failing.

Cheers,
Ray

[PetrM]

Hi Raymond,

I would say that the issue is related to SQL Server VSS writer, I doubt that the issue comes from our code. Veeam Agent for Windows just sends a request to create a shadow copy while the shadow copy creation process is managed by VSS framework. Also, the state VSS_WS_FAILED_AT_PREPARE_SNAPSHOT usually indicates an issue at the OS level (check VSSADMIN LIST WRITERS output).

You may ask our engineer to help you with manual testing of VSS framework according to the instructions in this KB. If the diskshadow tool fails with the same error, it will be a good starting point for communication with Microsoft support. Also, you may collect vsstrace output and Windows event logs so that our support engineers can review them.

Thanks!