SQLVSSWriter Errors on multiple servers

[Sanai95]

Hello,

Case numer #05572428.

We are getting silimar errors to this for multiple customers:

"Processing WFO-DC-01 Error: Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unbekannter Fehler. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{7b26d6f7-1b70-4cf7-830d-1776c4dcf2a5}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]."

They are mostly occuring on the Domain Controllers. After restarting the Writers service it sometimes works, but not all the time.
One customer (D) got this error on re SQL Server.

In the eventlog we got the events:
"SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=6220. Thread=8372. Client. Instance=LOCALDB#SHA17731. VD=Global\{43A9108A-7E77-4B2C-B47E-A73DFDE9CBCB}1_SQLVDIMemoryName_0. "

On other server it's another instance(just for a database for a application that the customers uses).
But also this one:
"Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013
Error state: 1, Severity: 16
Source: Microsoft SQL Server Native Client 11.0
Error message: BACKUP DATABASE is terminating abnormally.
SQLSTATE: 42000, Native Error: 18210
Error state: 1, Severity: 16
Source: Microsoft SQL Server Native Client 11.0
Error message: BackupVirtualDeviceSet::Initialize: CoCreateInstance failure on backup device '{43A9108A-7E77-4B2C-B47E-A73DFDE9CBCB}3'. Operating system error 0x800703fa(Es wurde versucht, einen Registrierungsschlüssel einem unzulässigen Vorgang zu unterziehen, der zum Löschen markiert wurde.)."


We belive those may be linked to the lastest microsoft updates.
Those were installed in the last days on the backued up servers:

KB: 890830 [Windows-KB890830-x64-V5.104.exe] - Windows Malicious Software Removal -> F, J, P, D
KB: 5012170 [Windows10.0-KB5012170-x64.cab] - Security Update for Boot DBX -> F, J and D (P has to install updates, that why the update ist not installed so far)
KB: 5017095 [Windows10.0-KB5017095-x64.cab] - Updates to servicing stack - P and D
KB: 5016622 [Windows10.0-KB5016622-x64.cab] - Security Update for Printers, KDC, LSASS, -> J, P, D

Only F:
KB: 5016623 [Windows10.0-KB5016623-x64.cab] - Security Update for Local Security Authority Server Service -> Only F

F has a WS2019, other customers WS2016.



Anyone else experiencing this errors?

[HannesK]

Hello,
yes, I saw something yesterday. The partner did not want to open a case, so the post was deleted.

If I remember correctly, then the solution to the problem was restarting the SQL writer service on the SQL machine.

Best regards,
Hannes

[Sanai95]

Hello,

I restarted the correspondig VSS Service "SQL Server VSS Writer".

For most of those servers there is no SQL Service, since it's "only" a Domain controller without an additional SQL server installed.

We try deinstalling the said windows updates on one system.
On another we try to de- and reinstall the "SQL VSS Writer" (Recommandation from the veeam support).

Best regards,
Maik

[LickABrick]

I posted about this yesterday. For some customers stopping instead of restarting the writer service fixed the issue.

[SnakeSK]

Yep, currently have to stop the VSS SQL service as part of the prefreeze script.

Also guys - do you have Azure AD connect installed on those servers?

[LickABrick]

I've seen the issue on servers without AzureAD connect (I'm pretty sure), a colleague of mine mentioned the following fixes the issue on server with AAD Connect though: https://www.veeam.com/kb2326

[n.koch]

I have the same case. On some servers it was enough to restart the VSS Writer. For others it had to be a server reboot.

What I noticed was that all DomainControllers were running Windows 2019.

[mjr.epicfail]

Ive seen an increase as well. But only on domainservers with Azure AD connect.

[Sanai95]

Hallo,

Update to the last steps.
Both deinstalling the last updates and re-registering the VSSWriters work.

Link on how to re-registering the writers:
http://woshub.com/vss-writer-failed-re- ... ws-server/

Best regards
Maik

[YouGotServered]

All - I can confirm that we have seen this at several clients with Azure AD Connect, now and in the past. The light version of SQL that MS installs with AD Connect is garbage. We have tried various "permanent" solutions across the web with mixed results. If anyone finds one that holds up in several environments for more than a couple of months, I'd love to hear about it.

[Steve-nIP]

Yep, currently have to stop the VSS SQL service as part of the prefreeze script.

Also guys - do you have Azure AD connect installed on those servers?

That was my first thought. Azure AD connect, when it updates, causes this issue. Restarting the server fixes the issue. (If it doesn't, repair localDB and restart).
I've seen the same problem for many years, it's a classic.