Bunch of servers VSS Writer Errror 0x800423f4

[infused]

Getting this on a bunch of servers with the new Azure AD Sync tool installed. Searched around, but not getting a lot.

Veeam Error:

Code: Select all

Unable to release guest. Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

On the server, the VSS error in event log is error 0x800423f4 when PrepareForSnapShot

There's no other SQL services on the server, nor is there any other backup software.

Any ideas?

[Gostev]

Hi, please include support case ID for this issue, as requested when you click New Topic. Thanks!

[infused]

Haven't made one yet, but happy to go and make one? I'm pretty sure this isn't a Veeam issue, but something to do with how the AD Azure Connect client upgrades on the new version. I can't find much information on this as the new AD Connect is quite new.

[infused]

Fixed it. Wrote up a little guide on how to fix it: http://www.tecfused.com/2016/09/veeam-b ... d-connect/

[hyvokar]

Hi!

We also have this problem. Azure ad connect auto-updates itself and I believe that the permissions on the db are reset after that resulting the error.
Would like to hear if anyone has overcome this.

[jrp76]

Anyone else have any luck with this? We are experiencing it as well here a half year later. Does the repair for SQL always prompt for the media? I assume this is installed with the Azure AD Connect package?

[Daniel N.]

It didn't ask me for the media. Hope this works.

[Daniel N.]

Yay, it worked. I hope this doesn't happen again. Else it would be great if you guys could talk to Microsoft and work out a definitive solution.

[SdC]

It was happening again on our customers servers after recent Windows "quality" rollups that I previously fixed manually with the procedure in the blog mentioned above. So I wrote a crude little powershell script to automate the process. I've ran it on about 25 servers in different domains successfully, and a reboot is not required.
I hope this helps.

Code: Select all

stop-service -displayname "Microsoft Azure AD Sync"
$FQlogonaccount = Get-WmiObject -Class Win32_Service | ? { $_.displayname -match "Microsoft Azure AD Sync"} | select Startname 
$split = $FQlogonaccount.startname.Split("\"[0])
$username = $split[1]
$sqlprocess = Get-WmiObject -Query "Select * from Win32_Process where name = 'sqlservr.exe'" |
Select Name, Handle, @{Label='Owner';Expression={$_.GetOwner().User}} | ? { $_.owner -match $username} | select handle | Out-String 
$sqlpid= $sqlprocess.Split("`n")[3]
Stop-Process -id $sqlpid -force
start-process -filepath "MsiExec.exe" -argumentlist "/f {6C026A91-640F-4A23-8B68-05D589CC6F18}" -wait
Start-Service -displayname "Microsoft Azure AD Sync"

[omf]

Thanks for these notes - we've been having the exact same issue.

[damadhatter]

I am having an odd issues that seems to be spreading like the plague. It has not hit 3 of my clients and one of them twice. This is causing jobs to fail on Windows 2012R2 & Windows 2016 VMware VMs. Guest indexing and application processing is enabled. The common theme on these servers is the following: Domain Controller & Azure AD Sync (for Office 365 password sync).

The jobs fail with this error:

Code: Select all

7/13/2017 11:02:30 PM :: Error: Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{d9561457-108c-410a-bda9-034d41a92cfb}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].] 

I opened a support case on this in the past, and support wanted me to run SQL Management Studio to gather SQL logs. I installed SQL Management Studio but no SQL instances were found. They had me list the VSS writers and found this:

Code: Select all

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {abebd680-a45c-4c72-9ac9-26ee988057c9}
   State: [8] Failed
   Last error: Non-retryable error

Support had me stop the SQLWriter service and the job was successful. Support then had me delete the service as they seemed to think it wasn't needed. This resolved the issue at this customer for a short period of time, it then came back to that customer and hit a few others.

I am quite perplexed at what is going on, things worked great for some time and all of a sudden fall apart. I have 10 clients running Azure AD Sync so unsure why its only hitting 3 clients. Anyone seen anything like this? I know by default a SQL Server 2012 Express LocalDB (a light version of SQL Server Express) is installed with Azure AD Sync so that is likely the culprit, but why randomly does Veeam hang up on this DB?

[veremin]

I have 10 clients running Azure AD Sync so unsure why its only hitting 3 clients.

Are you running application aware backups for all of them or just for three mentioned ones? Seems like a third-party VSS provider is a root cause of the experienced behaviour, similar to the problem discussed here.Thanks.

[damadhatter]

I found the issue, looks like Azure AD updates itself and messes things up, go Microsoft!

[veremin]

Any particular update to blame? Might be helpful for future readers.

[damadhatter]

I actually found it on this forum, I should have searched more before I actually made my post, shame on me!

veeam-backup-replication-f2/bunch-of-se ... 37483.html

[veremin]

Thanks for providing corresponding links - I've just merged two topics. Much more convenient to have everything within one discussion.

[infused]

Since this threads still going, we've had this over and over again. Repairing the SQL server seems to be the go too. It seems the update doesn't complete correctly in a lot of cases. We've had so many AD Sync issues it's not funny.

[hyvokar]

What is funny, is that problem was first addressed almost a year ago and it's still not fixed. Nothing new in MS boat.

[Bopp]

Hi, have the same Issue on serveral customers sites (17+, Count is raising each week...).
Repaired the "Microsoft SQL Server 2012 Express LocalDB" on two sites for testing the given fix.

But after roughly two weeks I'm back on square one... Error came back on both sites.

Repair/Restart is no solution...
Has someone already found a permanent fix for this?

[ICIQ]

Dear all,

Our support case ID is Case # 02286447

We have the same problem. Repair SQL Server 2012 Express LocalDB is not a fix solution.

We need a fix solution.

Kind Regards,

[foggy]

Have you also involved MS? Since the issue seems to be on their side.

[damadhatter]

That fix only seems to be temporary, this still happens every other week or so. I run the repair and things work fine the next time around. Getting pretty annoying!!! Is anyone else seeing this?

[roawan2]

Yes we are still seeing this across a number of customer sites. Very annoying. Repairing the SQL instance only seems to last a short time before the same error reappears.

[RobertMallett]

Yes I'm seeing this too. The result of the SQL fix seems random, I've a task to reboot the server and then another task 5 mins after to run the SQL fix, one day it works good the next day not. I've had 7 days success having changed nothing but would not be surprised if I see a failure tomorrow.
I appreciate this is an MS issue but you would think someone at Veeam would have more power to lean on MS for a fix than us poor little users, and I do think it's a bit poor that they have just shrugged their shoulders and said ' nothing to do with us - your problem'.

[foggy]

Hi Robert, do you have a case open with Veeam technical support for this issue?

[RobertMallett]

No because I read the posts where someone else did and was told - go talk to Microsoft since the issue seems to be on their side.. Shall I?

[foggy]

You can involve both parties into investigation, our engineers will be able to assist you in discussions with MS regarding the issue.

[bertus02]

My 2 cents. We are getting these VSS errors across multiple Azure Connect servers after the client auto-upgrades. Simple fix that corrects the VSS errors is to reboot the servers. The VSS errors will then dissapera only to reappear after the Azure AD connect client auto-upgrades. You can however disable the Auto-update via Powershell.

https://docs.microsoft.com/en-us/azure/ ... ic-upgrade

[hyvokar]

My 2 cents. We are getting these VSS errors across multiple Azure Connect servers after the client auto-upgrades. Simple fix that corrects the VSS errors is to reboot the servers. The VSS errors will then dissapera only to reappear after the Azure AD connect client auto-upgrades. You can however disable the Auto-update via Powershell.

https://docs.microsoft.com/en-us/azure/ ... ic-upgrade

Just tested with couple of servers that had autoupdated their ad sync on 15.9 and reboot worked.
Now, since I don't want to disable auto update (for now), I guess the only viable workaround is to schedule a reboot for ad sync servers daily before backup.

[RobertMallett]

This is what I did and it works - apart from occasionally it doesn't for no reason! It was good for 7 days then failed and so far good for 15 days!