Comprehensive data protection for all workloads
infused
Service Provider
Posts: 133
Liked: 6 times
Joined: Apr 20, 2013 9:25 am
Full Name: Hayden Kirk
Contact:

Bunch of servers VSS Writer Errror 0x800423f4

Post by infused »

Getting this on a bunch of servers with the new Azure AD Sync tool installed. Searched around, but not getting a lot.

Veeam Error:

Code: Select all

Unable to release guest. Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
Error: Unfreeze error (over VIX): [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{6323fe10-0205-47df-b015-4a5ff60c31e2}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
On the server, the VSS error in event log is error 0x800423f4 when PrepareForSnapShot

There's no other SQL services on the server, nor is there any other backup software.

Any ideas?
https://www.tecfused.com - my blog

Gostev
SVP, Product Management
Posts: 26706
Liked: 4277 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by Gostev »

Hi, please include support case ID for this issue, as requested when you click New Topic. Thanks!

infused
Service Provider
Posts: 133
Liked: 6 times
Joined: Apr 20, 2013 9:25 am
Full Name: Hayden Kirk
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by infused »

Haven't made one yet, but happy to go and make one? I'm pretty sure this isn't a Veeam issue, but something to do with how the AD Azure Connect client upgrades on the new version. I can't find much information on this as the new AD Connect is quite new.
https://www.tecfused.com - my blog

infused
Service Provider
Posts: 133
Liked: 6 times
Joined: Apr 20, 2013 9:25 am
Full Name: Hayden Kirk
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by infused » 1 person likes this post

Fixed it. Wrote up a little guide on how to fix it: http://www.tecfused.com/2016/09/veeam-b ... d-connect/
https://www.tecfused.com - my blog

hyvokar
Expert
Posts: 371
Liked: 24 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by hyvokar »

Hi!

We also have this problem. Azure ad connect auto-updates itself and I believe that the permissions on the db are reset after that resulting the error.
Would like to hear if anyone has overcome this.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

jrp76
Novice
Posts: 8
Liked: never
Joined: Mar 15, 2017 9:33 pm
Full Name: Jon Pletcher

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by jrp76 »

Anyone else have any luck with this? We are experiencing it as well here a half year later. Does the repair for SQL always prompt for the media? I assume this is installed with the Azure AD Connect package?

Daniel N.
Enthusiast
Posts: 41
Liked: 3 times
Joined: Feb 15, 2015 7:03 am
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by Daniel N. »

It didn't ask me for the media. Hope this works.

Daniel N.
Enthusiast
Posts: 41
Liked: 3 times
Joined: Feb 15, 2015 7:03 am
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by Daniel N. »

Yay, it worked. I hope this doesn't happen again. Else it would be great if you guys could talk to Microsoft and work out a definitive solution.

SdC
Novice
Posts: 3
Liked: 2 times
Joined: May 19, 2017 7:03 am
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by SdC » 2 people like this post

It was happening again on our customers servers after recent Windows "quality" rollups that I previously fixed manually with the procedure in the blog mentioned above. So I wrote a crude little powershell script to automate the process. I've ran it on about 25 servers in different domains successfully, and a reboot is not required.
I hope this helps.

Code: Select all

stop-service -displayname "Microsoft Azure AD Sync"
$FQlogonaccount = Get-WmiObject -Class Win32_Service | ? { $_.displayname -match "Microsoft Azure AD Sync"} | select Startname 
$split = $FQlogonaccount.startname.Split("\"[0])
$username = $split[1]
$sqlprocess = Get-WmiObject -Query "Select * from Win32_Process where name = 'sqlservr.exe'" |
Select Name, Handle, @{Label='Owner';Expression={$_.GetOwner().User}} | ? { $_.owner -match $username} | select handle | Out-String 
$sqlpid= $sqlprocess.Split("`n")[3]
Stop-Process -id $sqlpid -force
start-process -filepath "MsiExec.exe" -argumentlist "/f {6C026A91-640F-4A23-8B68-05D589CC6F18}" -wait
Start-Service -displayname "Microsoft Azure AD Sync"

omf
Novice
Posts: 4
Liked: 1 time
Joined: Nov 16, 2009 10:05 pm
Full Name: Oscar Fowler
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by omf »

Thanks for these notes - we've been having the exact same issue.

damadhatter
Service Provider
Posts: 36
Liked: 2 times
Joined: Feb 21, 2014 5:15 am
Full Name: Chris A
Contact:

[MERGED] Jobs failing on Azure AD Sync SQL express database

Post by damadhatter »

I am having an odd issues that seems to be spreading like the plague. It has not hit 3 of my clients and one of them twice. This is causing jobs to fail on Windows 2012R2 & Windows 2016 VMware VMs. Guest indexing and application processing is enabled. The common theme on these servers is the following: Domain Controller & Azure AD Sync (for Office 365 password sync).

The jobs fail with this error:

Code: Select all

7/13/2017 11:02:30 PM :: Error: Unfreeze error: [Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{d9561457-108c-410a-bda9-034d41a92cfb}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].] 
I opened a support case on this in the past, and support wanted me to run SQL Management Studio to gather SQL logs. I installed SQL Management Studio but no SQL instances were found. They had me list the VSS writers and found this:

Code: Select all

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {abebd680-a45c-4c72-9ac9-26ee988057c9}
   State: [8] Failed
   Last error: Non-retryable error
Support had me stop the SQLWriter service and the job was successful. Support then had me delete the service as they seemed to think it wasn't needed. This resolved the issue at this customer for a short period of time, it then came back to that customer and hit a few others.

I am quite perplexed at what is going on, things worked great for some time and all of a sudden fall apart. I have 10 clients running Azure AD Sync so unsure why its only hitting 3 clients. Anyone seen anything like this? I know by default a SQL Server 2012 Express LocalDB (a light version of SQL Server Express) is installed with Azure AD Sync so that is likely the culprit, but why randomly does Veeam hang up on this DB?

veremin
Product Manager
Posts: 17834
Liked: 1663 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Jobs failing on Azure AD Sync SQL express database

Post by veremin »

I have 10 clients running Azure AD Sync so unsure why its only hitting 3 clients.
Are you running application aware backups for all of them or just for three mentioned ones? Seems like a third-party VSS provider is a root cause of the experienced behaviour, similar to the problem discussed here.Thanks.

damadhatter
Service Provider
Posts: 36
Liked: 2 times
Joined: Feb 21, 2014 5:15 am
Full Name: Chris A
Contact:

Re: Jobs failing on Azure AD Sync SQL express database

Post by damadhatter »

I found the issue, looks like Azure AD updates itself and messes things up, go Microsoft!

veremin
Product Manager
Posts: 17834
Liked: 1663 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Jobs failing on Azure AD Sync SQL express database

Post by veremin »

Any particular update to blame? Might be helpful for future readers.

damadhatter
Service Provider
Posts: 36
Liked: 2 times
Joined: Feb 21, 2014 5:15 am
Full Name: Chris A
Contact:

Re: Jobs failing on Azure AD Sync SQL express database

Post by damadhatter »

I actually found it on this forum, I should have searched more before I actually made my post, shame on me!

veeam-backup-replication-f2/bunch-of-se ... 37483.html

veremin
Product Manager
Posts: 17834
Liked: 1663 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by veremin »

Thanks for providing corresponding links - I've just merged two topics. Much more convenient to have everything within one discussion.

infused
Service Provider
Posts: 133
Liked: 6 times
Joined: Apr 20, 2013 9:25 am
Full Name: Hayden Kirk
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by infused »

Since this threads still going, we've had this over and over again. Repairing the SQL server seems to be the go too. It seems the update doesn't complete correctly in a lot of cases. We've had so many AD Sync issues it's not funny.
https://www.tecfused.com - my blog

hyvokar
Expert
Posts: 371
Liked: 24 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by hyvokar »

What is funny, is that problem was first addressed almost a year ago and it's still not fixed. Nothing new in MS boat.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

Bopp
Influencer
Posts: 13
Liked: 2 times
Joined: Nov 10, 2015 7:28 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by Bopp »

Hi, have the same Issue on serveral customers sites (17+, Count is raising each week...).
Repaired the "Microsoft SQL Server 2012 Express LocalDB" on two sites for testing the given fix.

But after roughly two weeks I'm back on square one... Error came back on both sites.

Repair/Restart is no solution...
Has someone already found a permanent fix for this?

ICIQ
Lurker
Posts: 1
Liked: never
Joined: Aug 22, 2017 9:43 am
Full Name: ICIQ
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by ICIQ »

Dear all,

Our support case ID is Case # 02286447

We have the same problem. Repair SQL Server 2012 Express LocalDB is not a fix solution.

We need a fix solution.

Kind Regards,

foggy
Veeam Software
Posts: 19465
Liked: 1767 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by foggy »

Have you also involved MS? Since the issue seems to be on their side.

damadhatter
Service Provider
Posts: 36
Liked: 2 times
Joined: Feb 21, 2014 5:15 am
Full Name: Chris A
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by damadhatter »

That fix only seems to be temporary, this still happens every other week or so. I run the repair and things work fine the next time around. Getting pretty annoying!!! Is anyone else seeing this?

roawan2
Service Provider
Posts: 2
Liked: 1 time
Joined: Mar 13, 2009 6:55 pm
Full Name: Ronan
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by roawan2 » 1 person likes this post

Yes we are still seeing this across a number of customer sites. Very annoying. Repairing the SQL instance only seems to last a short time before the same error reappears.

RobertMallett
Novice
Posts: 4
Liked: never
Joined: Aug 15, 2017 12:20 pm
Full Name: Robert Mallett
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by RobertMallett »

Yes I'm seeing this too. The result of the SQL fix seems random, I've a task to reboot the server and then another task 5 mins after to run the SQL fix, one day it works good the next day not. I've had 7 days success having changed nothing but would not be surprised if I see a failure tomorrow.
I appreciate this is an MS issue but you would think someone at Veeam would have more power to lean on MS for a fix than us poor little users, and I do think it's a bit poor that they have just shrugged their shoulders and said ' nothing to do with us - your problem'.

foggy
Veeam Software
Posts: 19465
Liked: 1767 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by foggy »

Hi Robert, do you have a case open with Veeam technical support for this issue?

RobertMallett
Novice
Posts: 4
Liked: never
Joined: Aug 15, 2017 12:20 pm
Full Name: Robert Mallett
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by RobertMallett »

No because I read the posts where someone else did and was told - go talk to Microsoft since the issue seems to be on their side.. Shall I?

foggy
Veeam Software
Posts: 19465
Liked: 1767 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by foggy »

You can involve both parties into investigation, our engineers will be able to assist you in discussions with MS regarding the issue.

bertus02
Novice
Posts: 5
Liked: never
Joined: Sep 18, 2017 1:23 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by bertus02 »

My 2 cents. We are getting these VSS errors across multiple Azure Connect servers after the client auto-upgrades. Simple fix that corrects the VSS errors is to reboot the servers. The VSS errors will then dissapera only to reappear after the Azure AD connect client auto-upgrades. You can however disable the Auto-update via Powershell.

https://docs.microsoft.com/en-us/azure/ ... ic-upgrade

hyvokar
Expert
Posts: 371
Liked: 24 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by hyvokar »

bertus02 wrote:My 2 cents. We are getting these VSS errors across multiple Azure Connect servers after the client auto-upgrades. Simple fix that corrects the VSS errors is to reboot the servers. The VSS errors will then dissapera only to reappear after the Azure AD connect client auto-upgrades. You can however disable the Auto-update via Powershell.

https://docs.microsoft.com/en-us/azure/ ... ic-upgrade
Just tested with couple of servers that had autoupdated their ad sync on 15.9 and reboot worked.
Now, since I don't want to disable auto update (for now), I guess the only viable workaround is to schedule a reboot for ad sync servers daily before backup.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

RobertMallett
Novice
Posts: 4
Liked: never
Joined: Aug 15, 2017 12:20 pm
Full Name: Robert Mallett
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by RobertMallett »

This is what I did and it works - apart from occasionally it doesn't for no reason! It was good for 7 days then failed and so far good for 15 days!

Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Google [Bot] and 34 guests