Bunch of servers VSS Writer Errror 0x800423f4

[bertus02]

Perhaps check your event logs to see if AD Connect upgraded around the time the VSS errors started.

"With the connectivity to Azure AD verified, it is time to look into the eventlogs. Start the event viewer and look in the Application eventlog. Add an eventlog filter for the source Azure AD Connect Upgrade and the event id range 300-399."

[RobertMallett]

I can see Azure AD Connect Upgrade - Persisted state loaded successfully. Of course I'm not seeing any errors at present as everything is a sit should be - as soon as I get a fail I'll look up the logs around that time.

[damadhatter]

This continues to happen for me every 3-4 days, quite annoying. I found this script while searching for other people having this issue,

stop-service -displayname “Microsoft Azure AD Sync”
$FQlogonaccount = Get-WmiObject -Class Win32_Service | ? { $_.displayname -match “Microsoft Azure AD Sync”} | select Startname
$split = $FQlogonaccount.startname.Split(“\”[0])
$username = $split[1]
$sqlprocess = Get-WmiObject -Query “Select * from Win32_Process where name = ‘sqlservr.exe'” |
Select Name, Handle, @{Label=’Owner’;Expression={$_.GetOwner().User}} | ? { $_.owner -match $username} | select handle | Out-String
$sqlpid= $sqlprocess.Split(“`n”)[3]
Stop-Process -id $sqlpid -force
start-process -filepath “MsiExec.exe” -argumentlist “/f {6C026A91-640F-4A23-8B68-05D589CC6F18}” -wait
Start-Service -displayname “Microsoft Azure AD Sync”

Script found in the comments section of this posting. http://www.tecfused.com/2016/09/veeam-b ... d-connect/

Still would be nice to figure out a fix to prevent the initial failure, wish Veeam could work with Microsoft on the issue.

[hyvokar]

After first reported over a year ago, this still is a problem (and very annoying one). Is there anything Veeam can do to help it's customers?

[mamosorre84]

Hi guys, any news? The solution "repair LocalDB and reboot" didn't fix for me, I had to restart VSS Writers failed manually..

[rasmusan]

Hi

Anyone found solution to this?

I have several customers started reporting this error in during the last week - all related to VMs with Azure AD Connect. It seems it has updated itself on 19/10 after which the issues has started. Repair and reboot does not fix issue


/Rasmus

[hyvokar]

Faced this problem again last week. Finally got fed up randomly rebooting servers dunring day time and scheduled a daily reboot every night just before backup. This workaround sucks just a little bit less than manually rebooting every time backup fails.

[bertus02]

Is Veeam aware of the issues we are facing with Azure AD Connect? This is becoming tedious to manually reboot 5+ ad connect servers. I have been contemplating adding the script mentioned in this task to the pre-freeze script in the job. Has anyone tested this workaround?

[bertus02]

I added the script mentioned on page 1 to the pre-freeze script and it resolved the issue on one of our Azure AD connect servers. We went from the following to success. I will be rolling the fix out to our other servers as it pops up.

"Unable to release guest. Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{c0c13030-f827-4f40-b0d8-b8f5d4f32edd}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{c0c13030-f827-4f40-b0d8-b8f5d4f32edd}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

[Deon]

Good afternoon,

I see a bunch of questions directed to nowhere/without a response, all asking if Veeam is aware of it.
I participate in support activity so I do know that this issue is known, however it does not seem to have a strict solution, and even Microsoft DPM has an issue with it from time to time, because it lies in problems with VSS framework.

For anyone having such issue, I would advise to open a multivendor support case in Veeam to be connected with Microsoft support for a resolution, at least it would be the most reliable way to go in my opinion.
Of course you could try to reach Microsoft by yourself, but sometimes it may be more handy to have an engineer assisting you from our end as well, just to be sure we look at the issue from different perspectives.

[SdC]

This continues to happen for me every 3-4 days, quite annoying. I found this script while searching for other people having this issue,

stop-service -....
soft Azure AD Sync”

Script found in the comments section of this posting. http://www.tecfused.com/2016/09/veeam-b ... d-connect/

Still would be nice to figure out a fix to prevent the initial failure, wish Veeam could work with Microsoft on the issue.

That is my script and it is also on THIS VERY THREAD ON PAGE 1, right above some other posts you made

[mcz]

I'm having the same issue and I wonder if you could exclude this VSS writer from backup?

[bdufour]

ive been experiencing this issue - i found this - https://social.technet.microsoft.com/Fo ... errecovery

typically, i have to restart the server for it to complete a backup/rep. it usually requires a restart every few day as related to Azure AD Sync updating likely and causing issues with permissions. i changed the log on account of the SQL server VSS writer service to a domain service account with proper permissions to the DB. for now that seemed to solve it. i will give an update in a week or so to confirm whether or not the issue has resurfaced.

[reaperhammer]

I use the script on this page to fix the issue temporarily
http://www.tecfused.com/2017/10/veeam-b ... -31102017/

As a permanent fix you can update the SQL VSS writer by using any full installer for SQL and selecting “Upgrade from SQL 2005, 2008…” then follow the wizard and reboot, or select “New SQL Server Stand-alone installation…”and selecting only This can fix the ADSync backup as well as break the backup for other SQL instances if there are any on the same server so better take a test backup right after doing this and rollback if any other strange things happen.Another option is to move ADSync database to a SQL Express or Standard instance on the same server or another server: https://social.technet.microsoft.com/wi ... ation.aspx

Or the less recommended option of disabling Azure AD connect updates
https://docs.microsoft.com/en-us/azure/ ... ic-upgrade

[mcz]

Yeah, I tried to upgrade SQL Server to the Version 2016 and since then the issue seems to be gone.

[Datatekniker]

Hi,

To "MCZ", what version of SQLserver were you running before the upgrade ?

and did you get any new issues after the upgrade ?

[mcz]

Previous version was 2012 Express Edition (so the basic setup which comes with installation of AD Connect) and now I see that I only installed SP1 for SQLServer 2016. In my case it worked and I didn't receive any other issues since then. Probably it is enough to repair the existing installation??

[mcz]

...ok I have to revert what I wrote some hours ago because the issue is still there...

[scott_thomson]

Hi all,

Not a Veeam user, but since this thread was what initially clued me in to the issue with the product we use for Hyper-V host-level backups of guests, I thought I'd chime in.

This definitely looks like a MS issue, but I haven't found any real acknowledgement of the problem anywhere from them. In hopes that we can raise the profile of the issue, I've posted to the AD Connect UserVoice as well as the MS Partner Support forums. Hopefully if you can add your voices to this, someone at MS will get involved:

https://feedback.azure.com/forums/16940 ... it-doesn-t
https://partnersupport.microsoft.com/en ... 9729458808

[GregorS]

Having the same issue, just subscribing to this thread, thanks.

[mats.jansson]

The only solution that workt for us was to uninstall azure ad connect, install sql express (we used 2016 sp1) and reinstalled azure ad connect with option to use the manual installed express and a service account for sql (it's also working to use a sql on another server).

[Datatekniker]

This works,

If you can accept that the AADconnect, is dead in the wather for a short period of time, a simple solution is this.

Use task scheduler:

1. Stop the AADconnect service shortly before the Veeambackup is set to run. (Powershell Stop-Service -DisplayName 'Microsoft Azure AD Sync')

---- The backup is running

2. Start the AADconnect again after a the backup has finished. (mine takes about 15min, so I start the service after 30min.) (Powershell Start-Service -DisplayName 'Microsoft Azure AD Sync')

Thats it.
My backup has been running 3 times this weekend without any further problems

[mcz]

2. Start the AADconnect again after a the backup has finished. (mine takes about 15min, so I start the service after 30min.) (Powershell Start-Service -DisplayName 'Microsoft Azure AD Sync')

You should be able to start the service just after the snapshot has been taken (start it in the post-thaw script) because veeam releases vss-snapshot after vmware snapshot (if you are not indexing the filesystem).

[bdufour]

i haven't had to restart this server since i wrote my original post about changing the login permissions to the service..

Re: Bunch of servers VSS Writer Errror 0x800423f4
Veeam Logo by bdufour » Wed Nov 01, 2017 9:01 pm

ive been experiencing this issue - i found this - https://social.technet.microsoft.com/Fo ... errecovery

typically, i have to restart the server for it to complete a backup/rep. it usually requires a restart every few day as related to Azure AD Sync updating likely and causing issues with permissions. i changed the log on account of the SQL server VSS writer service to a domain service account with proper permissions to the DB. for now that seemed to solve it. i will give an update in a week or so to confirm whether or not the issue has resurfaced.

[jeradries]

This definitely looks like a MS issue, but I haven't found any real acknowledgement of the problem anywhere from them. In hopes that we can raise the profile of the issue, I've posted to the AD Connect UserVoice as well as the MS Partner Support forums. Hopefully if you can add your voices to this, someone at MS will get involved:

https://feedback.azure.com/forums/16940 ... it-doesn-t
https://partnersupport.microsoft.com/en ... 9729458808

I have been following your post on the MS Partner Support forum and when looking at the SyncEngine-AutoUpgrader-XXXXXXXX-XXXXXX.log files in C:\ProgramData\AADConnect, I am seeing the auto-upgrade process being triggered every 6 hours as well starting on 10/23. However, what caught my eye is "Upgrading AADConnect to version: 1.1.648.0".

According to the version release history at https://docs.microsoft.com/en-us/azure/ ... on-history, this isn't a valid version number. Are you seeing the same thing?

[mcz]

Yep, I see the same version number in the logfile.

[dnorth]

i changed the log on account of the SQL server VSS writer service to a domain service account with proper permissions to the DB. for now that seemed to solve it.

Changing the service account seems to be working for me. The first Veeam job after an Auto (non-)Upgrade completed without an error anyway. Thanks, Blake.

[bdufour]

Changing the service account seems to be working for me. The first Veeam job after an Auto (non-)Upgrade completed without an error anyway. Thanks, Blake.[/quote]

no problem, i still haven't had to restart this server and all jobs for it are still completing without any errors. at this point, i don't think it's just a coincidence and this may have fixed the issue on my end.

[mcz]

In my case, repairing the express-db-instance has resolved the issue...

[Marius_Aad]

Upgrading MSSQL Server 2012 Express LocalDB to SP4 fixed the issue for me. Multiple back up jobs finished successfully after that without restarting the vm.