Comprehensive data protection for all workloads
bertus02
Novice
Posts: 5
Liked: never
Joined: Sep 18, 2017 1:23 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by bertus02 » Sep 19, 2017 1:11 pm

Perhaps check your event logs to see if AD Connect upgraded around the time the VSS errors started.

"With the connectivity to Azure AD verified, it is time to look into the eventlogs. Start the event viewer and look in the Application eventlog. Add an eventlog filter for the source Azure AD Connect Upgrade and the event id range 300-399."

RobertMallett
Novice
Posts: 4
Liked: never
Joined: Aug 15, 2017 12:20 pm
Full Name: Robert Mallett
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by RobertMallett » Sep 19, 2017 2:20 pm

I can see Azure AD Connect Upgrade - Persisted state loaded successfully. Of course I'm not seeing any errors at present as everything is a sit should be - as soon as I get a fail I'll look up the logs around that time.

damadhatter
Service Provider
Posts: 30
Liked: 2 times
Joined: Feb 21, 2014 5:15 am
Full Name: Chris A
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by damadhatter » Sep 21, 2017 6:19 pm

This continues to happen for me every 3-4 days, quite annoying. I found this script while searching for other people having this issue,

stop-service -displayname “Microsoft Azure AD Sync”
$FQlogonaccount = Get-WmiObject -Class Win32_Service | ? { $_.displayname -match “Microsoft Azure AD Sync”} | select Startname
$split = $FQlogonaccount.startname.Split(“\”[0])
$username = $split[1]
$sqlprocess = Get-WmiObject -Query “Select * from Win32_Process where name = ‘sqlservr.exe'” |
Select Name, Handle, @{Label=’Owner’;Expression={$_.GetOwner().User}} | ? { $_.owner -match $username} | select handle | Out-String
$sqlpid= $sqlprocess.Split(“`n”)[3]
Stop-Process -id $sqlpid -force
start-process -filepath “MsiExec.exe” -argumentlist “/f {6C026A91-640F-4A23-8B68-05D589CC6F18}” -wait
Start-Service -displayname “Microsoft Azure AD Sync”

Script found in the comments section of this posting. http://www.tecfused.com/2016/09/veeam-b ... d-connect/

Still would be nice to figure out a fix to prevent the initial failure, wish Veeam could work with Microsoft on the issue.

hyvokar
Expert
Posts: 360
Liked: 23 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by hyvokar » Oct 18, 2017 5:23 am

After first reported over a year ago, this still is a problem (and very annoying one). Is there anything Veeam can do to help it's customers?
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

mamosorre84
Service Provider
Posts: 82
Liked: 6 times
Joined: Oct 24, 2016 3:56 pm
Full Name: Marco Sorrentino
Location: Ancona - Italy
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by mamosorre84 » Oct 26, 2017 7:30 am

Hi guys, any news? The solution "repair LocalDB and reboot" didn't fix for me, I had to restart VSS Writers failed manually..

rasmusan
Enthusiast
Posts: 45
Liked: never
Joined: Jan 20, 2015 9:03 pm
Full Name: Rasmus Andersen
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by rasmusan » Oct 27, 2017 5:56 am

Hi

Anyone found solution to this?

I have several customers started reporting this error in during the last week - all related to VMs with Azure AD Connect. It seems it has updated itself on 19/10 after which the issues has started. Repair and reboot does not fix issue


/Rasmus

hyvokar
Expert
Posts: 360
Liked: 23 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by hyvokar » Oct 27, 2017 6:50 am

Faced this problem again last week. Finally got fed up randomly rebooting servers dunring day time and scheduled a daily reboot every night just before backup. This workaround sucks just a little bit less than manually rebooting every time backup fails.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

bertus02
Novice
Posts: 5
Liked: never
Joined: Sep 18, 2017 1:23 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by bertus02 » Oct 27, 2017 4:44 pm

Is Veeam aware of the issues we are facing with Azure AD Connect? This is becoming tedious to manually reboot 5+ ad connect servers. I have been contemplating adding the script mentioned in this task to the pre-freeze script in the job. Has anyone tested this workaround?
Image

bertus02
Novice
Posts: 5
Liked: never
Joined: Sep 18, 2017 1:23 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by bertus02 » Oct 27, 2017 11:01 pm

I added the script mentioned on page 1 to the pre-freeze script and it resolved the issue on one of our Azure AD connect servers. We went from the following to success. I will be rolling the fix out to our other servers as it pops up.

"Unable to release guest. Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{c0c13030-f827-4f40-b0d8-b8f5d4f32edd}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]
Error: Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [SqlServerWriter]. Class ID: [{a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}]. Instance ID: [{c0c13030-f827-4f40-b0d8-b8f5d4f32edd}]. Writer's state: [VSS_WS_FAILED_AT_PREPARE_SNAPSHOT]. Error code: [0x800423f4].]

Deon
Veeam Software
Posts: 24
Liked: 8 times
Joined: Jun 26, 2014 7:02 pm
Full Name: Denis Churaev
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by Deon » Oct 28, 2017 2:51 pm

Good afternoon,

I see a bunch of questions directed to nowhere/without a response, all asking if Veeam is aware of it.
I participate in support activity so I do know that this issue is known, however it does not seem to have a strict solution, and even Microsoft DPM has an issue with it from time to time, because it lies in problems with VSS framework.

For anyone having such issue, I would advise to open a multivendor support case in Veeam to be connected with Microsoft support for a resolution, at least it would be the most reliable way to go in my opinion.
Of course you could try to reach Microsoft by yourself, but sometimes it may be more handy to have an engineer assisting you from our end as well, just to be sure we look at the issue from different perspectives.

SdC
Novice
Posts: 3
Liked: 2 times
Joined: May 19, 2017 7:03 am
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by SdC » Oct 30, 2017 6:48 am

damadhatter wrote:This continues to happen for me every 3-4 days, quite annoying. I found this script while searching for other people having this issue,

stop-service -....
soft Azure AD Sync”

Script found in the comments section of this posting. http://www.tecfused.com/2016/09/veeam-b ... d-connect/

Still would be nice to figure out a fix to prevent the initial failure, wish Veeam could work with Microsoft on the issue.
That is my script and it is also on THIS VERY THREAD ON PAGE 1, right above some other posts you made :wink:

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by mcz » Oct 30, 2017 3:44 pm

I'm having the same issue and I wonder if you could exclude this VSS writer from backup?

bdufour
Expert
Posts: 198
Liked: 29 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by bdufour » Nov 01, 2017 9:01 pm

ive been experiencing this issue - i found this - https://social.technet.microsoft.com/Fo ... errecovery

typically, i have to restart the server for it to complete a backup/rep. it usually requires a restart every few day as related to Azure AD Sync updating likely and causing issues with permissions. i changed the log on account of the SQL server VSS writer service to a domain service account with proper permissions to the DB. for now that seemed to solve it. i will give an update in a week or so to confirm whether or not the issue has resurfaced.

reaperhammer
Service Provider
Posts: 27
Liked: 7 times
Joined: Aug 18, 2016 7:59 pm
Full Name: Will S
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by reaperhammer » Nov 02, 2017 12:57 am 1 person likes this post

I use the script on this page to fix the issue temporarily
http://www.tecfused.com/2017/10/veeam-b ... -31102017/

As a permanent fix you can update the SQL VSS writer by using any full installer for SQL and selecting “Upgrade from SQL 2005, 2008…” then follow the wizard and reboot, or select “New SQL Server Stand-alone installation…”and selecting only This can fix the ADSync backup as well as break the backup for other SQL instances if there are any on the same server so better take a test backup right after doing this and rollback if any other strange things happen.Another option is to move ADSync database to a SQL Express or Standard instance on the same server or another server: https://social.technet.microsoft.com/wi ... ation.aspx

Or the less recommended option of disabling Azure AD connect updates
https://docs.microsoft.com/en-us/azure/ ... ic-upgrade

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by mcz » Nov 02, 2017 7:40 am

Yeah, I tried to upgrade SQL Server to the Version 2016 and since then the issue seems to be gone.

Datatekniker
Lurker
Posts: 2
Liked: never
Joined: Nov 02, 2017 7:59 am
Full Name: Flemming
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by Datatekniker » Nov 02, 2017 8:12 am

Hi,

To "MCZ", what version of SQLserver were you running before the upgrade ?

and did you get any new issues after the upgrade ?

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by mcz » Nov 02, 2017 10:33 am

Previous version was 2012 Express Edition (so the basic setup which comes with installation of AD Connect) and now I see that I only installed SP1 for SQLServer 2016. In my case it worked and I didn't receive any other issues since then. Probably it is enough to repair the existing installation??

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by mcz » Nov 02, 2017 3:46 pm

...ok I have to revert what I wrote some hours ago because the issue is still there...

scott_thomson
Lurker
Posts: 1
Liked: never
Joined: Nov 03, 2017 5:08 pm
Full Name: Scott Thomson
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by scott_thomson » Nov 03, 2017 5:18 pm

Hi all,

Not a Veeam user, but since this thread was what initially clued me in to the issue with the product we use for Hyper-V host-level backups of guests, I thought I'd chime in.

This definitely looks like a MS issue, but I haven't found any real acknowledgement of the problem anywhere from them. In hopes that we can raise the profile of the issue, I've posted to the AD Connect UserVoice as well as the MS Partner Support forums. Hopefully if you can add your voices to this, someone at MS will get involved:

https://feedback.azure.com/forums/16940 ... it-doesn-t
https://partnersupport.microsoft.com/en ... 9729458808

GregorS
Veeam ProPartner
Posts: 33
Liked: 10 times
Joined: Apr 10, 2013 12:11 pm
Full Name: Gregor Smerke
Location: Ljubljana, SI - Slovenija
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by GregorS » Nov 04, 2017 3:05 pm

Having the same issue, just subscribing to this thread, thanks.

mats.jansson
Service Provider
Posts: 17
Liked: 1 time
Joined: Mar 18, 2014 9:13 am
Full Name: Mats

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by mats.jansson » Nov 04, 2017 4:39 pm

The only solution that workt for us was to uninstall azure ad connect, install sql express (we used 2016 sp1) and reinstalled azure ad connect with option to use the manual installed express and a service account for sql (it's also working to use a sql on another server).
Mats

Datatekniker
Lurker
Posts: 2
Liked: never
Joined: Nov 02, 2017 7:59 am
Full Name: Flemming
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by Datatekniker » Nov 06, 2017 6:20 am

This works,

If you can accept that the AADconnect, is dead in the wather for a short period of time, a simple solution is this.

Use task scheduler:

1. Stop the AADconnect service shortly before the Veeambackup is set to run. (Powershell Stop-Service -DisplayName 'Microsoft Azure AD Sync')

---- The backup is running

2. Start the AADconnect again after a the backup has finished. (mine takes about 15min, so I start the service after 30min.) (Powershell Start-Service -DisplayName 'Microsoft Azure AD Sync')

Thats it.
My backup has been running 3 times this weekend without any further problems :-)

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by mcz » Nov 06, 2017 8:08 am 1 person likes this post

Datatekniker wrote: 2. Start the AADconnect again after a the backup has finished. (mine takes about 15min, so I start the service after 30min.) (Powershell Start-Service -DisplayName 'Microsoft Azure AD Sync')
You should be able to start the service just after the snapshot has been taken (start it in the post-thaw script) because veeam releases vss-snapshot after vmware snapshot (if you are not indexing the filesystem).

bdufour
Expert
Posts: 198
Liked: 29 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by bdufour » Nov 06, 2017 4:30 pm 1 person likes this post

i haven't had to restart this server since i wrote my original post about changing the login permissions to the service..

Re: Bunch of servers VSS Writer Errror 0x800423f4
Veeam Logo by bdufour » Wed Nov 01, 2017 9:01 pm

ive been experiencing this issue - i found this - https://social.technet.microsoft.com/Fo ... errecovery

typically, i have to restart the server for it to complete a backup/rep. it usually requires a restart every few day as related to Azure AD Sync updating likely and causing issues with permissions. i changed the log on account of the SQL server VSS writer service to a domain service account with proper permissions to the DB. for now that seemed to solve it. i will give an update in a week or so to confirm whether or not the issue has resurfaced.

jeradries
Lurker
Posts: 1
Liked: never
Joined: Nov 07, 2017 8:22 pm
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by jeradries » Nov 07, 2017 8:29 pm

scott_thomson wrote:This definitely looks like a MS issue, but I haven't found any real acknowledgement of the problem anywhere from them. In hopes that we can raise the profile of the issue, I've posted to the AD Connect UserVoice as well as the MS Partner Support forums. Hopefully if you can add your voices to this, someone at MS will get involved:

https://feedback.azure.com/forums/16940 ... it-doesn-t
https://partnersupport.microsoft.com/en ... 9729458808
I have been following your post on the MS Partner Support forum and when looking at the SyncEngine-AutoUpgrader-XXXXXXXX-XXXXXX.log files in C:\ProgramData\AADConnect, I am seeing the auto-upgrade process being triggered every 6 hours as well starting on 10/23. However, what caught my eye is "Upgrading AADConnect to version: 1.1.648.0".

According to the version release history at https://docs.microsoft.com/en-us/azure/ ... on-history, this isn't a valid version number. Are you seeing the same thing?

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by mcz » Nov 08, 2017 7:47 am

Yep, I see the same version number in the logfile.

dnorth
Novice
Posts: 3
Liked: never
Joined: Nov 08, 2017 4:16 pm
Full Name: David North
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by dnorth » Nov 08, 2017 5:04 pm

bdufour wrote:i changed the log on account of the SQL server VSS writer service to a domain service account with proper permissions to the DB. for now that seemed to solve it.
Changing the service account seems to be working for me. The first Veeam job after an Auto (non-)Upgrade completed without an error anyway. Thanks, Blake.

bdufour
Expert
Posts: 198
Liked: 29 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by bdufour » Nov 08, 2017 6:36 pm

Changing the service account seems to be working for me. The first Veeam job after an Auto (non-)Upgrade completed without an error anyway. Thanks, Blake.[/quote]

no problem, i still haven't had to restart this server and all jobs for it are still completing without any errors. at this point, i don't think it's just a coincidence and this may have fixed the issue on my end.

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by mcz » Nov 09, 2017 7:45 am

In my case, repairing the express-db-instance has resolved the issue...

Marius_Aad
Lurker
Posts: 2
Liked: 1 time
Joined: Nov 09, 2017 8:47 am
Full Name: Marius
Contact:

Re: Bunch of servers VSS Writer Errror 0x800423f4

Post by Marius_Aad » Nov 09, 2017 8:56 am

Upgrading MSSQL Server 2012 Express LocalDB to SP4 fixed the issue for me. Multiple back up jobs finished successfully after that without restarting the vm.

Post Reply

Who is online

Users browsing this forum: emil.davis, Google [Bot] and 18 guests