Availability for the Always-On Enterprise
cjack03
Enthusiast
Posts: 28
Liked: 9 times
Joined: Jul 01, 2013 3:26 pm
Full Name: Christopher
Contact:

upgrade to v9 and Exchange 2013 DAG backup warning

Post by cjack03 » Feb 02, 2016 5:03 pm 1 person likes this post

Hi all,

After upgrading to v9 we are receiving warnings when backing up our Exchange 2013 DAG environment:

Code: Select all

VSS: Backup job failed.
Cannot notify writers about the 'BACKUP FINISH' event.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{44eed754-5b98-4c2b-8aeb-3922993eafe2}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3]. 
This error is only thrown on the active member of the DAG. We have tried to fail over and rerun the backup and we re-produce the same error.

Before opening a ticket I wondered if anyone else had come across this?

Thanks

allroy1975
Novice
Posts: 3
Liked: never
Joined: Feb 02, 2016 10:03 pm
Full Name: matt
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by allroy1975 » Feb 02, 2016 10:25 pm

I'm having it. Never got it with 8. We just got the backup running in the last few days.

the weird thing is ...On Sunday (1-29-2016) the job from Saturday night threw these errors:

Code: Select all

Processing EXCMB1 - Exchange 2013 Mailbox/Transport Server Error: SQL server is not available
Error: SQL server is not available
Uhhhh....yeah....it's NOT available...it's not a SQL server!
when it failed with that error, it kicked off again, redid the whole job and finished a few hours later with no errors:

Code: Select all

EXCMB1 - Exchange 2013 Mailbox/Transport Server Success10:26:40 AM3:15:32 PM3.7 TB233.3 GB64.8 GB4:48:52 
Every Job since then has thrown:

Code: Select all

VSS: Backup job failed. Cannot notify writers about the 'BACKUP FINISH' event. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{7ed6c0f3-7148-4eee-8080-78d56a5a0cb3}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3].
Granted, that's only the Sunday night and Monday night backups. I was going to dig into this today, but ...time slipped away, and just found your post hoping for something I could try late in the day.... :D

https://www.veeam.com/kb1680 makes a couple suggestions....UAC is disabled and I'm running the job as the same account that ran it successfully on Sunday....so...for now I'm lost, but I wanted to chime in real quick and at least tell you I'm having the same problem with 9 on a 2013 dag member.

foggy
Veeam Software
Posts: 16691
Liked: 1343 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by foggy » Feb 03, 2016 11:59 am

Guys, I encourage you both to contact support to investigate the failures and will appreciate posting your case ID numbers here. Thanks.

cjack03
Enthusiast
Posts: 28
Liked: 9 times
Joined: Jul 01, 2013 3:26 pm
Full Name: Christopher
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by cjack03 » Feb 04, 2016 10:58 am 1 person likes this post

Case number is 01688706. Opened by a colleague not myself. Thanks for the help.

allroy1975
Novice
Posts: 3
Liked: never
Joined: Feb 02, 2016 10:03 pm
Full Name: matt
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by allroy1975 » Feb 04, 2016 5:27 pm

Look what I got last night!

Code: Select all

EXCMB1 - Exchange 2013 Mailbox/Transport ServerSuccess8:40:24 PM12:04:43 AM (+1)3.7 TB271.5 GB92.1 GB3:24:19 
apparently one of my co-workers did some work on his own yesterday morning. He ran this command from an elevated command prompt on the mailbox server:

Code: Select all

vssadmin list writers
it said Writer name: 'Microsoft Exchange Writer' was Failed, and his research said to try stopping and starting the exchange Information Store and see if that fixed it....or reboot. He stopped and started the Information Store, but that didn't fix it. So, he rebooted the Exchange server yesterday morning. Then the Exchange writer said it was stable. Last night we got a good backup. w00t!

just wanted to share this since we had pretty much the exact same error. Hopefully it'll help you too, or someone else. :D

ts7
Lurker
Posts: 2
Liked: never
Joined: Feb 04, 2016 9:50 pm
Full Name: Tom S
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by ts7 » Feb 04, 2016 10:01 pm

We have been seeing the exact same problem since upgrading to V9 and have opened case# 1678362. Rebooting the Exchange Server resulted in a successful backup that evening but the next day it failed again with the same error. It seems the only way to pull a successful backup is to reboot the Exchange server before starting the backup. The last suggestion was to try re-registering VSS on the Exchange server but I'm not sure exactly how to do that and much of what I have found looking for how to do it suggest NOT to do it on Windows Server 2008 or higher due to the manifest and the possibility it will result in the need to rebuild the server - definitely not a good thing when you can't get backups to pull successfully!!

Since multiple customers seem to be experiencing the exact same issue after upgrading to V9 it would appear to be an issue that Veeam needs to get fixed. Fortunately this only seems to occur on the last server in the Exchange backup. We do not see it on any other servers.

allroy1975
Novice
Posts: 3
Liked: never
Joined: Feb 02, 2016 10:03 pm
Full Name: matt
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by allroy1975 » Feb 05, 2016 4:06 pm

That sucks Tom.....Mine ran successfully last night...so...2 in a row for me. I hope I don't have to come back to this thread...but...

ts7
Lurker
Posts: 2
Liked: never
Joined: Feb 04, 2016 9:50 pm
Full Name: Tom S
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by ts7 » Feb 05, 2016 4:15 pm

I will try to install any outstanding Windows Updates then reboot all my Exchange servers and my backup server this weekend unless I hear anything else from Veeam. I'm a bit disappointed in Veeam support as they have not owned up to the problem, have not provided a solution that works and responses are slow (a day or more after I submit an update to the case). I tried escalating to the top priority but was reminded that since we did not have a production down situation it was not worthy of that status (it doesn't seem to matter what will happen if I lose an Exchange Server without a successful backup).

DeLiriOusNoMaD
Novice
Posts: 6
Liked: never
Joined: Dec 29, 2015 10:25 pm
Full Name: Bill Athineos
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by DeLiriOusNoMaD » Feb 07, 2016 9:16 pm

allroy1975 wrote:Look what I got last night!
EXCMB1 - Exchange 2013 Mailbox/Transport ServerSuccess8:40:24 PM12:04:43 AM (+1)3.7 TB271.5 GB92.1 GB3:24:19

apparently one of my co-workers did some work on his own yesterday morning. He ran this command from an elevated command prompt on the mailbox server:
vssadmin list writers

it said Writer name: 'Microsoft Exchange Writer' was Failed, and his research said to try stopping and starting the exchange Information Store and see if that fixed it....or reboot. He stopped and started the Information Store, but that didn't fix it. So, he rebooted the Exchange server yesterday morning. Then the Exchange writer said it was stable. Last night we got a good backup. w00t!

just wanted to share this since we had pretty much the exact same error. Hopefully it'll help you too, or someone else. :D
im also have a similar issue with an Exchange 2013 DAG that i didnt have with v8. Ive also opened a case but it didnt get anywhere. Now that i see others are having writers misbehaving maybe someone at Veeam will take notice. One other workaround which always works for me restarting the MS Exchange Replication Service. Maybe ill reopen my case or open a new one

Andreas Neufert
Veeam Software
Posts: 2818
Liked: 478 times
Joined: May 04, 2011 8:36 am
Full Name: @AndyandtheVMs Veeam PM
Location: Germany
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by Andreas Neufert » Feb 08, 2016 8:20 am


jromalino
Influencer
Posts: 10
Liked: 1 time
Joined: Apr 29, 2013 2:30 pm
Full Name: John Romalino
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by jromalino » Feb 08, 2016 4:35 pm

We are having the same problem with v9. No problems in v8. All databases are mounted. Opened case number 01691782 but have't been contacted yet.

MurkyBuffalo
Lurker
Posts: 2
Liked: never
Joined: Feb 08, 2016 5:48 pm
Full Name: EJ Pennyman
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by MurkyBuffalo » Feb 08, 2016 6:02 pm

Hello, gents.

I am about to add a second Exchange server, creating a DAG. This post has me nervous. I've seen VSS writer issues before that were related to timeouts being exceeded.

To those having the issue: What is your storage situation? Are you on HDDs or SSDs?

Andreas Neufert
Veeam Software
Posts: 2818
Liked: 478 times
Joined: May 04, 2011 8:36 am
Full Name: @AndyandtheVMs Veeam PM
Location: Germany
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by Andreas Neufert » Feb 09, 2016 9:45 am

Timeout related VSS writer problems have been addressed with v8 new method of VSS SnapShot. If B&R detect, that an Exchange Server needs more time than the hard coded Exchange 20sec VSS writer timouts we create a special VSS SnapShot to help Exchange to get the consistency. From what I can see the problems above have nothing to do with timeouts.

VSS Timout problems are not a typical DAG phenomene and the above described errors can happen as well in non DAG environments. You can google the above error code and it will tell you a lot of root causes.

Best option if you see the above problem is to find the root cause in Exchange and fix it together with our support.

Murkey if you change your exchange to a DAG enironment you need to increase heartbeat time.

Andreas Neufert
Veeam Software
Posts: 2818
Liked: 478 times
Joined: May 04, 2011 8:36 am
Full Name: @AndyandtheVMs Veeam PM
Location: Germany
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by Andreas Neufert » Feb 09, 2016 8:30 pm

Hi everybody,

To answer your questions based on Feedback from Veeam support (this is not a support forum):
Did we changed our code in v9 regarding VSS processing (beside support new Microsoft products). No.
Did we changed our code in case of reporting VSS related errors. Yes. There is a technical reason why we report this error now. Please fix the root cause of the VSS problem at Exchange. Our support team can assist you.

The error 0x800423f3 is pretty generic. If you google it you can find a lot of root causes.
Maybe it is as well a good idea to install latest Exchange and OS updates and perform a reboot to see if this fixes the error.

If you fixed it with support, please help other Forum members and let us know the root cause.

Gostev
Veeam Software
Posts: 22808
Liked: 2801 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by Gostev » Feb 09, 2016 8:38 pm 2 people like this post

jromalino wrote:We are having the same problem with v9. No problems in v8.
All, I've checked on this issue with the devs. It appears that v8 had a defect (ID 60731) which caused jobs not to log a warning in the situation when Microsoft Exchange VSS Writer would stuck in VSS_WS_FAILED_AT_BACKUP_COMPLETE state during backup. This is the reason why you did not see this warning in v8 (it does not mean you did not have the actual issue with your Exchange server).

The bigger issue is that VSS Writer stucking in this state causes transaction logs to not be truncated. We had a customer open a support ticket where transaction logs would not truncate despite the job reporting success. After investigating this situation,- we have delivered a hot fix for v8 U3 to correctly report this state as a warning, and then rolled it into v9.

Thanks!

P.S. OK, looks like Andreas beat me by a few minutes.

Post Reply

Who is online

Users browsing this forum: No registered users and 25 guests