upgrade to v9 and Exchange 2013 DAG backup warning

Availability for the Always-On Enterprise

upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby cjack03 » Tue Feb 02, 2016 5:03 pm 1 person likes this post

Hi all,

After upgrading to v9 we are receiving warnings when backing up our Exchange 2013 DAG environment:
Code: Select all
VSS: Backup job failed.
Cannot notify writers about the 'BACKUP FINISH' event.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{44eed754-5b98-4c2b-8aeb-3922993eafe2}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3].

This error is only thrown on the active member of the DAG. We have tried to fail over and rerun the backup and we re-produce the same error.

Before opening a ticket I wondered if anyone else had come across this?

Thanks
cjack03
Enthusiast
 
Posts: 28
Liked: 9 times
Joined: Mon Jul 01, 2013 3:26 pm
Full Name: Christopher

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby allroy1975 » Tue Feb 02, 2016 10:25 pm

I'm having it. Never got it with 8. We just got the backup running in the last few days.

the weird thing is ...On Sunday (1-29-2016) the job from Saturday night threw these errors:

Code: Select all
Processing EXCMB1 - Exchange 2013 Mailbox/Transport Server Error: SQL server is not available
Error: SQL server is not available


Uhhhh....yeah....it's NOT available...it's not a SQL server!
when it failed with that error, it kicked off again, redid the whole job and finished a few hours later with no errors:

Code: Select all
EXCMB1 - Exchange 2013 Mailbox/Transport Server Success10:26:40 AM3:15:32 PM3.7 TB233.3 GB64.8 GB4:48:52


Every Job since then has thrown:

Code: Select all
VSS: Backup job failed. Cannot notify writers about the 'BACKUP FINISH' event. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{7ed6c0f3-7148-4eee-8080-78d56a5a0cb3}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3].


Granted, that's only the Sunday night and Monday night backups. I was going to dig into this today, but ...time slipped away, and just found your post hoping for something I could try late in the day.... :D

https://www.veeam.com/kb1680 makes a couple suggestions....UAC is disabled and I'm running the job as the same account that ran it successfully on Sunday....so...for now I'm lost, but I wanted to chime in real quick and at least tell you I'm having the same problem with 9 on a 2013 dag member.
allroy1975
Novice
 
Posts: 3
Liked: never
Joined: Tue Feb 02, 2016 10:03 pm
Full Name: matt

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby foggy » Wed Feb 03, 2016 11:59 am

Guys, I encourage you both to contact support to investigate the failures and will appreciate posting your case ID numbers here. Thanks.
foggy
Veeam Software
 
Posts: 15094
Liked: 1111 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby cjack03 » Thu Feb 04, 2016 10:58 am 1 person likes this post

Case number is 01688706. Opened by a colleague not myself. Thanks for the help.
cjack03
Enthusiast
 
Posts: 28
Liked: 9 times
Joined: Mon Jul 01, 2013 3:26 pm
Full Name: Christopher

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby allroy1975 » Thu Feb 04, 2016 5:27 pm

Look what I got last night!

Code: Select all
EXCMB1 - Exchange 2013 Mailbox/Transport ServerSuccess8:40:24 PM12:04:43 AM (+1)3.7 TB271.5 GB92.1 GB3:24:19

apparently one of my co-workers did some work on his own yesterday morning. He ran this command from an elevated command prompt on the mailbox server:

Code: Select all
vssadmin list writers


it said Writer name: 'Microsoft Exchange Writer' was Failed, and his research said to try stopping and starting the exchange Information Store and see if that fixed it....or reboot. He stopped and started the Information Store, but that didn't fix it. So, he rebooted the Exchange server yesterday morning. Then the Exchange writer said it was stable. Last night we got a good backup. w00t!

just wanted to share this since we had pretty much the exact same error. Hopefully it'll help you too, or someone else. :D
allroy1975
Novice
 
Posts: 3
Liked: never
Joined: Tue Feb 02, 2016 10:03 pm
Full Name: matt

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby ts7 » Thu Feb 04, 2016 10:01 pm

We have been seeing the exact same problem since upgrading to V9 and have opened case# 1678362. Rebooting the Exchange Server resulted in a successful backup that evening but the next day it failed again with the same error. It seems the only way to pull a successful backup is to reboot the Exchange server before starting the backup. The last suggestion was to try re-registering VSS on the Exchange server but I'm not sure exactly how to do that and much of what I have found looking for how to do it suggest NOT to do it on Windows Server 2008 or higher due to the manifest and the possibility it will result in the need to rebuild the server - definitely not a good thing when you can't get backups to pull successfully!!

Since multiple customers seem to be experiencing the exact same issue after upgrading to V9 it would appear to be an issue that Veeam needs to get fixed. Fortunately this only seems to occur on the last server in the Exchange backup. We do not see it on any other servers.
ts7
Lurker
 
Posts: 2
Liked: never
Joined: Thu Feb 04, 2016 9:50 pm
Full Name: Tom S

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby allroy1975 » Fri Feb 05, 2016 4:06 pm

That sucks Tom.....Mine ran successfully last night...so...2 in a row for me. I hope I don't have to come back to this thread...but...
allroy1975
Novice
 
Posts: 3
Liked: never
Joined: Tue Feb 02, 2016 10:03 pm
Full Name: matt

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby ts7 » Fri Feb 05, 2016 4:15 pm

I will try to install any outstanding Windows Updates then reboot all my Exchange servers and my backup server this weekend unless I hear anything else from Veeam. I'm a bit disappointed in Veeam support as they have not owned up to the problem, have not provided a solution that works and responses are slow (a day or more after I submit an update to the case). I tried escalating to the top priority but was reminded that since we did not have a production down situation it was not worthy of that status (it doesn't seem to matter what will happen if I lose an Exchange Server without a successful backup).
ts7
Lurker
 
Posts: 2
Liked: never
Joined: Thu Feb 04, 2016 9:50 pm
Full Name: Tom S

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby DeLiriOusNoMaD » Sun Feb 07, 2016 9:16 pm

allroy1975 wrote:Look what I got last night!
EXCMB1 - Exchange 2013 Mailbox/Transport ServerSuccess8:40:24 PM12:04:43 AM (+1)3.7 TB271.5 GB92.1 GB3:24:19

apparently one of my co-workers did some work on his own yesterday morning. He ran this command from an elevated command prompt on the mailbox server:
vssadmin list writers

it said Writer name: 'Microsoft Exchange Writer' was Failed, and his research said to try stopping and starting the exchange Information Store and see if that fixed it....or reboot. He stopped and started the Information Store, but that didn't fix it. So, he rebooted the Exchange server yesterday morning. Then the Exchange writer said it was stable. Last night we got a good backup. w00t!

just wanted to share this since we had pretty much the exact same error. Hopefully it'll help you too, or someone else. :D


im also have a similar issue with an Exchange 2013 DAG that i didnt have with v8. Ive also opened a case but it didnt get anywhere. Now that i see others are having writers misbehaving maybe someone at Veeam will take notice. One other workaround which always works for me restarting the MS Exchange Replication Service. Maybe ill reopen my case or open a new one
DeLiriOusNoMaD
Novice
 
Posts: 6
Liked: never
Joined: Tue Dec 29, 2015 10:25 pm
Full Name: Bill Athineos

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby Andreas Neufert » Mon Feb 08, 2016 8:20 am

Andreas Neufert
Veeam Software
 
Posts: 2271
Liked: 378 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby jromalino » Mon Feb 08, 2016 4:35 pm

We are having the same problem with v9. No problems in v8. All databases are mounted. Opened case number 01691782 but have't been contacted yet.
jromalino
Influencer
 
Posts: 10
Liked: 1 time
Joined: Mon Apr 29, 2013 2:30 pm
Full Name: John Romalino

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby MurkyBuffalo » Mon Feb 08, 2016 6:02 pm

Hello, gents.

I am about to add a second Exchange server, creating a DAG. This post has me nervous. I've seen VSS writer issues before that were related to timeouts being exceeded.

To those having the issue: What is your storage situation? Are you on HDDs or SSDs?
MurkyBuffalo
Lurker
 
Posts: 1
Liked: never
Joined: Mon Feb 08, 2016 5:48 pm
Full Name: EJ Pennyman

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby Andreas Neufert » Tue Feb 09, 2016 9:45 am

Timeout related VSS writer problems have been addressed with v8 new method of VSS SnapShot. If B&R detect, that an Exchange Server needs more time than the hard coded Exchange 20sec VSS writer timouts we create a special VSS SnapShot to help Exchange to get the consistency. From what I can see the problems above have nothing to do with timeouts.

VSS Timout problems are not a typical DAG phenomene and the above described errors can happen as well in non DAG environments. You can google the above error code and it will tell you a lot of root causes.

Best option if you see the above problem is to find the root cause in Exchange and fix it together with our support.

Murkey if you change your exchange to a DAG enironment you need to increase heartbeat time.
Andreas Neufert
Veeam Software
 
Posts: 2271
Liked: 378 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby Andreas Neufert » Tue Feb 09, 2016 8:30 pm

Hi everybody,

To answer your questions based on Feedback from Veeam support (this is not a support forum):
Did we changed our code in v9 regarding VSS processing (beside support new Microsoft products). No.
Did we changed our code in case of reporting VSS related errors. Yes. There is a technical reason why we report this error now. Please fix the root cause of the VSS problem at Exchange. Our support team can assist you.

The error 0x800423f3 is pretty generic. If you google it you can find a lot of root causes.
Maybe it is as well a good idea to install latest Exchange and OS updates and perform a reboot to see if this fixes the error.

If you fixed it with support, please help other Forum members and let us know the root cause.
Andreas Neufert
Veeam Software
 
Posts: 2271
Liked: 378 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby Gostev » Tue Feb 09, 2016 8:38 pm 2 people like this post

jromalino wrote:We are having the same problem with v9. No problems in v8.

All, I've checked on this issue with the devs. It appears that v8 had a defect (ID 60731) which caused jobs not to log a warning in the situation when Microsoft Exchange VSS Writer would stuck in VSS_WS_FAILED_AT_BACKUP_COMPLETE state during backup. This is the reason why you did not see this warning in v8 (it does not mean you did not have the actual issue with your Exchange server).

The bigger issue is that VSS Writer stucking in this state causes transaction logs to not be truncated. We had a customer open a support ticket where transaction logs would not truncate despite the job reporting success. After investigating this situation,- we have delivered a hot fix for v8 U3 to correctly report this state as a warning, and then rolled it into v9.

Thanks!

P.S. OK, looks like Andreas beat me by a few minutes.
Gostev
Veeam Software
 
Posts: 21517
Liked: 2383 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot] and 52 guests