upgrade to v9 and Exchange 2013 DAG backup warning

[cjack03]

Hi all,

After upgrading to v9 we are receiving warnings when backing up our Exchange 2013 DAG environment:

Code: Select all

VSS: Backup job failed.
Cannot notify writers about the 'BACKUP FINISH' event.
A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{44eed754-5b98-4c2b-8aeb-3922993eafe2}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3]. 

This error is only thrown on the active member of the DAG. We have tried to fail over and rerun the backup and we re-produce the same error.

Before opening a ticket I wondered if anyone else had come across this?

Thanks

[allroy1975]

I'm having it. Never got it with 8. We just got the backup running in the last few days.

the weird thing is ...On Sunday (1-29-2016) the job from Saturday night threw these errors:

Code: Select all

Processing EXCMB1 - Exchange 2013 Mailbox/Transport Server Error: SQL server is not available
Error: SQL server is not available

Uhhhh....yeah....it's NOT available...it's not a SQL server!
when it failed with that error, it kicked off again, redid the whole job and finished a few hours later with no errors:

Code: Select all

EXCMB1 - Exchange 2013 Mailbox/Transport Server Success10:26:40 AM3:15:32 PM3.7 TB233.3 GB64.8 GB4:48:52 

Every Job since then has thrown:

Code: Select all

VSS: Backup job failed. Cannot notify writers about the 'BACKUP FINISH' event. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{7ed6c0f3-7148-4eee-8080-78d56a5a0cb3}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3].

Granted, that's only the Sunday night and Monday night backups. I was going to dig into this today, but ...time slipped away, and just found your post hoping for something I could try late in the day....

https://www.veeam.com/kb1680 makes a couple suggestions....UAC is disabled and I'm running the job as the same account that ran it successfully on Sunday....so...for now I'm lost, but I wanted to chime in real quick and at least tell you I'm having the same problem with 9 on a 2013 dag member.

[foggy]

Guys, I encourage you both to contact support to investigate the failures and will appreciate posting your case ID numbers here. Thanks.

[cjack03]

Case number is 01688706. Opened by a colleague not myself. Thanks for the help.

[allroy1975]

Look what I got last night!

Code: Select all

EXCMB1 - Exchange 2013 Mailbox/Transport ServerSuccess8:40:24 PM12:04:43 AM (+1)3.7 TB271.5 GB92.1 GB3:24:19 

apparently one of my co-workers did some work on his own yesterday morning. He ran this command from an elevated command prompt on the mailbox server:

Code: Select all

vssadmin list writers

it said Writer name: 'Microsoft Exchange Writer' was Failed, and his research said to try stopping and starting the exchange Information Store and see if that fixed it....or reboot. He stopped and started the Information Store, but that didn't fix it. So, he rebooted the Exchange server yesterday morning. Then the Exchange writer said it was stable. Last night we got a good backup. w00t!

just wanted to share this since we had pretty much the exact same error. Hopefully it'll help you too, or someone else.

[ts7]

We have been seeing the exact same problem since upgrading to V9 and have opened case# 1678362. Rebooting the Exchange Server resulted in a successful backup that evening but the next day it failed again with the same error. It seems the only way to pull a successful backup is to reboot the Exchange server before starting the backup. The last suggestion was to try re-registering VSS on the Exchange server but I'm not sure exactly how to do that and much of what I have found looking for how to do it suggest NOT to do it on Windows Server 2008 or higher due to the manifest and the possibility it will result in the need to rebuild the server - definitely not a good thing when you can't get backups to pull successfully!!

Since multiple customers seem to be experiencing the exact same issue after upgrading to V9 it would appear to be an issue that Veeam needs to get fixed. Fortunately this only seems to occur on the last server in the Exchange backup. We do not see it on any other servers.

[allroy1975]

That sucks Tom.....Mine ran successfully last night...so...2 in a row for me. I hope I don't have to come back to this thread...but...

[ts7]

I will try to install any outstanding Windows Updates then reboot all my Exchange servers and my backup server this weekend unless I hear anything else from Veeam. I'm a bit disappointed in Veeam support as they have not owned up to the problem, have not provided a solution that works and responses are slow (a day or more after I submit an update to the case). I tried escalating to the top priority but was reminded that since we did not have a production down situation it was not worthy of that status (it doesn't seem to matter what will happen if I lose an Exchange Server without a successful backup).

[DeLiriOusNoMaD]

Look what I got last night!
EXCMB1 - Exchange 2013 Mailbox/Transport ServerSuccess8:40:24 PM12:04:43 AM (+1)3.7 TB271.5 GB92.1 GB3:24:19

apparently one of my co-workers did some work on his own yesterday morning. He ran this command from an elevated command prompt on the mailbox server:
vssadmin list writers

it said Writer name: 'Microsoft Exchange Writer' was Failed, and his research said to try stopping and starting the exchange Information Store and see if that fixed it....or reboot. He stopped and started the Information Store, but that didn't fix it. So, he rebooted the Exchange server yesterday morning. Then the Exchange writer said it was stable. Last night we got a good backup. w00t!

just wanted to share this since we had pretty much the exact same error. Hopefully it'll help you too, or someone else.

im also have a similar issue with an Exchange 2013 DAG that i didnt have with v8. Ive also opened a case but it didnt get anywhere. Now that i see others are having writers misbehaving maybe someone at Veeam will take notice. One other workaround which always works for me restarting the MS Exchange Replication Service. Maybe ill reopen my case or open a new one

[Andreas Neufert]

Maybe this is helpful:
http://www.alexandreviot.net/2015/06/04 ... -800423f3/

[jromalino]

We are having the same problem with v9. No problems in v8. All databases are mounted. Opened case number 01691782 but have't been contacted yet.

[MurkyBuffalo]

Hello, gents.

I am about to add a second Exchange server, creating a DAG. This post has me nervous. I've seen VSS writer issues before that were related to timeouts being exceeded.

To those having the issue: What is your storage situation? Are you on HDDs or SSDs?

[Andreas Neufert]

Timeout related VSS writer problems have been addressed with v8 new method of VSS SnapShot. If B&R detect, that an Exchange Server needs more time than the hard coded Exchange 20sec VSS writer timouts we create a special VSS SnapShot to help Exchange to get the consistency. From what I can see the problems above have nothing to do with timeouts.

VSS Timout problems are not a typical DAG phenomene and the above described errors can happen as well in non DAG environments. You can google the above error code and it will tell you a lot of root causes.

Best option if you see the above problem is to find the root cause in Exchange and fix it together with our support.

Murkey if you change your exchange to a DAG enironment you need to increase heartbeat time.

[Andreas Neufert]

Hi everybody,

To answer your questions based on Feedback from Veeam support (this is not a support forum):
Did we changed our code in v9 regarding VSS processing (beside support new Microsoft products). No.
Did we changed our code in case of reporting VSS related errors. Yes. There is a technical reason why we report this error now. Please fix the root cause of the VSS problem at Exchange. Our support team can assist you.

The error 0x800423f3 is pretty generic. If you google it you can find a lot of root causes.
Maybe it is as well a good idea to install latest Exchange and OS updates and perform a reboot to see if this fixes the error.

If you fixed it with support, please help other Forum members and let us know the root cause.

[Gostev]

We are having the same problem with v9. No problems in v8.

All, I've checked on this issue with the devs. It appears that v8 had a defect (ID 60731) which caused jobs not to log a warning in the situation when Microsoft Exchange VSS Writer would stuck in VSS_WS_FAILED_AT_BACKUP_COMPLETE state during backup. This is the reason why you did not see this warning in v8 (it does not mean you did not have the actual issue with your Exchange server).

The bigger issue is that VSS Writer stucking in this state causes transaction logs to not be truncated. We had a customer open a support ticket where transaction logs would not truncate despite the job reporting success. After investigating this situation,- we have delivered a hot fix for v8 U3 to correctly report this state as a warning, and then rolled it into v9.

Thanks!

P.S. OK, looks like Andreas beat me by a few minutes.

[jromalino]

I just got off the phone with Microsoft Pro support about this problem. All the testing of the Microsoft Exchange Writer from a Microsoft standpoint passed (and believe me they tried to make it fail). The problem seems to be with Veeam v9 and the processes they are passing to the Microsoft Exchange Writer. Trying to get my Veeam case escalated as Veeam support said it was a Microsoft issue. This problem didn't start until I upgraded to v9.

[Andreas Neufert]

Thanks for your feedback. To avoid any fingerpointing like steps, please contact our support.
They have a set of native windows commands that start a VSS based snapshot which we start at backup times as well.
It is based on Windows "diskshadow". With this you can figure out if Windows/Exchange or Veeam is the problem here.
Please forward our support your MS ticket number as well.

Thanks in advance... Andy

[btanguay]

Hi jromalino,
Did you get any update on your MS ticket or veeam support case. i've just upgrade to v9 yesterday and got the error two time in a row.

Andreeas, Gosdev,
Any update on that issue from Veeam? I've got some 2010 and 2013 mailbox server and this only happen in the 2013 active server mailbox.

[btanguay]

Hi,

fyi, if the Exchange Writer is in the failed state and you can't restart your Exchange server, you need to restart these services to get it in stable state. It does not solve the issue, but at least it faster to test again.

- COM+ Event System
- Volume Shadow Copy
- Microsoft Exchange Information Store (my Dag failover to secondary node, but i'm not sure is required.. i will test again on the next failed backup)
- Microsoft Exchange Replication service.

[Andreas Neufert]

Thanks Benoit,
just want to add here that after service restart, the Writer sometimes need 1-2 min till it register itself at writers list again.
We had situations where customers rebooted some of the services before backup and no Application Awareness was processed for Exchange as the writer was not there.
So wait 2-3min after service restart before you start the backup.
You can check with "vssadmin list writers" on command line if it is back.

[phite]

Just wanted to add my name to the list here - We are also seeing this warning message post-V9 upgrade. Like a previous poster, it only appears on the last server in the DAG to get backed up. We've rebooted and ensured the writer goes back to a stable state, but it just fails again. However, as far as I can tell, our Exchange logs are properly truncating. We have a 3-node DAG with 6 databases.

MS has been a nightmare on this and just points back at Veeam. I'm not saying it isn't an MS issue, but since there appear to be several of us in the same boat it would be great if Veeam could provide a KB once someone determines the cause of the problem

[bmars]

I was having the very same issue. My jobs completed successfully (sometimes) but even though they were successful half the DB's wouldn't mount when testing a restore, so they were essentially useless. I was to the point of rebooting each morning so that I wouldn't get the errors. I ended up creating a new job and that appears to have cleared up the issue in my case. 3 successful backups and I am able to mount and restore from each of them.

[btanguay]

Hi,
Anybody already try that solutions ?
http://exchangeserverpro.com/event-id-2 ... ox-server/ ?

[PramodNambiar]

We had similar issue on our single server Exchange 2013 setup.
We have been using Veeam B&R v8 for more than a month and Exchange backup completes without any error. But I noticed that the Logs are not being truncated.
So I have been working with the Veeam support team on this issue for a while now without any success.
Today I decided to upgrade Veeam to v9 and got the warning message when I ran the Exchange backup.
The warning message said “VSS: Backup job failed. Cannot notify writers about the 'BACKUP FINISH' event. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eb03ad45-bdc3-471b-a9bb-eccbbe6718eb}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3]”

When I dig deep in the exchange events I realized that the one of the unmounted database making the VSS writer to unstable status and failing to commit the logs.
So I removed the database from the exchange server and the backup completed without any errors and truncated the logs.

[Andreas Neufert]

Hello everybody,
just wanted to share that there was another situation fixed with our support. Situation was that all standard backup application (that do not use VMware SnapShots) worked flawless and only Veeam had the problem with 0x800423f3. It turned out that at VMware SnapShot commit the DAG cluster performed an failover which prevented Veeam to perform logfile truncation. Our support fixed the Failover root cause (Increase of Exchange heartbeat time and change from Hotadd to NBD mode) and everything worked again as expected.

I took this as a chance to update my general Exchange DAG + VMware + Veeam Tips and Tricks Blogpost:
http://andyandthevms.com/exchange-dag-v ... plication/

[jromalino]

Hi jromalino,
Did you get any update on your MS ticket or veeam support case. i've just upgrade to v9 yesterday and got the error two time in a row.

Andreeas, Gosdev,
Any update on that issue from Veeam? I've got some 2010 and 2013 mailbox server and this only happen in the 2013 active server mailbox.

Everything tested from a Microsoft standpoint worked as expected. They have since closed the case. I really don't think it's a Microsoft problem as I am not having any Exchange issues and this problem didn't start until I upgraded to V9. Never had the problem with V8.

No luck with Veeam support. They escalated the case to tier 2 support but still don't have a resolution. I get the impression they don't know what it is. I've been through at least 4 or 5 remote support sessions only to watch them poke around looking at logs. The last thing they had me do was run a chkdsk /f on all the volumes and sfc /scannow. Everything came up clean and caused me a lot of extra Exchange maintenance due to the need to remove the server from the DAG before running the commands. Needless to say, I am very frustrated with the situation. I have even tried extending the DAG heartbeat as indicated but it did not make a difference.

[Andreas Neufert]

Can you please check if a failover happens at Snapshot commit?

[jromalino]

Yes. Fail over is occurring during the backup job. I had a separate case opened with support and asked if this could be part of the issue and they told me no. They did have me change the DAG heartbeat values but it has not solved the issue of fail over during a backup job or the VSS problem.

[Andreas Neufert]

Hi John,

so this is I belive the root cause of the error. Please work on the Failover and the other message will go away. Please use my tips and tricks here to solve that.
http://andyandthevms.com/exchange-dag-v ... plication/

[btanguay]

Finally open a support case : Case # 01721088 for that issue.
I'm also having an issue with the retention policy. It's does not seems to delete the old copy of the backup even if the job completed with warning. The Synthetic full backup is created successfully, but since the upgrade the retention does not apply and storage start to fillout. Anybody is having the same issue with the retention policy on the Exchange jobs.