Comprehensive data protection for all workloads
jromalino
Influencer
Posts: 10
Liked: 1 time
Joined: Apr 29, 2013 2:30 pm
Full Name: John Romalino
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by jromalino » Feb 10, 2016 4:34 pm 1 person likes this post

I just got off the phone with Microsoft Pro support about this problem. All the testing of the Microsoft Exchange Writer from a Microsoft standpoint passed (and believe me they tried to make it fail). The problem seems to be with Veeam v9 and the processes they are passing to the Microsoft Exchange Writer. Trying to get my Veeam case escalated as Veeam support said it was a Microsoft issue. This problem didn't start until I upgraded to v9.

Andreas Neufert
Veeam Software
Posts: 3230
Liked: 564 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert Director of Product Management Global Alliances
Location: Germany
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by Andreas Neufert » Feb 12, 2016 9:46 am

Thanks for your feedback. To avoid any fingerpointing like steps, please contact our support.
They have a set of native windows commands that start a VSS based snapshot which we start at backup times as well.
It is based on Windows "diskshadow". With this you can figure out if Windows/Exchange or Veeam is the problem here.
Please forward our support your MS ticket number as well.

Thanks in advance... Andy

btanguay
Influencer
Posts: 18
Liked: 2 times
Joined: Nov 28, 2014 6:58 pm
Full Name: Benoit Tanguay
Location: Boucherville, QC, Canada
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by btanguay » Feb 17, 2016 3:05 am

Hi jromalino,
Did you get any update on your MS ticket or veeam support case. i've just upgrade to v9 yesterday and got the error two time in a row.

Andreeas, Gosdev,
Any update on that issue from Veeam? I've got some 2010 and 2013 mailbox server and this only happen in the 2013 active server mailbox.

btanguay
Influencer
Posts: 18
Liked: 2 times
Joined: Nov 28, 2014 6:58 pm
Full Name: Benoit Tanguay
Location: Boucherville, QC, Canada
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by btanguay » Feb 17, 2016 3:26 am

Hi,

fyi, if the Exchange Writer is in the failed state and you can't restart your Exchange server, you need to restart these services to get it in stable state. It does not solve the issue, but at least it faster to test again.

- COM+ Event System
- Volume Shadow Copy
- Microsoft Exchange Information Store (my Dag failover to secondary node, but i'm not sure is required.. i will test again on the next failed backup)
- Microsoft Exchange Replication service.

Andreas Neufert
Veeam Software
Posts: 3230
Liked: 564 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert Director of Product Management Global Alliances
Location: Germany
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by Andreas Neufert » Feb 17, 2016 8:04 am 1 person likes this post

Thanks Benoit,
just want to add here that after service restart, the Writer sometimes need 1-2 min till it register itself at writers list again.
We had situations where customers rebooted some of the services before backup and no Application Awareness was processed for Exchange as the writer was not there.
So wait 2-3min after service restart before you start the backup.
You can check with "vssadmin list writers" on command line if it is back.

phite
Lurker
Posts: 1
Liked: never
Joined: Feb 17, 2016 2:10 pm
Full Name: Paul Hite
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by phite » Feb 17, 2016 3:04 pm

Just wanted to add my name to the list here - We are also seeing this warning message post-V9 upgrade. Like a previous poster, it only appears on the last server in the DAG to get backed up. We've rebooted and ensured the writer goes back to a stable state, but it just fails again. However, as far as I can tell, our Exchange logs are properly truncating. We have a 3-node DAG with 6 databases.

MS has been a nightmare on this and just points back at Veeam. I'm not saying it isn't an MS issue, but since there appear to be several of us in the same boat it would be great if Veeam could provide a KB once someone determines the cause of the problem :)

bmars
Lurker
Posts: 1
Liked: never
Joined: Aug 25, 2015 12:43 pm
Full Name: Ben Marstaller
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by bmars » Feb 18, 2016 2:07 pm

I was having the very same issue. My jobs completed successfully (sometimes) but even though they were successful half the DB's wouldn't mount when testing a restore, so they were essentially useless. I was to the point of rebooting each morning so that I wouldn't get the errors. I ended up creating a new job and that appears to have cleared up the issue in my case. 3 successful backups and I am able to mount and restore from each of them.

btanguay
Influencer
Posts: 18
Liked: 2 times
Joined: Nov 28, 2014 6:58 pm
Full Name: Benoit Tanguay
Location: Boucherville, QC, Canada
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by btanguay » Feb 29, 2016 5:36 pm

Hi,
Anybody already try that solutions ?
http://exchangeserverpro.com/event-id-2 ... ox-server/ ?

PramodNambiar
Lurker
Posts: 1
Liked: never
Joined: Mar 01, 2016 10:42 am
Full Name: Pramod Nambiar
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by PramodNambiar » Mar 01, 2016 11:26 am

We had similar issue on our single server Exchange 2013 setup.
We have been using Veeam B&R v8 for more than a month and Exchange backup completes without any error. But I noticed that the Logs are not being truncated.
So I have been working with the Veeam support team on this issue for a while now without any success.
Today I decided to upgrade Veeam to v9 and got the warning message when I ran the Exchange backup.
The warning message said “VSS: Backup job failed. Cannot notify writers about the 'BACKUP FINISH' event. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eb03ad45-bdc3-471b-a9bb-eccbbe6718eb}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3]”

When I dig deep in the exchange events I realized that the one of the unmounted database making the VSS writer to unstable status and failing to commit the logs.
So I removed the database from the exchange server and the backup completed without any errors and truncated the logs.

Andreas Neufert
Veeam Software
Posts: 3230
Liked: 564 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert Director of Product Management Global Alliances
Location: Germany
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by Andreas Neufert » Mar 04, 2016 10:50 pm 1 person likes this post

Hello everybody,
just wanted to share that there was another situation fixed with our support. Situation was that all standard backup application (that do not use VMware SnapShots) worked flawless and only Veeam had the problem with 0x800423f3. It turned out that at VMware SnapShot commit the DAG cluster performed an failover which prevented Veeam to perform logfile truncation. Our support fixed the Failover root cause (Increase of Exchange heartbeat time and change from Hotadd to NBD mode) and everything worked again as expected.

I took this as a chance to update my general Exchange DAG + VMware + Veeam Tips and Tricks Blogpost:
http://andyandthevms.com/exchange-dag-v ... plication/

jromalino
Influencer
Posts: 10
Liked: 1 time
Joined: Apr 29, 2013 2:30 pm
Full Name: John Romalino
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by jromalino » Mar 07, 2016 3:04 am

btanguay wrote:Hi jromalino,
Did you get any update on your MS ticket or veeam support case. i've just upgrade to v9 yesterday and got the error two time in a row.

Andreeas, Gosdev,
Any update on that issue from Veeam? I've got some 2010 and 2013 mailbox server and this only happen in the 2013 active server mailbox.
Everything tested from a Microsoft standpoint worked as expected. They have since closed the case. I really don't think it's a Microsoft problem as I am not having any Exchange issues and this problem didn't start until I upgraded to V9. Never had the problem with V8.

No luck with Veeam support. They escalated the case to tier 2 support but still don't have a resolution. I get the impression they don't know what it is. I've been through at least 4 or 5 remote support sessions only to watch them poke around looking at logs. The last thing they had me do was run a chkdsk /f on all the volumes and sfc /scannow. Everything came up clean and caused me a lot of extra Exchange maintenance due to the need to remove the server from the DAG before running the commands. Needless to say, I am very frustrated with the situation. I have even tried extending the DAG heartbeat as indicated but it did not make a difference.

Andreas Neufert
Veeam Software
Posts: 3230
Liked: 564 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert Director of Product Management Global Alliances
Location: Germany
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by Andreas Neufert » Mar 07, 2016 5:46 am

Can you please check if a failover happens at Snapshot commit?

jromalino
Influencer
Posts: 10
Liked: 1 time
Joined: Apr 29, 2013 2:30 pm
Full Name: John Romalino
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by jromalino » Mar 07, 2016 5:07 pm

Yes. Fail over is occurring during the backup job. I had a separate case opened with support and asked if this could be part of the issue and they told me no. They did have me change the DAG heartbeat values but it has not solved the issue of fail over during a backup job or the VSS problem.

Andreas Neufert
Veeam Software
Posts: 3230
Liked: 564 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert Director of Product Management Global Alliances
Location: Germany
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by Andreas Neufert » Mar 08, 2016 8:12 am 1 person likes this post

Hi John,

so this is I belive the root cause of the error. Please work on the Failover and the other message will go away. Please use my tips and tricks here to solve that.
http://andyandthevms.com/exchange-dag-v ... plication/

btanguay
Influencer
Posts: 18
Liked: 2 times
Joined: Nov 28, 2014 6:58 pm
Full Name: Benoit Tanguay
Location: Boucherville, QC, Canada
Contact:

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Post by btanguay » Mar 09, 2016 2:16 am

Finally open a support case : Case # 01721088 for that issue.
I'm also having an issue with the retention policy. It's does not seems to delete the old copy of the backup even if the job completed with warning. The Synthetic full backup is created successfully, but since the upgrade the retention does not apply and storage start to fillout. Anybody is having the same issue with the retention policy on the Exchange jobs.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 31 guests