upgrade to v9 and Exchange 2013 DAG backup warning

Availability for the Always-On Enterprise

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby jromalino » Wed Feb 10, 2016 4:34 pm 1 person likes this post

I just got off the phone with Microsoft Pro support about this problem. All the testing of the Microsoft Exchange Writer from a Microsoft standpoint passed (and believe me they tried to make it fail). The problem seems to be with Veeam v9 and the processes they are passing to the Microsoft Exchange Writer. Trying to get my Veeam case escalated as Veeam support said it was a Microsoft issue. This problem didn't start until I upgraded to v9.
jromalino
Influencer
 
Posts: 10
Liked: 1 time
Joined: Mon Apr 29, 2013 2:30 pm
Full Name: John Romalino

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby Andreas Neufert » Fri Feb 12, 2016 9:46 am

Thanks for your feedback. To avoid any fingerpointing like steps, please contact our support.
They have a set of native windows commands that start a VSS based snapshot which we start at backup times as well.
It is based on Windows "diskshadow". With this you can figure out if Windows/Exchange or Veeam is the problem here.
Please forward our support your MS ticket number as well.

Thanks in advance... Andy
Andreas Neufert
Veeam Software
 
Posts: 2289
Liked: 383 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby btanguay » Wed Feb 17, 2016 3:05 am

Hi jromalino,
Did you get any update on your MS ticket or veeam support case. i've just upgrade to v9 yesterday and got the error two time in a row.

Andreeas, Gosdev,
Any update on that issue from Veeam? I've got some 2010 and 2013 mailbox server and this only happen in the 2013 active server mailbox.
btanguay
Influencer
 
Posts: 17
Liked: 2 times
Joined: Fri Nov 28, 2014 6:58 pm
Location: Boucherville, QC, Canada
Full Name: Benoit Tanguay

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby btanguay » Wed Feb 17, 2016 3:26 am

Hi,

fyi, if the Exchange Writer is in the failed state and you can't restart your Exchange server, you need to restart these services to get it in stable state. It does not solve the issue, but at least it faster to test again.

- COM+ Event System
- Volume Shadow Copy
- Microsoft Exchange Information Store (my Dag failover to secondary node, but i'm not sure is required.. i will test again on the next failed backup)
- Microsoft Exchange Replication service.
btanguay
Influencer
 
Posts: 17
Liked: 2 times
Joined: Fri Nov 28, 2014 6:58 pm
Location: Boucherville, QC, Canada
Full Name: Benoit Tanguay

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby Andreas Neufert » Wed Feb 17, 2016 8:04 am 1 person likes this post

Thanks Benoit,
just want to add here that after service restart, the Writer sometimes need 1-2 min till it register itself at writers list again.
We had situations where customers rebooted some of the services before backup and no Application Awareness was processed for Exchange as the writer was not there.
So wait 2-3min after service restart before you start the backup.
You can check with "vssadmin list writers" on command line if it is back.
Andreas Neufert
Veeam Software
 
Posts: 2289
Liked: 383 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby phite » Wed Feb 17, 2016 3:04 pm

Just wanted to add my name to the list here - We are also seeing this warning message post-V9 upgrade. Like a previous poster, it only appears on the last server in the DAG to get backed up. We've rebooted and ensured the writer goes back to a stable state, but it just fails again. However, as far as I can tell, our Exchange logs are properly truncating. We have a 3-node DAG with 6 databases.

MS has been a nightmare on this and just points back at Veeam. I'm not saying it isn't an MS issue, but since there appear to be several of us in the same boat it would be great if Veeam could provide a KB once someone determines the cause of the problem :)
phite
Lurker
 
Posts: 1
Liked: never
Joined: Wed Feb 17, 2016 2:10 pm
Full Name: Paul Hite

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby bmars » Thu Feb 18, 2016 2:07 pm

I was having the very same issue. My jobs completed successfully (sometimes) but even though they were successful half the DB's wouldn't mount when testing a restore, so they were essentially useless. I was to the point of rebooting each morning so that I wouldn't get the errors. I ended up creating a new job and that appears to have cleared up the issue in my case. 3 successful backups and I am able to mount and restore from each of them.
bmars
Lurker
 
Posts: 1
Liked: never
Joined: Tue Aug 25, 2015 12:43 pm
Full Name: Ben Marstaller

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby btanguay » Mon Feb 29, 2016 5:36 pm

Hi,
Anybody already try that solutions ?
http://exchangeserverpro.com/event-id-2 ... ox-server/ ?
btanguay
Influencer
 
Posts: 17
Liked: 2 times
Joined: Fri Nov 28, 2014 6:58 pm
Location: Boucherville, QC, Canada
Full Name: Benoit Tanguay

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby PramodNambiar » Tue Mar 01, 2016 11:26 am

We had similar issue on our single server Exchange 2013 setup.
We have been using Veeam B&R v8 for more than a month and Exchange backup completes without any error. But I noticed that the Logs are not being truncated.
So I have been working with the Veeam support team on this issue for a while now without any success.
Today I decided to upgrade Veeam to v9 and got the warning message when I ran the Exchange backup.
The warning message said “VSS: Backup job failed. Cannot notify writers about the 'BACKUP FINISH' event. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eb03ad45-bdc3-471b-a9bb-eccbbe6718eb}]. Writer's state: [VSS_WS_FAILED_AT_BACKUP_COMPLETE]. Error code: [0x800423f3]”

When I dig deep in the exchange events I realized that the one of the unmounted database making the VSS writer to unstable status and failing to commit the logs.
So I removed the database from the exchange server and the backup completed without any errors and truncated the logs.
PramodNambiar
Lurker
 
Posts: 1
Liked: never
Joined: Tue Mar 01, 2016 10:42 am
Full Name: Pramod Nambiar

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby Andreas Neufert » Fri Mar 04, 2016 10:50 pm 1 person likes this post

Hello everybody,
just wanted to share that there was another situation fixed with our support. Situation was that all standard backup application (that do not use VMware SnapShots) worked flawless and only Veeam had the problem with 0x800423f3. It turned out that at VMware SnapShot commit the DAG cluster performed an failover which prevented Veeam to perform logfile truncation. Our support fixed the Failover root cause (Increase of Exchange heartbeat time and change from Hotadd to NBD mode) and everything worked again as expected.

I took this as a chance to update my general Exchange DAG + VMware + Veeam Tips and Tricks Blogpost:
http://andyandthevms.com/exchange-dag-v ... plication/
Andreas Neufert
Veeam Software
 
Posts: 2289
Liked: 383 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby jromalino » Mon Mar 07, 2016 3:04 am

btanguay wrote:Hi jromalino,
Did you get any update on your MS ticket or veeam support case. i've just upgrade to v9 yesterday and got the error two time in a row.

Andreeas, Gosdev,
Any update on that issue from Veeam? I've got some 2010 and 2013 mailbox server and this only happen in the 2013 active server mailbox.


Everything tested from a Microsoft standpoint worked as expected. They have since closed the case. I really don't think it's a Microsoft problem as I am not having any Exchange issues and this problem didn't start until I upgraded to V9. Never had the problem with V8.

No luck with Veeam support. They escalated the case to tier 2 support but still don't have a resolution. I get the impression they don't know what it is. I've been through at least 4 or 5 remote support sessions only to watch them poke around looking at logs. The last thing they had me do was run a chkdsk /f on all the volumes and sfc /scannow. Everything came up clean and caused me a lot of extra Exchange maintenance due to the need to remove the server from the DAG before running the commands. Needless to say, I am very frustrated with the situation. I have even tried extending the DAG heartbeat as indicated but it did not make a difference.
jromalino
Influencer
 
Posts: 10
Liked: 1 time
Joined: Mon Apr 29, 2013 2:30 pm
Full Name: John Romalino

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby Andreas Neufert » Mon Mar 07, 2016 5:46 am

Can you please check if a failover happens at Snapshot commit?
Andreas Neufert
Veeam Software
 
Posts: 2289
Liked: 383 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby jromalino » Mon Mar 07, 2016 5:07 pm

Yes. Fail over is occurring during the backup job. I had a separate case opened with support and asked if this could be part of the issue and they told me no. They did have me change the DAG heartbeat values but it has not solved the issue of fail over during a backup job or the VSS problem.
jromalino
Influencer
 
Posts: 10
Liked: 1 time
Joined: Mon Apr 29, 2013 2:30 pm
Full Name: John Romalino

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby Andreas Neufert » Tue Mar 08, 2016 8:12 am 1 person likes this post

Hi John,

so this is I belive the root cause of the error. Please work on the Failover and the other message will go away. Please use my tips and tricks here to solve that.
http://andyandthevms.com/exchange-dag-v ... plication/
Andreas Neufert
Veeam Software
 
Posts: 2289
Liked: 383 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: upgrade to v9 and Exchange 2013 DAG backup warning

Veeam Logoby btanguay » Wed Mar 09, 2016 2:16 am

Finally open a support case : Case # 01721088 for that issue.
I'm also having an issue with the retention policy. It's does not seems to delete the old copy of the backup even if the job completed with warning. The Synthetic full backup is created successfully, but since the upgrade the retention does not apply and storage start to fillout. Anybody is having the same issue with the retention policy on the Exchange jobs.
btanguay
Influencer
 
Posts: 17
Liked: 2 times
Joined: Fri Nov 28, 2014 6:58 pm
Location: Boucherville, QC, Canada
Full Name: Benoit Tanguay

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 1 guest