Veeam Backup Corrupting Exchange 2013 DB

Availability for the Always-On Enterprise

Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby tsukraw » Wed Sep 09, 2015 5:31 am

Hey Guys,
Just throwing this out there to see if anyone else has seen a similar issue.
In the past few weeks we have now seen this happen 3 separate times.

In 3 separate cases we were doing a new server builds for customers.
New 2012.R2 servers built with Exchange 2013 CU9.
All 3 cases are on Veeam v8 build 2018 and build 2030

In all 3 cases we had imported mail from PST files into the exchange server which i don't think is a factor but is common between all 3.

What we found in comparing time stamps in event viewer and the backup logs is when the very first Veeam Backup of the exchange server ran when it got to removing the snapshot and completing the task the exchange DB would dismount and from that point on we could not remount it. The event logs listed below are identical between all 3 separate cases.

In two of the cases we opened tickets with Microsoft and since we had no "good" backup yet they both required a rebuild from scratch due to the database being corrupted. We had not opened a ticket with Veeam since the backup job appeared to have ran just fine. No issues were reported so we did not suspect it.
The third one happened here tonight and since we were in the building state we will most likely just rebuild it as well, but will be calling Veeam in the AM to see what they think of this situation will post case ID at that time.

We just find it extremely strange that we have had 3 instances of this happen. And besides the type of setup being the same the only common factor is it happens at the exact same time the first backup is completely.

Anyone ever seen a issue like this?

Here are the event logs that are posted at the same time the job is completing. By the way the events sound it is almost like the exchange DB is not accessible for a bit when it is trying to write data to it.

Event 2008 - Microsoft Exchange Information Store worker process (1732) has encountered a fatal database exception (Microsoft.Isam.Esent.Interop.EsentReadLostFlushVerifyFailureException: The database page read from disk had a previous write not represented on the page

Event 1013 - The mailbox with mailboxguid "915bb7d7-e9ba-4003-9854-166827d82c2e" caused crash or resource outage on database (GUID="ef8c381d-5717-4336-b380-c38f528eab48").

Event 530 - Information Store - Mailbox Database (1732) Mailbox Database: The database page read from the file "d:\Exchange Database\Mailbox Database.edb" at offset 4880728064 (0x0000000122ea0000) (database page 148947 (0x245D3)) for 32768 (0x00008000) bytes failed verification due to a lost flush detection timestamp mismatch. The read operation will fail with error -1119 (0xfffffba1).
tsukraw
Service Provider
 
Posts: 26
Liked: 2 times
Joined: Fri Jul 06, 2012 8:28 pm
Full Name: Tucker Sukraw

Re: Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby jirkava » Wed Sep 09, 2015 6:58 am

Hi, if you have vSphere 6, you can try this KB: http://kb.vmware.com/selfservice/micros ... Id=2114076

JV
jirkava
Lurker
 
Posts: 2
Liked: never
Joined: Fri Oct 21, 2011 1:41 pm
Full Name: Jiri Vana

Re: Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby VladV » Wed Sep 09, 2015 2:47 pm

What Jiri posted refers to not being able to use CBT during backups.

If you say that this happens during committal of the snapshot, are you seeing any VM freeze when this happens? What backup transport protocol are you using?
VladV
Expert
 
Posts: 216
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby Gostev » Thu Sep 10, 2015 12:47 am

Vlad, you are right that this issue results (particularly) in inability to use CBT, however it does result in VM data corruption at snapshot commit as well. Thanks!
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby james411 » Thu Sep 10, 2015 3:02 am

So is this an ESXi 6 issue? I just installed a new 2013 exchange server on a 2012 r2 vm a couple of weeks ago and am using the latest veeam to back it up and it seems to be working fine but now you've got me nervous. I'm on ESXi 5.5 though. Is this a larger issue with veeam and Exchange?
james411
Enthusiast
 
Posts: 69
Liked: 1 time
Joined: Mon Jul 01, 2013 12:13 am

Re: Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby VladV » Thu Sep 10, 2015 5:51 am

Sorry for the misinformation. II quickly read through it and confused it with another bug that affected the use of CBT but because of heap size issues.
VladV
Expert
 
Posts: 216
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby VladV » Thu Sep 10, 2015 6:50 am

Also, didn't know this resulted in VM corruption. We were affected by this bug but haven't experienced any corruption. Lucky us I guess.
VladV
Expert
 
Posts: 216
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby tsukraw » Thu Sep 10, 2015 6:28 pm

We are going to build a lab setup and test this out a bit farther... What we think is happening "this coming from our exchange guy" is that we have circular logging turned on for the DB, but during circular logging there are a hand full of temporary transaction files that are constantly getting created before rolling in. What he thinks is happening is when veeam runs and we have the application aware snapshot turned on, we are getting so many transactions happening so fast "From the PST import" that the application aware snapshot is changing something in the transaction logs that results in a corruption.

This is just a idea and should be fairly easy to test.
Just to confirm something i was mistaken on an item i said earlier. This does NOT just happen on the first backup in our case. We had a previously created a backup of exchange right after it was built but before we had imported any data. The issue came up after we started the import of data and when the backup ran that evening.

One question i had and support could not answer, at least not yet. If you have circular logging turned on for exchange database is it best practice to set application aware processing to "COPY ONLY"? Just thinking about it i think it woudl make sense that you wouldnt want application aware processing to attempt to roll any logs up correct?

For reference case #01021099
tsukraw
Service Provider
 
Posts: 26
Liked: 2 times
Joined: Fri Jul 06, 2012 8:28 pm
Full Name: Tucker Sukraw

Re: Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby james411 » Thu Sep 10, 2015 7:09 pm

tsukraw wrote:One question i had and support could not answer, at least not yet. If you have circular logging turned on for exchange database is it best practice to set application aware processing to "COPY ONLY"? Just thinking about it i think it woudl make sense that you wouldnt want application aware processing to attempt to roll any logs up correct?

Just my $0.02, as I mentioned I just moved to a new 2013 server and I still have circular logging enable, due to moving mailboxes from the old 2007 server, and have not noticed any issues yet. I have Application Aware Processsing turned on.
james411
Enthusiast
 
Posts: 69
Liked: 1 time
Joined: Mon Jul 01, 2013 12:13 am

Re: Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby mfc » Mon Sep 14, 2015 1:57 pm

tsukraw wrote:One question i had and support could not answer, at least not yet. If you have circular logging turned on for exchange database is it best practice to set application aware processing to "COPY ONLY"? Just thinking about it i think it woudl make sense that you wouldnt want application aware processing to attempt to roll any logs up correct?


I don't think it's the cause of your problem but you'd probably want to use Copy Only. If you're in circular logging there's no need for your backup software to truncate the logs.
mfc
Influencer
 
Posts: 12
Liked: never
Joined: Tue Sep 17, 2013 12:48 pm
Full Name: Mike Czajkowski

Re: Veeam Backup Corrupting Exchange 2013 DB

Veeam Logoby edvspanner » Tue Apr 10, 2018 3:17 pm

hi, we have the same issue if we migrate from the old Ex to our new Ex2013. If Veean creates a incremental Backup and we move a Mailbox the Exchnage DB crashs... Any idea?
edvspanner
Novice
 
Posts: 3
Liked: 1 time
Joined: Tue Apr 10, 2018 2:45 pm
Full Name: edvspanner

[MERGED] Veeam destroy Exchange DB

Veeam Logoby edvspanner » Tue Apr 10, 2018 3:26 pm

Hello,

we are migrating from EX2010 to EX2013. If we migrate a Mailbox and run a incremental Backup Job on the new Exchange 2013, veeam destroy the Database.
So we cant migrate to the new Server.

Windows Event Log:

Information Store - 2013_Otto Spanner GmbH (17152) 2013_Otto Spanner GmbH: Datenbank E:\OTTOSPANNERGMBH\OttoSpannerGmbH.edb: Fehler bei Überprüfung von Seite 225042 (0x00036f12) wegen einer Zeitstempel-Nichtübereinstimmung. Der Zeitstempel der letzten Überprüfung im Protokoll war "0x54c35c", aber der tatsächliche Zeitstempel auf der Seite lautete "0x53c0a9". Der Updatezeitstempel nach der Überprüfung, dem der Zeitstempel auf der Seite entsprechen müsste, lautet "0x551424". Wiederherstellung führt zu Fehler -566. Wenn dieser Zustand länger andauert, stellen Sie die Datenbank aus einer früheren Sicherung wieder her. Diesem Problem liegt vermutlich ein Hardwarefehler zugrunde, durch den in der Vergangenheit mindestens eine Leerung dieser Seite verloren gegangen ist. Wenden Sie sich an den Hardwarelieferanten, um Hilfe bei der Problemdiagnose zu erhalten.

any Idea?

thanks

Case # 02754104
edvspanner
Novice
 
Posts: 3
Liked: 1 time
Joined: Tue Apr 10, 2018 2:45 pm
Full Name: edvspanner

Re: Veeam destroy Exchange DB

Veeam Logoby edvspanner » Wed Apr 11, 2018 10:56 am 1 person likes this post

Hi,

after ESXi Upgrade to 6.0 U3 it works !
edvspanner
Novice
 
Posts: 3
Liked: 1 time
Joined: Tue Apr 10, 2018 2:45 pm
Full Name: edvspanner

Re: Veeam destroy Exchange DB

Veeam Logoby DGrinev » Wed Apr 11, 2018 1:13 pm

Hi,

Please can you share the error message in English next time.
Anyway, I'm glad you've nailed it. Thanks!
DGrinev
Veeam Software
 
Posts: 1036
Liked: 109 times
Joined: Thu Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: No registered users and 37 guests