Veeam Backup Corrupting Exchange 2013 DB

[tsukraw]

Hey Guys,
Just throwing this out there to see if anyone else has seen a similar issue.
In the past few weeks we have now seen this happen 3 separate times.

In 3 separate cases we were doing a new server builds for customers.
New 2012.R2 servers built with Exchange 2013 CU9.
All 3 cases are on Veeam v8 build 2018 and build 2030

In all 3 cases we had imported mail from PST files into the exchange server which i don't think is a factor but is common between all 3.

What we found in comparing time stamps in event viewer and the backup logs is when the very first Veeam Backup of the exchange server ran when it got to removing the snapshot and completing the task the exchange DB would dismount and from that point on we could not remount it. The event logs listed below are identical between all 3 separate cases.

In two of the cases we opened tickets with Microsoft and since we had no "good" backup yet they both required a rebuild from scratch due to the database being corrupted. We had not opened a ticket with Veeam since the backup job appeared to have ran just fine. No issues were reported so we did not suspect it.
The third one happened here tonight and since we were in the building state we will most likely just rebuild it as well, but will be calling Veeam in the AM to see what they think of this situation will post case ID at that time.

We just find it extremely strange that we have had 3 instances of this happen. And besides the type of setup being the same the only common factor is it happens at the exact same time the first backup is completely.

Anyone ever seen a issue like this?

Here are the event logs that are posted at the same time the job is completing. By the way the events sound it is almost like the exchange DB is not accessible for a bit when it is trying to write data to it.

Event 2008 - Microsoft Exchange Information Store worker process (1732) has encountered a fatal database exception (Microsoft.Isam.Esent.Interop.EsentReadLostFlushVerifyFailureException: The database page read from disk had a previous write not represented on the page

Event 1013 - The mailbox with mailboxguid "915bb7d7-e9ba-4003-9854-166827d82c2e" caused crash or resource outage on database (GUID="ef8c381d-5717-4336-b380-c38f528eab48").

Event 530 - Information Store - Mailbox Database (1732) Mailbox Database: The database page read from the file "d:\Exchange Database\Mailbox Database.edb" at offset 4880728064 (0x0000000122ea0000) (database page 148947 (0x245D3)) for 32768 (0x00008000) bytes failed verification due to a lost flush detection timestamp mismatch. The read operation will fail with error -1119 (0xfffffba1).

[jirkava]

Hi, if you have vSphere 6, you can try this KB: http://kb.vmware.com/selfservice/micros ... Id=2114076

JV

[VladV]

What Jiri posted refers to not being able to use CBT during backups.

If you say that this happens during committal of the snapshot, are you seeing any VM freeze when this happens? What backup transport protocol are you using?

[Gostev]

Vlad, you are right that this issue results (particularly) in inability to use CBT, however it does result in VM data corruption at snapshot commit as well. Thanks!

[james411]

So is this an ESXi 6 issue? I just installed a new 2013 exchange server on a 2012 r2 vm a couple of weeks ago and am using the latest veeam to back it up and it seems to be working fine but now you've got me nervous. I'm on ESXi 5.5 though. Is this a larger issue with veeam and Exchange?

[VladV]

Sorry for the misinformation. II quickly read through it and confused it with another bug that affected the use of CBT but because of heap size issues.

[VladV]

Also, didn't know this resulted in VM corruption. We were affected by this bug but haven't experienced any corruption. Lucky us I guess.

[tsukraw]

We are going to build a lab setup and test this out a bit farther... What we think is happening "this coming from our exchange guy" is that we have circular logging turned on for the DB, but during circular logging there are a hand full of temporary transaction files that are constantly getting created before rolling in. What he thinks is happening is when veeam runs and we have the application aware snapshot turned on, we are getting so many transactions happening so fast "From the PST import" that the application aware snapshot is changing something in the transaction logs that results in a corruption.

This is just a idea and should be fairly easy to test.
Just to confirm something i was mistaken on an item i said earlier. This does NOT just happen on the first backup in our case. We had a previously created a backup of exchange right after it was built but before we had imported any data. The issue came up after we started the import of data and when the backup ran that evening.

One question i had and support could not answer, at least not yet. If you have circular logging turned on for exchange database is it best practice to set application aware processing to "COPY ONLY"? Just thinking about it i think it woudl make sense that you wouldnt want application aware processing to attempt to roll any logs up correct?

For reference case #01021099

[james411]

One question i had and support could not answer, at least not yet. If you have circular logging turned on for exchange database is it best practice to set application aware processing to "COPY ONLY"? Just thinking about it i think it woudl make sense that you wouldnt want application aware processing to attempt to roll any logs up correct?

Just my $0.02, as I mentioned I just moved to a new 2013 server and I still have circular logging enable, due to moving mailboxes from the old 2007 server, and have not noticed any issues yet. I have Application Aware Processsing turned on.

[mfc]

One question i had and support could not answer, at least not yet. If you have circular logging turned on for exchange database is it best practice to set application aware processing to "COPY ONLY"? Just thinking about it i think it woudl make sense that you wouldnt want application aware processing to attempt to roll any logs up correct?

I don't think it's the cause of your problem but you'd probably want to use Copy Only. If you're in circular logging there's no need for your backup software to truncate the logs.

[edvspanner]

hi, we have the same issue if we migrate from the old Ex to our new Ex2013. If Veean creates a incremental Backup and we move a Mailbox the Exchnage DB crashs... Any idea?

[edvspanner]

Hello,

we are migrating from EX2010 to EX2013. If we migrate a Mailbox and run a incremental Backup Job on the new Exchange 2013, veeam destroy the Database.
So we cant migrate to the new Server.

Windows Event Log:

Information Store - 2013_Otto Spanner GmbH (17152) 2013_Otto Spanner GmbH: Datenbank E:\OTTOSPANNERGMBH\OttoSpannerGmbH.edb: Fehler bei Überprüfung von Seite 225042 (0x00036f12) wegen einer Zeitstempel-Nichtübereinstimmung. Der Zeitstempel der letzten Überprüfung im Protokoll war "0x54c35c", aber der tatsächliche Zeitstempel auf der Seite lautete "0x53c0a9". Der Updatezeitstempel nach der Überprüfung, dem der Zeitstempel auf der Seite entsprechen müsste, lautet "0x551424". Wiederherstellung führt zu Fehler -566. Wenn dieser Zustand länger andauert, stellen Sie die Datenbank aus einer früheren Sicherung wieder her. Diesem Problem liegt vermutlich ein Hardwarefehler zugrunde, durch den in der Vergangenheit mindestens eine Leerung dieser Seite verloren gegangen ist. Wenden Sie sich an den Hardwarelieferanten, um Hilfe bei der Problemdiagnose zu erhalten.

any Idea?

thanks

Case # 02754104

[edvspanner]

Hi,

after ESXi Upgrade to 6.0 U3 it works !

[DGrinev]

Hi,

Please can you share the error message in English next time.
Anyway, I'm glad you've nailed it. Thanks!