Comprehensive data protection for all workloads
Post Reply
tsukraw
Enthusiast
Posts: 46
Liked: 2 times
Joined: Jul 06, 2012 8:28 pm
Full Name: Tucker Sukraw
Contact:

Veeam Backup Corrupting Exchange 2013 DB

Post by tsukraw »

Hey Guys,
Just throwing this out there to see if anyone else has seen a similar issue.
In the past few weeks we have now seen this happen 3 separate times.

In 3 separate cases we were doing a new server builds for customers.
New 2012.R2 servers built with Exchange 2013 CU9.
All 3 cases are on Veeam v8 build 2018 and build 2030

In all 3 cases we had imported mail from PST files into the exchange server which i don't think is a factor but is common between all 3.

What we found in comparing time stamps in event viewer and the backup logs is when the very first Veeam Backup of the exchange server ran when it got to removing the snapshot and completing the task the exchange DB would dismount and from that point on we could not remount it. The event logs listed below are identical between all 3 separate cases.

In two of the cases we opened tickets with Microsoft and since we had no "good" backup yet they both required a rebuild from scratch due to the database being corrupted. We had not opened a ticket with Veeam since the backup job appeared to have ran just fine. No issues were reported so we did not suspect it.
The third one happened here tonight and since we were in the building state we will most likely just rebuild it as well, but will be calling Veeam in the AM to see what they think of this situation will post case ID at that time.

We just find it extremely strange that we have had 3 instances of this happen. And besides the type of setup being the same the only common factor is it happens at the exact same time the first backup is completely.

Anyone ever seen a issue like this?

Here are the event logs that are posted at the same time the job is completing. By the way the events sound it is almost like the exchange DB is not accessible for a bit when it is trying to write data to it.

Event 2008 - Microsoft Exchange Information Store worker process (1732) has encountered a fatal database exception (Microsoft.Isam.Esent.Interop.EsentReadLostFlushVerifyFailureException: The database page read from disk had a previous write not represented on the page

Event 1013 - The mailbox with mailboxguid "915bb7d7-e9ba-4003-9854-166827d82c2e" caused crash or resource outage on database (GUID="ef8c381d-5717-4336-b380-c38f528eab48").

Event 530 - Information Store - Mailbox Database (1732) Mailbox Database: The database page read from the file "d:\Exchange Database\Mailbox Database.edb" at offset 4880728064 (0x0000000122ea0000) (database page 148947 (0x245D3)) for 32768 (0x00008000) bytes failed verification due to a lost flush detection timestamp mismatch. The read operation will fail with error -1119 (0xfffffba1).
Tucker Sukraw
Network Architect
West Central Technology
jirkava
Novice
Posts: 4
Liked: never
Joined: Oct 21, 2011 1:41 pm
Full Name: Jiri Vana
Contact:

Re: Veeam Backup Corrupting Exchange 2013 DB

Post by jirkava »

Hi, if you have vSphere 6, you can try this KB: http://kb.vmware.com/selfservice/micros ... Id=2114076

JV
VladV
Expert
Posts: 224
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Veeam Backup Corrupting Exchange 2013 DB

Post by VladV »

What Jiri posted refers to not being able to use CBT during backups.

If you say that this happens during committal of the snapshot, are you seeing any VM freeze when this happens? What backup transport protocol are you using?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Backup Corrupting Exchange 2013 DB

Post by Gostev »

Vlad, you are right that this issue results (particularly) in inability to use CBT, however it does result in VM data corruption at snapshot commit as well. Thanks!
james411
Enthusiast
Posts: 69
Liked: 1 time
Joined: Jul 01, 2013 12:13 am
Contact:

Re: Veeam Backup Corrupting Exchange 2013 DB

Post by james411 »

So is this an ESXi 6 issue? I just installed a new 2013 exchange server on a 2012 r2 vm a couple of weeks ago and am using the latest veeam to back it up and it seems to be working fine but now you've got me nervous. I'm on ESXi 5.5 though. Is this a larger issue with veeam and Exchange?
VladV
Expert
Posts: 224
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Veeam Backup Corrupting Exchange 2013 DB

Post by VladV »

Sorry for the misinformation. II quickly read through it and confused it with another bug that affected the use of CBT but because of heap size issues.
VladV
Expert
Posts: 224
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Veeam Backup Corrupting Exchange 2013 DB

Post by VladV »

Also, didn't know this resulted in VM corruption. We were affected by this bug but haven't experienced any corruption. Lucky us I guess.
tsukraw
Enthusiast
Posts: 46
Liked: 2 times
Joined: Jul 06, 2012 8:28 pm
Full Name: Tucker Sukraw
Contact:

Re: Veeam Backup Corrupting Exchange 2013 DB

Post by tsukraw »

We are going to build a lab setup and test this out a bit farther... What we think is happening "this coming from our exchange guy" is that we have circular logging turned on for the DB, but during circular logging there are a hand full of temporary transaction files that are constantly getting created before rolling in. What he thinks is happening is when veeam runs and we have the application aware snapshot turned on, we are getting so many transactions happening so fast "From the PST import" that the application aware snapshot is changing something in the transaction logs that results in a corruption.

This is just a idea and should be fairly easy to test.
Just to confirm something i was mistaken on an item i said earlier. This does NOT just happen on the first backup in our case. We had a previously created a backup of exchange right after it was built but before we had imported any data. The issue came up after we started the import of data and when the backup ran that evening.

One question i had and support could not answer, at least not yet. If you have circular logging turned on for exchange database is it best practice to set application aware processing to "COPY ONLY"? Just thinking about it i think it woudl make sense that you wouldnt want application aware processing to attempt to roll any logs up correct?

For reference case #01021099
Tucker Sukraw
Network Architect
West Central Technology
james411
Enthusiast
Posts: 69
Liked: 1 time
Joined: Jul 01, 2013 12:13 am
Contact:

Re: Veeam Backup Corrupting Exchange 2013 DB

Post by james411 »

tsukraw wrote: One question i had and support could not answer, at least not yet. If you have circular logging turned on for exchange database is it best practice to set application aware processing to "COPY ONLY"? Just thinking about it i think it woudl make sense that you wouldnt want application aware processing to attempt to roll any logs up correct?
Just my $0.02, as I mentioned I just moved to a new 2013 server and I still have circular logging enable, due to moving mailboxes from the old 2007 server, and have not noticed any issues yet. I have Application Aware Processsing turned on.
mfc
Enthusiast
Posts: 26
Liked: 3 times
Joined: Sep 17, 2013 12:48 pm
Full Name: Mike Czajkowski
Contact:

Re: Veeam Backup Corrupting Exchange 2013 DB

Post by mfc »

tsukraw wrote:One question i had and support could not answer, at least not yet. If you have circular logging turned on for exchange database is it best practice to set application aware processing to "COPY ONLY"? Just thinking about it i think it woudl make sense that you wouldnt want application aware processing to attempt to roll any logs up correct?
I don't think it's the cause of your problem but you'd probably want to use Copy Only. If you're in circular logging there's no need for your backup software to truncate the logs.
edvspanner
Novice
Posts: 3
Liked: 1 time
Joined: Apr 10, 2018 2:45 pm
Full Name: edvspanner
Contact:

Re: Veeam Backup Corrupting Exchange 2013 DB

Post by edvspanner »

hi, we have the same issue if we migrate from the old Ex to our new Ex2013. If Veean creates a incremental Backup and we move a Mailbox the Exchnage DB crashs... Any idea?
edvspanner
Novice
Posts: 3
Liked: 1 time
Joined: Apr 10, 2018 2:45 pm
Full Name: edvspanner
Contact:

[MERGED] Veeam destroy Exchange DB

Post by edvspanner »

Hello,

we are migrating from EX2010 to EX2013. If we migrate a Mailbox and run a incremental Backup Job on the new Exchange 2013, veeam destroy the Database.
So we cant migrate to the new Server.

Windows Event Log:

Information Store - 2013_Otto Spanner GmbH (17152) 2013_Otto Spanner GmbH: Datenbank E:\OTTOSPANNERGMBH\OttoSpannerGmbH.edb: Fehler bei Überprüfung von Seite 225042 (0x00036f12) wegen einer Zeitstempel-Nichtübereinstimmung. Der Zeitstempel der letzten Überprüfung im Protokoll war "0x54c35c", aber der tatsächliche Zeitstempel auf der Seite lautete "0x53c0a9". Der Updatezeitstempel nach der Überprüfung, dem der Zeitstempel auf der Seite entsprechen müsste, lautet "0x551424". Wiederherstellung führt zu Fehler -566. Wenn dieser Zustand länger andauert, stellen Sie die Datenbank aus einer früheren Sicherung wieder her. Diesem Problem liegt vermutlich ein Hardwarefehler zugrunde, durch den in der Vergangenheit mindestens eine Leerung dieser Seite verloren gegangen ist. Wenden Sie sich an den Hardwarelieferanten, um Hilfe bei der Problemdiagnose zu erhalten.

any Idea?

thanks

Case # 02754104
edvspanner
Novice
Posts: 3
Liked: 1 time
Joined: Apr 10, 2018 2:45 pm
Full Name: edvspanner
Contact:

Re: Veeam destroy Exchange DB

Post by edvspanner » 1 person likes this post

Hi,

after ESXi Upgrade to 6.0 U3 it works !
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Veeam destroy Exchange DB

Post by DGrinev »

Hi,

Please can you share the error message in English next time.
Anyway, I'm glad you've nailed it. Thanks!
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 241 guests