Comprehensive data protection for all workloads
Post Reply
vmrsk
Novice
Posts: 3
Liked: 1 time
Joined: May 06, 2015 3:14 pm
Contact:

Oracle DB corruption with ESXi 6.0 after backup

Post by vmrsk » May 06, 2015 3:28 pm

Hi,

we encountered Oracle DB corruption with ESXi 6.0 and Veeam B&R 8.0 patch 2.
There were no problems with ESXi 5.5U2 and Veeam B&R 8.0 patch 1.
OS with Oracle is CentOS and Windows.
Mutiple Oracle configurations are in use, also with RAC, ASM.
VMs are backed up with AAIP (Windows) and VMware Tools quiesceing (CentOS). No scripts.

There is no problem or warning logged during backup on Veeam or VMware side.
But Oracle guys find DBs critically damaged after backup - unrecoverably.
Restore and recovery from this backup is however possible.
So I suspect snapshot removal after backup kills DBs (errors arise 1-2 minutes after snapshot removal).

Anybody running successfull backup of Oracle VMs with ESXi 6.0 and Veeam B&R 8.0 patch 2?


P.S. this is what Oracle guy sees on all corrupted VMs:
********************* ATTENTION: ********************
The controlfile header block returned by the OS
has a sequence number that is too old.
The controlfile might be corrupted.
PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE
without following the steps below.
RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE
TO THE DATABASE, if the controlfile is truly corrupted.
In order to re-start the instance safely,
please do the following:
(1) Save all copies of the controlfile for later
analysis and contact your OS vendor and Oracle support.
(2) Mount the instance and issue:
ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
(3) Unmount the instance.
(4) Use the script in the trace file to
RE-CREATE THE CONTROLFILE and open the database.


Kind regards,
Richard

foggy
Veeam Software
Posts: 17725
Liked: 1483 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by foggy » May 06, 2015 4:28 pm

Richard, does making snapshot of Oracle VMs manually via vSphere Client result in similar errors?

gotMyBackup
Lurker
Posts: 1
Liked: never
Joined: May 07, 2015 4:51 am
Full Name: S A
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by gotMyBackup » May 07, 2015 4:54 am

I'm assuming you're fully aware of http://kb.vmware.com/selfservice/micros ... Id=2114076 ?
We've seen corrupt SQL DB's since upgrades to ESXi 6.0 and change block tracking backups via CommVault. Not sure if there is any relation but worth looking at.

vmrsk
Novice
Posts: 3
Liked: 1 time
Joined: May 06, 2015 3:14 pm
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by vmrsk » May 11, 2015 8:11 am

Hi,

I'm not allowed to do any tests right now.
But storage vmotion worked with "problematic" VM without problems (there was snapshot creation and removal).
Also looks like Oracle DBs without RAC are ok after backup (and snapshot removal).

Thank you
Richard

joergr
Expert
Posts: 386
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by joergr » May 11, 2015 8:14 am

Any news on this one?

I really don´t want to frighten anybody but the logs provided by OP *could* suggest that *maybe* a snapshot rollback instead of a snapshot commit was performed by ESXi 6 on the affected vm with a VADP-triggered snapshot. If that would be true it would be disastrous. But this is extremely unlikely - I can´t imaginge a bug of this dimension could slip through VMware QC. We really do need more intel here. So please - OP - could you give us an update here?

Best regards,
Joerg

LorettoCNY
Novice
Posts: 8
Liked: 3 times
Joined: May 12, 2015 12:56 pm
Full Name: Brandon
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by LorettoCNY » May 13, 2015 3:21 am 1 person likes this post

I recently upgraded to esxi 6 and have suffered corruption on multiple SQL servers, exchange, and active directory. No solution from VMware yet. I know it's not veeam as we just purchased it and we're using baxkup exec after migrating to 6.0

Knaecke1
Lurker
Posts: 1
Liked: never
Joined: Jun 03, 2015 8:20 am
Full Name: Thomas Huttinger
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by Knaecke1 » Jun 03, 2015 8:33 am

Same Problem here. We upgraded last week to ESXi 6.0. The last 3 days were horrible. First we had a corrupted Exchange DB, then a corrupted Pervasive DB and today out AD ist damaged. Has anybody a solution or new information?

joergr
Expert
Posts: 386
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by joergr » Jun 03, 2015 12:43 pm

Please double check if you have this update installed http://kb.vmware.com/selfservice/micros ... Id=2116125

khancock@sau81.org
Lurker
Posts: 1
Liked: never
Joined: Jul 02, 2015 5:28 pm
Full Name: Kyle Hancock
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by khancock@sau81.org » Jul 02, 2015 5:30 pm

We've been experiencing corrupt Exchange DBs on ESXi 6.0 the last several days. We've deployed the patch mentioned above but I'm curious if anyone has any feedback as to if that solved their problems or not...?

joergr
Expert
Posts: 386
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by joergr » Jul 03, 2015 11:38 am

Hi,

did you involve VMware support on this? It would be extremely interesting to know what they have to say about it, and/or if they found something which could relate to an esxi 6 issue. If not, i would like to highly encourage you to open a support ticket with vmware and have it checked out by them.

Best regards,
Joerg

PariPari
Service Provider
Posts: 88
Liked: 9 times
Joined: Nov 26, 2014 2:37 pm
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by PariPari » Nov 05, 2015 1:14 pm

Any news on this?
We have a corrupt exchange DB after replication from ESX 4.1 to a new ESX 6.0 environment (just installed recently, so latest patches for every system).

The did a full replica with offline server on 3 days ago + two incremental replica (offline Server). The last replica occurred yesterday and the server came up with a corrupt .edb
Could this be a CTB issue on ESX 4.1 side?
I could open a ticket but as far as i have seen there is no related error in the replica logs.

P.Tide
Product Manager
Posts: 4960
Liked: 419 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Oracle DB corruption with ESXi 6.0 after backup

Post by P.Tide » Nov 05, 2015 2:04 pm

Hi,

How about your backup job? Does it go smoothly for that server?

Thank you.

Post Reply

Who is online

Users browsing this forum: FredrikJ, Karinne, LDuffy, wishr and 43 guests