Oracle DB corruption with ESXi 6.0 after backup

[vmrsk]

Hi,

we encountered Oracle DB corruption with ESXi 6.0 and Veeam B&R 8.0 patch 2.
There were no problems with ESXi 5.5U2 and Veeam B&R 8.0 patch 1.
OS with Oracle is CentOS and Windows.
Mutiple Oracle configurations are in use, also with RAC, ASM.
VMs are backed up with AAIP (Windows) and VMware Tools quiesceing (CentOS). No scripts.

There is no problem or warning logged during backup on Veeam or VMware side.
But Oracle guys find DBs critically damaged after backup - unrecoverably.
Restore and recovery from this backup is however possible.
So I suspect snapshot removal after backup kills DBs (errors arise 1-2 minutes after snapshot removal).

Anybody running successfull backup of Oracle VMs with ESXi 6.0 and Veeam B&R 8.0 patch 2?


P.S. this is what Oracle guy sees on all corrupted VMs:
********************* ATTENTION: ********************
The controlfile header block returned by the OS
has a sequence number that is too old.
The controlfile might be corrupted.
PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE
without following the steps below.
RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE
TO THE DATABASE, if the controlfile is truly corrupted.
In order to re-start the instance safely,
please do the following:
(1) Save all copies of the controlfile for later
analysis and contact your OS vendor and Oracle support.
(2) Mount the instance and issue:
ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
(3) Unmount the instance.
(4) Use the script in the trace file to
RE-CREATE THE CONTROLFILE and open the database.


Kind regards,
Richard

[foggy]

Richard, does making snapshot of Oracle VMs manually via vSphere Client result in similar errors?

[gotMyBackup]

I'm assuming you're fully aware of http://kb.vmware.com/selfservice/micros ... Id=2114076 ?
We've seen corrupt SQL DB's since upgrades to ESXi 6.0 and change block tracking backups via CommVault. Not sure if there is any relation but worth looking at.

[vmrsk]

Hi,

I'm not allowed to do any tests right now.
But storage vmotion worked with "problematic" VM without problems (there was snapshot creation and removal).
Also looks like Oracle DBs without RAC are ok after backup (and snapshot removal).

Thank you
Richard

[joergr]

Any news on this one?

I really don´t want to frighten anybody but the logs provided by OP *could* suggest that *maybe* a snapshot rollback instead of a snapshot commit was performed by ESXi 6 on the affected vm with a VADP-triggered snapshot. If that would be true it would be disastrous. But this is extremely unlikely - I can´t imaginge a bug of this dimension could slip through VMware QC. We really do need more intel here. So please - OP - could you give us an update here?

Best regards,
Joerg

[LorettoCNY]

I recently upgraded to esxi 6 and have suffered corruption on multiple SQL servers, exchange, and active directory. No solution from VMware yet. I know it's not veeam as we just purchased it and we're using baxkup exec after migrating to 6.0

[Knaecke1]

Same Problem here. We upgraded last week to ESXi 6.0. The last 3 days were horrible. First we had a corrupted Exchange DB, then a corrupted Pervasive DB and today out AD ist damaged. Has anybody a solution or new information?

[joergr]

Please double check if you have this update installed http://kb.vmware.com/selfservice/micros ... Id=2116125

[khancock@sau81.org]

We've been experiencing corrupt Exchange DBs on ESXi 6.0 the last several days. We've deployed the patch mentioned above but I'm curious if anyone has any feedback as to if that solved their problems or not...?

[joergr]

Hi,

did you involve VMware support on this? It would be extremely interesting to know what they have to say about it, and/or if they found something which could relate to an esxi 6 issue. If not, i would like to highly encourage you to open a support ticket with vmware and have it checked out by them.

Best regards,
Joerg

[PariPari]

Any news on this?
We have a corrupt exchange DB after replication from ESX 4.1 to a new ESX 6.0 environment (just installed recently, so latest patches for every system).

The did a full replica with offline server on 3 days ago + two incremental replica (offline Server). The last replica occurred yesterday and the server came up with a corrupt .edb
Could this be a CTB issue on ESX 4.1 side?
I could open a ticket but as far as i have seen there is no related error in the replica logs.

[PTide]

Hi,

How about your backup job? Does it go smoothly for that server?

Thank you.