Veeam Backup of Oracle on Windows prior to 10.2.0.3??

[virtualwatts]

Hi,

I'm looking for a needle in a haystack - I'd appreciate a ping if you are it.

If there are any Veeam users who are backing up an Oracle on Windows database pre-10.2.0.3 (e.g. not VSS aware) would you please acknowledge or PM me?

We have virtualized some Ora 9i on Windows 2003EE servers and would really like to commiserate with another Veeam user in like circumstances.

Thanks
Rick

[Gostev]

Hi, you should be able to find a lot of useful information gathered directly from our customers in the following recorded webinar > Running Oracle on VMware. You can also find some oracle on Windows users on this forum by searching for Oracle using forum's search. I think this "passive" way that you have chosen will likely not work, because this topic will simply sink to the 2nd page in less than a day and nobody will see it... plus, your configuration (old Oracle version on Windows) is not very common to say the least. Thanks!

[TrevorBell]

Hi,

I run an Oracle 9i DB for our ERP main system ( IFSWORLD), i do not use Veeam as primary backup, i just use RMAN to make a hot backup of the live DB to another Server (which is a VM ) and that gets backed up with Veeam.
If you need the rman scripts info please let me know. I`m no Oracle expert but got a DBA to write the scripts for us but im willing to share them as its just command line stuff...


Trev

[virtualwatts]

Hi Trev -
Thank you for responding.

How do you plan to recover the underlying dbms server (is that a VM?)

We also have RMAN, and nightly exports for local granular restore. When Veeam 5.x came out I was hoping to actually back up the Oracle vm using Veeam. I've been attempting pre-freeze/post-thaw scripts. The solution, in theory is light weight and allows for quick DR. What I was hoping for was an image restored from Veeam (admittedly probably inconsistent) and then just an auto recovery using RMAN. However, 6 weeks and several service requests later, Dell's VMware support, VMware and Veeam have declared this is a no-go. I am holding out for a final chat with the VMware engineer types but will move on to another solution this week.

I have solid Oracle experience and can backup the dbms - but my challenge is how to recover the system, the dbms server itself. Since this is Windoze a significant part of the Oracle functionality resides in the registry. Being able to clone or restore the VM is half the battle for me as we will ultimately hand-off the process to in-country support.

For now we are using Veeam to backup the VM guest and that is only successful because we are excluding the datastores containing Oracle. That means I can restore the Windows VM but without the drives that contain Oracle. I would then require an entire RMAN restore then recovery.

Does that square with about where you are?

Regards
Rick

[Gostev]

Why no-go? I do not know if you had a chance to view the webinar, but backing up whole VM in crash-consistent state and then just restoring it entriely is what some of our customers have been doing successfully for years.

[tsightler]

It's important to note that a crash consistent backup in this case is actually slightly better than a "typical" crash consistent backup. In a normal "crash" it's possible that data has been written inconsistently to the underlying filesystem or that writes that were in progress could be lost. With a VMware snapshot the system is simply "paused" for an instant, the snapshot is taken of every volume at this exact same instant. This means that all recovery information should be available.

I have been using snapshot based backups of Oracle databases for about a decade, starting with Netapp based filers, but also using EMC and Equallogic storage, and now VMware and Veeam. I have never had an unrecoverable database from a properly designed and configured "crash consistent" backup. Just last week we restored three instances of our ERP system database (a total of about 800GB) using Veeam, all with no issues.

Assuming you know how snapshot backups work, and understand how Oracle logging works, it's not very difficult. That being said, we still use RMAN because of it's much enhanced capabilities. We actually use Veeam and RMAN as complementary technologies, Veeam allows us to quickly restore an entire environment, and if we don't need point-in-time recovery (for example development and test instances many times aren't critical) then we simply restore the VM and we're done, no DBA time needed. If we need a more sophisticated recovery, then we still restore the system with Veeam, and let the DBA either restore incrementals and roll logs, or restore the entire database if that's easier. It works great.

If you want some added safety in your snapshot backups you simply use the pre-freeze/post-thaw to put the database in hot-backup mode prior to the snapshot. This is the same methods that's been used for many decades prior to tools like RMAN and Oracle agents. Put the database in hot-backup, and then backup the filesystem while the database is still running. In hot backup mode the SCN's of the Oracle files are "frozen" (actually, this is a simple explanation that's not 100% correct, but it'll do) so that you can backup the files to tape while the database continues to function.

We've been backing up a 10.2.0.1 database on Windows that supports our PLM system with Veeam for about 2 years and we've preformed several restores of this environment, primarily for creating test environments for upgrades, but also as part of our normal "DR" testing. These restores have always worked without any problem.

[TrevorBell]

I have to say i have a test Oracle 9i DB and i just back it up with veeam and VSS its still working 100% it doesn`t crash or ever fell over, backup always completes i`ll shut it down and bring it back up in INSTANT RECOVERY MODE and see if i can log in and look at the records etc..may need soem help at lookign at the records from a programming guy but will conclude this is a few moments..

trev

[TrevorBell]

Well,

Shutdown the live test, brough the Vm up in INSTANT RECOVERY MODE backup within 3 minutes, logged onto the DB all is good ...programmer ran a query a little slow but i would expect that

[virtualwatts]

Well,

Shutdown the live test, brough the Vm up in INSTANT RECOVERY MODE backup within 3 minutes, logged onto the DB all is good ...programmer ran a query a little slow but i would expect that

Hi Trevor -

That's good to hear. I am about to run some test cycles with your similar settings - the VSS doesn't affect the Oracle instance but should calm the Win2003 OS. It will be interesting as I disable VSC/VSS in Windows out of habit because it is an I/O multiplier.

To confirm -
at Edit Backup Job -> Guest Processing you have checked the "Enable application-aware image processing" checkbox to enable MS VSS?
You don't have VMware quiescence checked
You don't have any pre-freeze/post-thaw scripts on the VM guest

Thanks
Rick

[virtualwatts]

Thanks Tsightler - I've read your other post and appreciate your input, very helpful. I've been working with Oracle since version 4, most of that time was in a *nix environment. I'm familiar with and have used for years EMC snapshots and replication on my Oracle instances. It is not the "concept" of snapshots that concerns me, it is the specific technology implementation. Yes? In this case we have VMware as one layer of tech then Veeam as another layer of tech (of course on top of Oracle and Windows). When your oracle db backup fails to come up who you gonna call?

I launched our architecture design with some preconcieved notions based upon my past - some have worked and some are just inappropriate in a VM or Windows environment. Therefore - I'm constantly probing and questioning.

In point of fact - and very dissapointing of VMware - our pre-freeze scripts were erroring out. Veeam of course is just calling the API so they passed it as a VMware issue. After two con calls yesterday with VMware support their conclusion was (wait for it) using snapshots is not a recommended practice for backing up a production environment. Like I said, don't think they got the point of the snapshot. More misleading than helpful.

I don't actually know that all I/O is stopped waiting for the snapshot to occur in VMware. There is as much mis-information as information in this space right now. Between me and the techs we have Ora & Windows covered, but the VMware is a recent addition for us and as we've learned over the past 7 months it introduces variables that even befuddle the vendors. If I had a dime for every time a tech support has said "well it shouldn't be doing that..." during our data center virtualization, well, I'd have more than a dollar.

Since we have multiple data stores housing our Oracle files and since we have seen the snapshot creation/merge take several minutes and since Veeam doesn't snapshot memory - I'm just feeling my way through this. Think about it - some unlucky DBA somewhere had to be the first to find out the VMware sync driver corrupts your db. Well, we're at a new ESXi, new bare metal platform and new backup tool release.

It helps to hear from peers who have walked this path already - but when you're in a tiny sub-sub-category of Oracle on Windows it isn't easy to locate them. I welcome your insight. Thanks to you and Trevor for responding.


Rick

[Gostev]

When your oracle db backup fails to come up who you gonna call?

This is exactly why we added SureBackup. Each of your backups will come up if needed, because actual recoverability of every backup you make is being tested. If your pre/post scripts fail, then just backup it as crash-consistent, like Tom does. Chance of "bad" crash-consistent backup is very low anyway, and with SureBackup you will know exactly if your backup is good or not.

[tsightler]

When your oracle db backup fails to come up who you gonna call?

Well, that's why you have RMAN in addition to Veeam. There are many reasons to run RMAN, for granular restores, log backups, easy cloning for DEV/TEST environments, the fact that RMAN backups check for silent block level corruption, etc. My point is simply that we've restored backups from Veeam probably 30 times, maybe more if you count all of our testing, and we've never had an Oracle instance fail to start.

I don't actually know that all I/O is stopped waiting for the snapshot to occur in VMware. There is as much mis-information as information in this space right now. Between me and the techs we have Ora & Windows covered, but the VMware is a recent addition for us and as we've learned over the past 7 months it introduces variables that even befuddle the vendors. If I had a dime for every time a tech support has said "well it shouldn't be doing that..." during our data center virtualization, well, I'd have more than a dollar.

Did I say all I/O is stopped? I don't think so. I said the system is "paused", just long enough for the snapshot to be created. If there are 100 outstanding writes, it's possible the snapshot is made directly in the middle of them, but there is no chance for a partial write of a block because it's not a crash, it's a snapshot. That means there's no chance for a "lost" write (in a physical server "crash" that's why caches have to have battery backup, so as not to loose writes).

Since we have multiple data stores housing our Oracle files and since we have seen the snapshot creation/merge take several minutes and since Veeam doesn't snapshot memory - I'm just feeling my way through this. Think about it - some unlucky DBA somewhere had to be the first to find out the VMware sync driver corrupts your db. Well, we're at a new ESXi, new bare metal platform and new backup tool release.

I understand your questions. Heck, we've been a virtual shop long before virtualization was "cool" (ESX 2.x days, I even did some early stuff with ESX 1.x). If you think you see strange answers from vendors today, you should have mentioned it to them back in the first part of the decade, most of them didn't even know what virtualization was.

[TrevorBell]

Correct,

(its actually Veeams VSS )It`s enabled on all my vm backups its good practice..

Have a go and see what results you get. ( test db )

Hi Trevor -

That's good to hear. I am about to run some test cycles with your similar settings - the VSS doesn't affect the Oracle instance but should calm the Win2003 OS. It will be interesting as I disable VSC/VSS in Windows out of habit because it is an I/O multiplier.

To confirm -
at Edit Backup Job -> Guest Processing you have checked the "Enable application-aware image processing" checkbox to enable MS VSS?
You don't have VMware quiescence checked
You don't have any pre-freeze/post-thaw scripts on the VM guest

Thanks
Rick

[virtualwatts]

Correct,

(its actually Veeams VSS )It`s enabled on all my vm backups its good practice..

Okay - but again, the Veeam app and dialogue words aren't always clear or consistent, my check box specifically states
"Quiesces applications inside VM using Microsoft VSS", that's why I'm being so particular. When someone on the forum says "use Veeam's VSS" I have yet to find that reference in the app panels.
If it's Veem's VSS but using the words "Microsoft's VSS" then I wanted to get that out there so I'm not flipping the wrong switches or missing a config setting.

Thanks for the confirmation
Rick

[Gostev]

"Veeam VSS" is old name of the feature, you will see it mentioned by those who has been using Veeam for a while now. "Application-aware image processing" feature in v5 includes functionality formerly known as "Veeam VSS", so this is the checkbox you want to select in v5.

[TrevorBell]

i stand corrected by Anton... must be an old timer

[virtualwatts]

Howdy, As a quick follow-up.

We changed over to crash consistent backups of our Oracle dbms VMs per this thread.
We have Selected Veeam VSS (AppAware Image Processing), Unselected quiescence option, Removed pre-freeze/post-thaw

The backups are successful. We are performing full restores to test the approach. The Veeam restores are successful -
However, the newly restored servers will not boot. The message is "Error Loading Operating System".


I expected the Oracle db to have to recover, but didn't expect the OS to fail.
The exact same settings for non-Oracle VM guest restore then boot just fine.

I've opened a ticket with Veeam but wanted to add our test results to this thread.

[tsightler]

That pretty much has to be something to do with your VM setup. Have you verified that the restored VM has the same disk mapping as the original VM, i.e. same disk images mapped to the same SCSI ID's. Any chance that the SCSI adapter version is not being set correctly? I'll be interested to here what support says.

[virtualwatts]

That pretty much has to be something to do with your VM setup. Have you verified that the restored VM has the same disk mapping as the original VM, i.e. same disk images mapped to the same SCSI ID's. Any chance that the SCSI adapter version is not being set correctly? I'll be interested to here what support says.

I briefly opened up the vmx file and the vmdk file and all of the pointers appear to be correct for the vmfs files. Last time I had this error in Veeam it was because the VMX was pointing to a vmdk that was referencing a snapshot that wasn't there. I'll spend more time later to see exactly what is the problem - I have confirmed with further testing that none of our Oracle dbms VMs will restore.

If it is the VM setup then why only the Oracle db VM and not the other VMs in the cluster?

Rick

[virtualwatts]

Well, okay, the CID in the vmdk does look odd "ffffffffffe" but I'll wait for the Veeam supt response.

[tsightler]

If it is the VM setup then why only the Oracle db VM and not the other VMs in the cluster?

Since I don't know anything about your environment, I don't know how to answer that, but I would suspect that perhaps those VM's were created using a different template or different method than the others in your infrastructure. I know in our environment most VM's were created either from clean VM templates, but some were P2V'd from various hardware along the way, and we've seen some pretty strange behavior from systems that were P2V'd, especially those that were P2V'd using early versions of the product.

Not only that, but Oracle simply doesn't have anything to do with the OS booting process so it doesn't seem plausible that Oracle itself would be the cause. Note that I'm not saying anything is wrong with the way the VM's are configured, only that there almost had to be something unique about them that Veeam is not handling correctly. At it's heart, Veeam is a very simple product, it takes a snapshot, and copies the virtual disks. From a disk perspective, restoring a Veeam backup should be no different than reverting a VMware snapshot, but, when restoring an entire VM, Veeam does create a new VMX file, and I've seen a couple of cases where Veeam did not handle some unique corner case for VMX file creation.

[virtualwatts]

Agreed -

All of our VMs are created from a standard template using the same method, it was a tenet of our architecture to get some consistency in the environment.
We have no P2Vs because of the condition of the original systems and the chance for oddities.
The main difference is that these dbms VMs are backed up with Veeam VSS selected whereas the other VMs are not.
I am able to clone these VMs (with Oracle running) just fine. It will be interesting to see where this comes out.

[virtualwatts]

So Veeam VSS isn't a factor. Testing confirms the restore of a quiesced (no pre-freeze) backup also is unable to boot.

[tsightler]

Yes, please keep us updated. I know I've seen this issue reported before on the forum but I can't remember the cause in that case. Definitely no RDM's involved in this case is there?

[barresi]

[merged]

Hello,

how can we backup an Windows 2003 VM with Oracle9 DB whit Veeam

Can we use Microsoft VSS (quiesing VM check) and we have an application consistant vm backup? What is the veeam best practices for this?

Kind regards Matthias

[omorin]

[merged]

Bonjour,

I have a Windows 2003 VM with an oracle 9 database engine, not compatible with VSS.
Is it safe or not to enable vmware tools quiescence with a database running in production during backup or replication with Vbr 5.