Does Veeam 9 stop and start ORACLE instances?

[KOPFteam]

In Veeam 8 I got the warning

27.01.2016 12:25:35 :: Excluded Oracle databases: Database AM83/AM83.WORLD'; Database BP71/BP71.WORLD'; Database Test83/TEST83.WORLD'; Database Test71/TEST71.WORLD';

on each run of the job.

In Veeam 9 I don't get that warning, but when I look at the eventlog on our ORACLE server (Windows 2008 R2; ORACLE 11g) all instances were stopped and restarted 2 minutes later.

After that our ERP system was not working anymore. I had to restart the whole server during production hours!

How can I avoid that? I have disabled the job now.

Thanks,
Florian Pürner
KOPFteam GmbH

[foggy]

Oracle databases processing was significantly improved in v9 to provide transactionally-consistent backup of Oracle servers and ability to recover database to any point in time (even to a state before a particular transaction). Previously, databases that did not meet the requirements for successful processing were automatically excluded from application-aware processing. Now Veeam B&R is able to correctly process databases and their transaction logs (using VSS or RMAN, according to Oracle recommendations), which in some cases could require stopping and re-starting instances back.

That said, the fact that you had to restart the server after the backup is unexpected, so I recommend contacting support for investigation.

[KOPFteam]

I'm very happy about the enhanced posibilities of the ORACLE backup, but I think I will have to change my replication job for the ORACLE server. Now I've scheduled it to run every 4 hours.

The ORACLE instances were down for about 2 minutes. This had 2 consequences:

The users couldn't work for 2 minutes. Not good.

The ERP software realized that its ORACLE instances were not available and stopped a few services. That's why the ERP software wasn't fully running after the ORACLE instances were back. To recover from this situation I choose the easy way: Restart server during production hours. Not good.

Do you have any recommendations for me?

[KOPFteam]

I opened a support case. The ID is 01683017

[KOPFteam]

Any idea what I could do (maybe on the Oracle server) to stop Veeam from ending the Oracle instances? Oracle is not running in archivelogmode.

[Gostev]

We are adding a registry key for this in Update 1. Sorry for the inconvenience!

[foggy]

Currently you can disable application-aware image processing for these VMs in advanced guest processing settings in the job and use VMware Tools Quiescence instead.

[KOPFteam]

Thanks for your replies!
I just changed the job settings und manually started it. I'll let you know what hapend when it finishes.
Looking forward to Update 1...

[KOPFteam]

No. didn't work:
29.01.2016 16:25:32 :: Creating VM snapshot
29.01.2016 16:26:19 :: Error: An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

The eventlog on the server shows an event 44 by Oracle.VSSWriter for each instance:

VSS-00044: Database in NOARCHIVELOG mode must be in a consistent state for backup.

Cause : The database was open read/write or mounted after a SHUTDOWN IMMEDIATE or SHUTDOWN ABORT.

Action : Either mount the database in a consistent state or open it read-only to back up the database files.

Additional info :
Oracle Database 11g Release 11.2.0.4.0 - 64bit Production
Oracle VSS writer version 11.2.0.2.0 Production
Error at line : 4576
Failure on PrepareForBackup event

The last thing that happened to that database was the stopping and starting of the instances this morning by Veeam. I'll reboot the Oracle server tonight and try again.

[KOPFteam]

No, same error after server reboot.
Any ideas what I can do to bring the Oracle instances into "consistent state" again?

[Gostev]

29.01.2016 16:26:19 :: Error: An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

Yes, it was not a very good suggestion to enable VMware Tools quiscence, as it is extremely unreliable at what it does.

Any ideas what I can do to bring the Oracle instances into "consistent state" again?

The only way to do this in NOARCHIVELOG is to shutdown the database before backup, which is exactly what we are doing anyway (and what prompted you to start this topic in the first place). Here is a quote from the official Oracle documentation:

If you run the database in NOARCHIVELOG mode, however, then the backup must be consistent; that is, you must shut down the database cleanly before the backup.

[kte]

activate the archive mode

[apronk]

I suppose the question remains, how do you stop Veeam from interfering with Oracle databases until Update 1 is available?

[Gostev]

Do not enable application-aware processing on jobs containing VMs with Oracle databases.
However, remember that this way your Oracle backup will be crash-consistent.

[KOPFteam]

I activated archivelogmode tonight. Let's see what happens during the next backup. I'll report back!

[KOPFteam]

Everything is working now

I asked the ERP support to activate archivelogmode in the Oracle instances. Now I get a archivelog file every 10 minutes. These files are much smaller than I expected: Normally between 1 and 5 MB, maybe once per hour one file grows to about 13 MB. This is also the file size I get once when I restart the whole Oracle server.

I have 2 jobs in Veeam: One backup job (once per night) that is pointing to a NAS in another building and a replication job (every 4 hours during working hours) that is pointing to another ESX in the same rack.

In the job definitions I choose the following settings under "Guest Processing" -> "Applications" -> Oracle VM name -> "Edit":

• "General" tab:"Require successful processing" and "Process transaction logs with this job" (Important: The "Oracle" tab is hidden when this option is not selected!)
• "Oracle" tab: Specify an account with Oracle SYSDBA privileges at the top and "Truncate logs over 10 GB"

With theses settings my Veeam jobs are running for 2 days now without any error or warning.

In the Windows Event log on the Oracle Server VM I find an event 4107 by Oracle.VSSWriter.InstanceName: "VSS-04107: successfully backed up component Oracle Database (FULL)" for each Oracle instance after each Veeam job run.

[dellock6]

Sounds good!
thanks for coming back to update the post.

[apronk]

Do not enable application-aware processing on jobs containing VMs with Oracle databases.
However, remember that this way your Oracle backup will be crash-consistent.

So for instance I have a job that contains 30 VM's and 2 of which are Oracle servers, I have to remove them from the job, create a separate job just for the Oracle VM's so that I can disable AAP for that specific job?

[foggy]

Yes, that's what Anton is talking about.

[apronk]

If that's the case I'll stick with VBR 8 for the time being.
That's a bug in my opinion.

[Gostev]

Hmm, not sure how us following Oracle requirements for Oracle database backup is a bug?

Do you not want to backup your Oracle in the only way supported by Oracle?

v8 did not have application-aware processing functionality available for Oracle, so it's not really a fix to stay with v8. This can be compared to turning the music back on in the car, just so that you do not hear engine knocking

[apronk]

Following the requirements is not a bug, actually it was a bug not following it
v8 did not have AAP functionality for Oracle, exactly which is exactly why in v8 I do not have a problem and will have if I upgrade to v9.
Our database servers do not run in ArchiveLog mode, simply because so many actions are ran per minute that the logs would fill up the drives in no time.
Dumps are made from the live environment whilst running, every 12 hours causing no downtime.
So downtime is not acceptable during backup even if it's only 2 minutes.

So you see, ArchiveLog is not an option for us, and downtime as the current v9 implementation of VBR does isn't either yet does so by default.
Hence this new "feature" is a bug for us compared to our current scenario with v8 not posing any trouble because it simple hasn't got AAP for Oracle (even though it did before patch #? which is why one of my first posts on this forum emerged)

[dellock6]

You are probably going to tell me that your use case is different, but how do you guarantee complete capabilities of restores without archive logs?
I've seen few high-end Oracle deployments, and every DBA always told me the solution was to size archive logs and the underlying storage accordingly, never heard of simply forgetting to do them.

[tsightler]

So for instance I have a job that contains 30 VM's and 2 of which are Oracle servers, I have to remove them from the job, create a separate job just for the Oracle VM's so that I can disable AAP for that specific job?

Just disable AAP for those two VMs. I can't think of any reason a second job would be required.

[apronk]

You are probably going to tell me that your use case is different, but how do you guarantee complete capabilities of restores without archive logs?
I've seen few high-end Oracle deployments, and every DBA always told me the solution was to size archive logs and the underlying storage accordingly, never heard of simply forgetting to do them.

Didn't you read? We do dumps every 12 hours so that we can always restore. This was a choice we made from even before Veeam came into play, which means we do not have to enable archivelog.

Just disable AAP for those two VMs. I can't think of any reason a second job would be required.

Thank you, I did not know it was possible to disable it per VM.

I'm not saying we should not rethink our strategy, nor do I want to start a discussion about it.
All I am trying to say is that this behaviour in the change of v8 to v9 can cause problems in some cases, such as ours and I am glad I found out on this forum instead of finding out the hard way like the topic starter had.

[Gostev]

Didn't you read? We do dumps every 12 hours so that we can always restore.

But if you don't have a consistent database state to roll these logs onto, what is the use of those dumps? Or may be I just don't understand your process.

[apronk]

Well to be honest with you, I don't understand the process exactly either.
Someone else does this and he says it has never failed, we use it constantly to import them into our local Oracle instance so seemingly they are consistent.

[tsightler]

Obviously I don't know for your specific situation, but usually the term "dump" refers to export, for example by using Oracle Data Pump, or perhaps even classic Oracle exp/imp commands. This is usually referred to as a "logical backup" (exporting the schema and data) vs "physical backup" (protecting the datafiles, log, and tablespaces themselves). Excellent for loading data between platforms and very commonly used in batch processing scenarios where data needs to be imported into a second system daily (or mulitple times daily). Back before transportable tablespaces it was also commonly used as a generic way to export/import data across platforms (Solaris to Linux for example) since it's a generic format. It doesn't get "everything" but it's completely good enough for many scenarios. If the database/tablespaces become corrupt you just run a simple script to recreate the DB from scratch and then import the latest dump.

[Gostev]

Hmm... certainly, recovery times from the complete server loss will be nothing to write home about with this approach

[tsightler]

Well, it depends on the data size and nothing stops you from backing up the VM but not the Oracle data. We did this a lot when I was a customer because the Oracle data was backed up by RMAN (or some other method) and many of are larger customers continue to use this approach. Veeam backs up the OS and ORACLE_HOME. Recovery times are amazing because you instant restore the server with OS/Oracle installed exactly as it was and hand the DBAs an empty system. They restore the database using whatever means they have. For a batch system that is reloading it's data from scratch daily anyway, this is actually a very awesome approach.