Comprehensive data protection for all workloads
KOPFteam
Enthusiast
Posts: 53
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Does Veeam 9 stop and start ORACLE instances?

Post by KOPFteam »

In Veeam 8 I got the warning
27.01.2016 12:25:35 :: Excluded Oracle databases: Database AM83/AM83.WORLD'; Database BP71/BP71.WORLD'; Database Test83/TEST83.WORLD'; Database Test71/TEST71.WORLD';
on each run of the job.

In Veeam 9 I don't get that warning, but when I look at the eventlog on our ORACLE server (Windows 2008 R2; ORACLE 11g) all instances were stopped and restarted 2 minutes later. :shock:

After that our ERP system was not working anymore. I had to restart the whole server during production hours!

How can I avoid that? I have disabled the job now.

Thanks,
Florian Pürner
KOPFteam GmbH
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by foggy »

Oracle databases processing was significantly improved in v9 to provide transactionally-consistent backup of Oracle servers and ability to recover database to any point in time (even to a state before a particular transaction). Previously, databases that did not meet the requirements for successful processing were automatically excluded from application-aware processing. Now Veeam B&R is able to correctly process databases and their transaction logs (using VSS or RMAN, according to Oracle recommendations), which in some cases could require stopping and re-starting instances back.

That said, the fact that you had to restart the server after the backup is unexpected, so I recommend contacting support for investigation.
KOPFteam
Enthusiast
Posts: 53
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by KOPFteam »

I'm very happy about the enhanced posibilities of the ORACLE backup, but I think I will have to change my replication job for the ORACLE server. Now I've scheduled it to run every 4 hours.

The ORACLE instances were down for about 2 minutes. This had 2 consequences:

The users couldn't work for 2 minutes. Not good.

The ERP software realized that its ORACLE instances were not available and stopped a few services. That's why the ERP software wasn't fully running after the ORACLE instances were back. To recover from this situation I choose the easy way: Restart server during production hours. Not good.

Do you have any recommendations for me?
Loving (and fighting with) Veeam for over a decade!
KOPFteam
Enthusiast
Posts: 53
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by KOPFteam »

I opened a support case. The ID is 01683017
Loving (and fighting with) Veeam for over a decade!
KOPFteam
Enthusiast
Posts: 53
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by KOPFteam »

Any idea what I could do (maybe on the Oracle server) to stop Veeam from ending the Oracle instances? Oracle is not running in archivelogmode.
Loving (and fighting with) Veeam for over a decade!
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by Gostev »

We are adding a registry key for this in Update 1. Sorry for the inconvenience!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by foggy »

Currently you can disable application-aware image processing for these VMs in advanced guest processing settings in the job and use VMware Tools Quiescence instead.
KOPFteam
Enthusiast
Posts: 53
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by KOPFteam »

Thanks for your replies!
I just changed the job settings und manually started it. I'll let you know what hapend when it finishes.
Looking forward to Update 1...
Loving (and fighting with) Veeam for over a decade!
KOPFteam
Enthusiast
Posts: 53
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by KOPFteam »

No. didn't work:
29.01.2016 16:25:32 :: Creating VM snapshot
29.01.2016 16:26:19 :: Error: An error occurred while saving the snapshot: Failed to quiesce the virtual machine.

The eventlog on the server shows an event 44 by Oracle.VSSWriter for each instance:
VSS-00044: Database in NOARCHIVELOG mode must be in a consistent state for backup.

Cause : The database was open read/write or mounted after a SHUTDOWN IMMEDIATE or SHUTDOWN ABORT.

Action : Either mount the database in a consistent state or open it read-only to back up the database files.

Additional info :
Oracle Database 11g Release 11.2.0.4.0 - 64bit Production
Oracle VSS writer version 11.2.0.2.0 Production
Error at line : 4576
Failure on PrepareForBackup event
The last thing that happened to that database was the stopping and starting of the instances this morning by Veeam. I'll reboot the Oracle server tonight and try again.
Loving (and fighting with) Veeam for over a decade!
KOPFteam
Enthusiast
Posts: 53
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by KOPFteam »

No, same error after server reboot.
Any ideas what I can do to bring the Oracle instances into "consistent state" again?
Loving (and fighting with) Veeam for over a decade!
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by Gostev »

KOPFteam wrote:29.01.2016 16:26:19 :: Error: An error occurred while saving the snapshot: Failed to quiesce the virtual machine.
Yes, it was not a very good suggestion to enable VMware Tools quiscence, as it is extremely unreliable at what it does.
KOPFteam wrote:Any ideas what I can do to bring the Oracle instances into "consistent state" again?
The only way to do this in NOARCHIVELOG is to shutdown the database before backup, which is exactly what we are doing anyway (and what prompted you to start this topic in the first place). Here is a quote from the official Oracle documentation:
Oracle wrote:If you run the database in NOARCHIVELOG mode, however, then the backup must be consistent; that is, you must shut down the database cleanly before the backup.
kte
Expert
Posts: 179
Liked: 8 times
Joined: Jul 02, 2013 7:48 pm
Full Name: Koen Teugels
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by kte »

activate the archive mode
apronk
Enthusiast
Posts: 76
Liked: 9 times
Joined: Mar 23, 2015 2:47 pm
Full Name: Arend Pronk
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by apronk »

I suppose the question remains, how do you stop Veeam from interfering with Oracle databases until Update 1 is available?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by Gostev »

Do not enable application-aware processing on jobs containing VMs with Oracle databases.
However, remember that this way your Oracle backup will be crash-consistent.
KOPFteam
Enthusiast
Posts: 53
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by KOPFteam » 1 person likes this post

I activated archivelogmode tonight. Let's see what happens during the next backup. I'll report back!
Loving (and fighting with) Veeam for over a decade!
KOPFteam
Enthusiast
Posts: 53
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

[SOLVED] Does Veeam 9 stop and start ORACLE instances?

Post by KOPFteam » 3 people like this post

Everything is working now :D

I asked the ERP support to activate archivelogmode in the Oracle instances. Now I get a archivelog file every 10 minutes. These files are much smaller than I expected: Normally between 1 and 5 MB, maybe once per hour one file grows to about 13 MB. This is also the file size I get once when I restart the whole Oracle server.

I have 2 jobs in Veeam: One backup job (once per night) that is pointing to a NAS in another building and a replication job (every 4 hours during working hours) that is pointing to another ESX in the same rack.

In the job definitions I choose the following settings under "Guest Processing" -> "Applications" -> Oracle VM name -> "Edit":
  • "General" tab:"Require successful processing" and "Process transaction logs with this job" (Important: The "Oracle" tab is hidden when this option is not selected!)
  • "Oracle" tab: Specify an account with Oracle SYSDBA privileges at the top and "Truncate logs over 10 GB"
With theses settings my Veeam jobs are running for 2 days now without any error or warning.

In the Windows Event log on the Oracle Server VM I find an event 4107 by Oracle.VSSWriter.InstanceName: "VSS-04107: successfully backed up component Oracle Database (FULL)" for each Oracle instance after each Veeam job run.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by dellock6 »

Sounds good!
thanks for coming back to update the post.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
apronk
Enthusiast
Posts: 76
Liked: 9 times
Joined: Mar 23, 2015 2:47 pm
Full Name: Arend Pronk
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by apronk »

Gostev wrote:Do not enable application-aware processing on jobs containing VMs with Oracle databases.
However, remember that this way your Oracle backup will be crash-consistent.
So for instance I have a job that contains 30 VM's and 2 of which are Oracle servers, I have to remove them from the job, create a separate job just for the Oracle VM's so that I can disable AAP for that specific job?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by foggy »

Yes, that's what Anton is talking about.
apronk
Enthusiast
Posts: 76
Liked: 9 times
Joined: Mar 23, 2015 2:47 pm
Full Name: Arend Pronk
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by apronk »

If that's the case I'll stick with VBR 8 for the time being.
That's a bug in my opinion.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by Gostev »

Hmm, not sure how us following Oracle requirements for Oracle database backup is a bug?

Do you not want to backup your Oracle in the only way supported by Oracle?

v8 did not have application-aware processing functionality available for Oracle, so it's not really a fix to stay with v8. This can be compared to turning the music back on in the car, just so that you do not hear engine knocking :D
apronk
Enthusiast
Posts: 76
Liked: 9 times
Joined: Mar 23, 2015 2:47 pm
Full Name: Arend Pronk
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by apronk »

Following the requirements is not a bug, actually it was a bug not following it :)
v8 did not have AAP functionality for Oracle, exactly which is exactly why in v8 I do not have a problem and will have if I upgrade to v9.
Our database servers do not run in ArchiveLog mode, simply because so many actions are ran per minute that the logs would fill up the drives in no time.
Dumps are made from the live environment whilst running, every 12 hours causing no downtime.
So downtime is not acceptable during backup even if it's only 2 minutes.

So you see, ArchiveLog is not an option for us, and downtime as the current v9 implementation of VBR does isn't either yet does so by default.
Hence this new "feature" is a bug for us compared to our current scenario with v8 not posing any trouble because it simple hasn't got AAP for Oracle (even though it did before patch #? which is why one of my first posts on this forum emerged)
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by dellock6 » 1 person likes this post

You are probably going to tell me that your use case is different, but how do you guarantee complete capabilities of restores without archive logs?
I've seen few high-end Oracle deployments, and every DBA always told me the solution was to size archive logs and the underlying storage accordingly, never heard of simply forgetting to do them.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by tsightler »

apronk wrote: So for instance I have a job that contains 30 VM's and 2 of which are Oracle servers, I have to remove them from the job, create a separate job just for the Oracle VM's so that I can disable AAP for that specific job?
Just disable AAP for those two VMs. I can't think of any reason a second job would be required.
apronk
Enthusiast
Posts: 76
Liked: 9 times
Joined: Mar 23, 2015 2:47 pm
Full Name: Arend Pronk
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by apronk »

dellock6 wrote:You are probably going to tell me that your use case is different, but how do you guarantee complete capabilities of restores without archive logs?
I've seen few high-end Oracle deployments, and every DBA always told me the solution was to size archive logs and the underlying storage accordingly, never heard of simply forgetting to do them.
Didn't you read? We do dumps every 12 hours so that we can always restore. This was a choice we made from even before Veeam came into play, which means we do not have to enable archivelog.
tsightler wrote:Just disable AAP for those two VMs. I can't think of any reason a second job would be required.
Thank you, I did not know it was possible to disable it per VM. :)

I'm not saying we should not rethink our strategy, nor do I want to start a discussion about it.
All I am trying to say is that this behaviour in the change of v8 to v9 can cause problems in some cases, such as ours and I am glad I found out on this forum instead of finding out the hard way like the topic starter had.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by Gostev »

apronk wrote:Didn't you read? We do dumps every 12 hours so that we can always restore.
But if you don't have a consistent database state to roll these logs onto, what is the use of those dumps? Or may be I just don't understand your process.
apronk
Enthusiast
Posts: 76
Liked: 9 times
Joined: Mar 23, 2015 2:47 pm
Full Name: Arend Pronk
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by apronk »

Well to be honest with you, I don't understand the process exactly either.
Someone else does this and he says it has never failed, we use it constantly to import them into our local Oracle instance so seemingly they are consistent.
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by tsightler »

Obviously I don't know for your specific situation, but usually the term "dump" refers to export, for example by using Oracle Data Pump, or perhaps even classic Oracle exp/imp commands. This is usually referred to as a "logical backup" (exporting the schema and data) vs "physical backup" (protecting the datafiles, log, and tablespaces themselves). Excellent for loading data between platforms and very commonly used in batch processing scenarios where data needs to be imported into a second system daily (or mulitple times daily). Back before transportable tablespaces it was also commonly used as a generic way to export/import data across platforms (Solaris to Linux for example) since it's a generic format. It doesn't get "everything" but it's completely good enough for many scenarios. If the database/tablespaces become corrupt you just run a simple script to recreate the DB from scratch and then import the latest dump.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by Gostev »

Hmm... certainly, recovery times from the complete server loss will be nothing to write home about with this approach :D
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Does Veeam 9 stop and start ORACLE instances?

Post by tsightler »

Well, it depends on the data size and nothing stops you from backing up the VM but not the Oracle data. We did this a lot when I was a customer because the Oracle data was backed up by RMAN (or some other method) and many of are larger customers continue to use this approach. Veeam backs up the OS and ORACLE_HOME. Recovery times are amazing because you instant restore the server with OS/Oracle installed exactly as it was and hand the DBAs an empty system. They restore the database using whatever means they have. For a batch system that is reloading it's data from scratch daily anyway, this is actually a very awesome approach.
Post Reply

Who is online

Users browsing this forum: Amazon [Bot], FelixW, Regnor and 150 guests