Comprehensive data protection for all workloads
Post Reply
virtualwatts
Enthusiast
Posts: 50
Liked: never
Joined: Nov 18, 2010 2:41 pm
Full Name: Rick Watts

Pre-Freeze and Post-Thaw without a VCB Proxy

Post by virtualwatts » Dec 02, 2010 11:22 pm

I haven't been able to get the pre-freeze/post-thaw scripts to fire. The documentation is scant and many forum posts refer to ESX and/or a VCB Proxy.

We are on ESXi 4.1 and Veeam B&R 5. The Veeam server is a VM guest, giving us direct SAN access. We are using the Virtual Appliance backup mode.
So there is no VCB Proxy involved.

Is the VCB Proxy required for pre-freeze/post-thaw (i.e. is Veeam calling VCB which calls these scripts).
Does direct SAN access preclude the use of pre-freeze/post-thaw scripts?

If so I guess the thing to do is powershell everything to provide control of the vm guest before and after backup.

Thanks for your insight
Rick

Gostev
SVP, Product Management
Posts: 24656
Liked: 3471 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by Gostev » Dec 02, 2010 11:29 pm

These scripts are called by VMware as a part of snapshot process (not by Veeam, VCB, vStorage API)

To make it work, make sure that:
1. VMware Tools quiescence is enabled in Advanced job settings (as these scripts are part of VMware Tools quiescence).
2. Application-aware processing is disabled for VM in question (as if enabled, it automatically disables VMware Tools quiescence to prevent conflicts).

Thanks!

virtualwatts
Enthusiast
Posts: 50
Liked: never
Joined: Nov 18, 2010 2:41 pm
Full Name: Rick Watts

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by virtualwatts » Dec 02, 2010 11:53 pm

Thanks Anton - that's the problem, no quiesce.

To be clear then Veeam is calling VMware to perform a snapshot and telling VMware to include memory and quiesce disk (if quiesce is checked)?

We have VMware Tools quiescence turned off because these are Oracle db servers. My understanding is that quiescence can corrupt MSSQL and Oracle databases.
I am testing some recent posts on this forum that it is possible to use Veeam to backup a running Oracle db which would change our VMware backup strategy significantly.

So - no Veeam quiesce no pre-freeze/post-thaw? Do I have that right?

Vitaliy S.
Product Manager
Posts: 22874
Liked: 1540 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by Vitaliy S. » Dec 03, 2010 10:57 am

Rick,

Pre-freeze and post-thaw scripts can only be triggered by using VMware Tools, you may modify your script to process both MSSQL and Oracle databases. The script could simply stop/start SQL Server service before snapshotting.

Thanks!

Gostev
SVP, Product Management
Posts: 24656
Liked: 3471 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by Gostev » Dec 03, 2010 11:58 am

virtualwatts wrote:To be clear then Veeam is calling VMware to perform a snapshot and telling VMware to include memory and quiesce disk (if quiesce is checked)?
Yes, but we never include memory, and whether we quiesce disk depends on the setting I mentioned above.
virtualwatts wrote:We have VMware Tools quiescence turned off because these are Oracle db servers. My understanding is that quiescence can corrupt MSSQL and Oracle databases. I am testing some recent posts on this forum that it is possible to use Veeam to backup a running Oracle db which would change our VMware backup strategy significantly.
You definitely want to view this recorded webinar of ours > Running Oracle on VMware, this is focused on backup and recovery of virtualized Oracle with Veeam.
virtualwatts wrote:So - no Veeam quiesce no pre-freeze/post-thaw? Do I have that right?
No VMware quiesce enabled, no pre-freeze/post-thaw.
Veeam app-aware processing enabled, no pre-freeze/post-thaw no matter if VMware quiesce is enabled or not.

virtualwatts
Enthusiast
Posts: 50
Liked: never
Joined: Nov 18, 2010 2:41 pm
Full Name: Rick Watts

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by virtualwatts » Dec 03, 2010 1:54 pm

I've seen that webinar - thanks for the refresh.
Most of the Webinar content presumes either VSS capable Oracle or Enterprise version of Veeam. Neither of which we have.

The webinar actually reiterates the very questions I have posted here.

The webinar says:
1) Use VSS for 11g and 10g(patched)
2) Use Oracle tools via pre-freeze/post-thaw (e.g. suspend/resume or tablespace backup)
3) Use Crash Consistent - do not use quiesce (beware of SYNC DRIVER!) - warning in original webinar, including exclamations

So the message I get is same as the forums - use the pre-freeze/post-thaw and beware of the SYNC Driver(!).
In the follow-on Q&A (which I didn't listen to the first time) Doug Hazelman reiterates the warning against using quiesce and cautions that the sync driver can corrupt the Oracle dbms. I'm taking that warning pretty seriously when the guy in charge of product strategy says don't use it.

Either the message is muddled or these are apparently conflicting positions.



btw - I'm still fending off all the SearchOracle spam Email from registering for that webinar ;)

Gostev
SVP, Product Management
Posts: 24656
Liked: 3471 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by Gostev » Dec 03, 2010 2:08 pm

I don't really understand what statements in your opinion is conflicting or muddled? Looks pretty clear to me. Actually I was the one who made this preso :D
virtualwatts wrote:btw - I'm still fending off all the SearchOracle spam Email from registering for that webinar ;)
You can only expect Veeam spam from registering to that webinar... ;)
So SearchOracle spam is definitely from some other registration.

virtualwatts
Enthusiast
Posts: 50
Liked: never
Joined: Nov 18, 2010 2:41 pm
Full Name: Rick Watts

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by virtualwatts » Dec 06, 2010 4:47 pm

Just for grins and giggles I filed an SR with Veeam Support asking for an official position whether I should use quiescence to backup an Oracle dbms.
The response was "No", do not use VMware quiescence as it will corrupt your dbms. Instead you should use pre-freeze/post-thaw.
Well, the only way to use pre-freeze/post-thaw is if you select VMware quiescence.... and I asked just that.
The clarification was that selecting quiescence within Veeam invokes the pre-freeze/post-thaw scripts rather than the SYNC driver (which we all know is the culprit)
Unfortunately it took some back and forth to get this spelled out.

So - why is this confusing or muddled?
Because the official position from Veeam is that "you should not use the VMware tools quiescence as it may corrupt your Oracle database"
While also stating that you should select the option on the Veeam v5 configuration advanced tab that says "Enable VMware Tools Quiescence".

I recommend Veeam change the name of the checkbox and/or actually document what the checkbox means.

Thanks for your patience.

Gostev
SVP, Product Management
Posts: 24656
Liked: 3471 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by Gostev » Dec 06, 2010 5:26 pm

I am not sure what specifically are you referring to when saying "official position from Veeam". But I have just reviewed the Oracle webinar PowerPoint (which I created anyway), and have found no conflicting statements. Support answer is correct as well (relative to the way you put your question): yes, you should not use VMware quiescence alone, without pre/post scripts.

So, let me re-iterate the official Veeam position here clearly:

1. If you take pre/post scripts approach to Oracle backups, you need to make sure "Enable VMware Tools quiescence" is checked because this is what will enable pre/post scripts to work. This will also enable SYNC driver, but the driver will not cause any problem in this case, as your applications will be quiesced.

2. If you take crash-consistent approach to Oracle backups, you need to make sure "Enable VMware Tools quiescence" is unchecked, because having this option selected will enable SYNC driver during snapshot to hit live, unquiesced applications. And this is exactly the scenario that may cause famous problems on active databases (and not just Oracle - actually, whe I first learnt about this, it was Microsoft SQL affected).

Hope this helps.

Thanks!

bhnguyen
Novice
Posts: 5
Liked: never
Joined: Feb 22, 2013 9:39 pm
Full Name: Bao Nguyen
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by bhnguyen » Feb 22, 2013 11:55 pm

2. If you take crash-consistent approach to Oracle backups, you need to make sure "Enable VMware Tools quiescence" is unchecked, because having this option selected will enable SYNC driver during snapshot to hit live, unquiesced applications. And this is exactly the scenario that may cause famous problems on active databases (and not just Oracle - actually, whe I first learnt about this, it was Microsoft SQL affected).

Hi, do you know if this is still a problem, since it's been two year's since your post? It seems to me there are two separate issues regarding the SYNC driver and VMTools quiescence in relation to Linux: 1) potential corruption of the live active database, and 2) making sure you have a backup that is transaction consistent. For #2, it makes sense to me that the backup will be most consistent if you stop the database via the pre-freeze-script before snapping, since Linux is VSS-less. For #1, I would like to ask if anyone can confirm that the SYNC driver may potentially cause corruption in the live database, for Linux or non-VSS compatible Windows applications? And would you please provide links to articles if you have them? There seems to be very little information about this problem, and VMWare told me they have not heard of it, although I have found a few references of this issue here and there. For my IT team it is very important to verify this because we have many databases running in Linux as well as MySQL in Windows, and when I bring this up people find it hard to believe that this problem even exists. Thanks!

tsightler
VP, Product Management
Posts: 5405
Liked: 2230 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by tsightler » Feb 23, 2013 6:15 pm 2 people like this post

For Oracle on Linux, assuming you are snapping all volumes, including data and logs, then the reality is that crash consistent is really all that is required. The sync driver is generally not required for Oracle because it's already writing transaction logs in a sync manner, that's pretty much the entire idea of it's transactionally consistent model. Oracle has supported snapshot based backups of "inconsistent" databases (i.e. databases not shut down or in any special backup mode) for a very long time as long as you meet the requirements stated in Oracle My Support ID 604683.1 Supported Backup, Restore and Recovery Operations using Third Party Snapshot Technologies. Unfortunately you must have an Oracle My Support account to see this note, however, it is referenced in many storage systems documents (Netapp, Dell, IBM) with regard to their snapshot backups using storage array snapshots, but the same principle appliance to VM based backups. Here is a small quote from the document:
The third party vendor needs to guarantee and held accountable that their snapshots conform to all the following requirements:
Integrated with Oracle’s recommended restore and recovery operations
Database crash consistent at the point of the snapshot
Write ordering is preserved for each file within a snapshot
VMware snapshots meet all of these requirements as long as the database is fully contained within VMDK volumes that are part of the VM snapshot, and the database will always be recoverable to the point-in-time of the VMware snapshot. If the full VM is restored Oracle will perform automatic media recovery of the database files on the next startup. You can also recover individual database instances as long as you restore all files which make up this databse, including database, redo logs, and control files. Individual database files could also be recovered as long as the required archive logs are available. Of course, the use of raw devices or ASM adds some additional complication to individual database or file recover, but full VM recovery is always possible and easy.

I have personally recovered dozens of Oracle databases in production with Veeam and have never had a failed recovery. I've recovered hundreds, perhaps thousands of Oracle databases (if you count automated scripting) from snapshot based backups (using various storage and virtualization snapshot technologies including Veeam) and I've only seen a handful of errors (mainly with storage array snapshots that were not taken atomically, and thus "smeared" the backup, even this can be solved by carefully setting the order of the storage snapshots).

Here are some documents on using snapshots in general to backup Oracle, they are not specific to VMware, but all refer to crash-consistent backup of Oracle which is what a VMware snapshot of Oracle on Linux is:

http://jianwu.blogspot.com/2010/11/usin ... p-and.html
http://mrothouse.wordpress.com/2012/02/ ... snapshots/

Some things that are important, assuming you are running Oracle in archive log mode (highly recommended) you will still need to handle the purging of your archive logs via some external method as Veeam will not do this automatically. If you are not using archive log mode, then you will get recovery only the to point of the Veeam backups themselves.

bhnguyen
Novice
Posts: 5
Liked: never
Joined: Feb 22, 2013 9:39 pm
Full Name: Bao Nguyen
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by bhnguyen » Feb 24, 2013 12:15 am

Thanks for the great information, it is very helpful! I do wonder if a restored crash consistent OS may not boot up or be corrupt, since it would be like pulling the plug. I understand that most of the time it probably would recover, but that it's not guaranteed and obviously not good for the filesystem. Is that just the chance you take with crash consistency? From an Oracle perspective, I see that you would still have the files inside the VM whether it boots or not. Thus you would not lose data if that were all you cared about, and had time to rebuild a non-booting or corrupt OS.

If you use the pre-freeze script to stop Oracle, then quiesce with the SYNC driver, it seems you would get the best of both worlds: a consistent filesystem as well as avoid corruption that the SYNC driver reportedly can cause to running databases. Is it harmful to restart Oracle twice a night via pre/post scripts for backup and replication? And is this SYNC driver corruption problem still an issue, as it's been two years since it was mentioned in this thread? What about other databases, is crash consistency sufficient as well? Thanks again!

tsightler
VP, Product Management
Posts: 5405
Liked: 2230 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by tsightler » Feb 24, 2013 3:33 am 2 people like this post

One important thing to remember, it's really not like "pulling the plug", at least not in the classic sense. In the old days the loss of power to a system could easily lead to corruption due to the loss of data in the write buffer or partial writes causing on disk corruption. Modern physical systems address this with a battery backed write cache so that pending writes are not lost, however, it could still potentially lead to filesystem corruption due to out of order writes and inconsistent metadata (the old days of chkdsk and fsck). However, a VMware snapshot is not really the equivalent of "pulling the plug". The hypervisor will pause all I/O to the guest and "stun" the guest, this eliminates any risk of partial write corruption as the stun occurs between I/O. There's also no concern for electrically caused corruption which can occur when hard drives are unexpectedly powered down such as in the "pull the plug" scenario, and there's not "head crash" or any other event since it's only a logical "pause" of operations.

On top of this, pretty much all modern file systems on Windows and Linux use journalling techniques to provide robust recovery in the event of an unexpected filesystem crash. It is exceptionally rare to require any thing more than a journal replay to bring a moden Linux filesystem back into a consistent state even after a hard crash, and this journal replay happens automatically as part of the mount process. This is a common process and has proven to be very reliable and is used even with native Linux LVM snapshots. Here's actually a nice article about that:

http://oracleonlinux.wordpress.com/2010 ... ork-part-1

Of course, if you want to use the sync driver and pre-freeze script or whatever, well, I'm not going to say it's a bad idea, actually it's probably quite a good idea, but I don't think that it is necessary to achieve highly reliable backups, and my experience with the Windows sync driver makes me very suspect. I'd probably be far more to just use a pre-freeze script that does the classic "sync; sync; sync". Over a decade of experience taking snapshot backups of Linux systems with various technologies have taught me that they are exceptionally reliable, and of course you can leverage Surebackup to test that the backups are recoverable in an automated fashion.

bhnguyen
Novice
Posts: 5
Liked: never
Joined: Feb 22, 2013 9:39 pm
Full Name: Bao Nguyen
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by bhnguyen » Feb 24, 2013 2:09 pm

That's a very good explanation, I appreciate your insight very much. So at the OS level crash consistent snapshots are adequate in Linux. And for Oracle you've indicated that crash consistency is sufficient as well (I also have seen elsewhere that Veeam's Doug Hazelman has officially recommended crash consistency as a viable option for Orace).

Now is it a different story for other database types? A couple of sources seem to indicate that it is critical to stop I/O in some databases to get transaction consistent backups: Veeam's Ricky Al-Qasem published a white paper Backup of MySQL on a Linux VM http://www.veeam.com/backup_of_mysql_on ... _2_wpp.pdf and especially Eric Seibert's video "VMware backup integrity tools" http://www.backupacademy.com/vmware-bac ... _tr_videos stresses that "quiescing is critical to consistency". Doug Hazelman also explained to me that without stopping I/O, the "chance for corruption is that when the snapshot occurs for the backup that the database will not “recover” properly on restore". Also, a person in this thread http://ngwlist.com/pipermail/ngw/2011-F ... 54982.html reported GroupWise corruption after restoring a Veeam backup. The recommendation to him in the future is to stop I/O before the snapshot to prevent this from happening again.

Thus is Oracle the exception because it is robustly designed and can recover well? As always, thank you for your help in clarifying this issue, as we don't want to take any risks when backing up data.

tsightler
VP, Product Management
Posts: 5405
Liked: 2230 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by tsightler » Feb 24, 2013 5:04 pm

Yes, my advice above was specific to Oracle, although should be applicable to any ACID compliant solution. Quite simply, the entire point of ACID was to provide transactional durability for the physical world in an era when hardware failures were far more common than they are now. Modern ACID compliant databases, when run on raw devices with Direct I/O, or on filesystems with jounalling, are exceptionally robust.

MySQL is a little different. MySQL actually offers different backend storage systems not all of which are full ACID compliant. I believe that the default storage engine for MySQL is still MyISAM, which does not meet the ACID requirement and would generally require full repair/rebuild after a crash. Note that MySQL comes with the myisamchk check utility specifically for this purpose, to verify and repair the integrity of MyISAM tables after any unexpected shutdown of MySQL. If you have MySQL running with MyISAM tables on Linux I would strongly suggest using pre-freeze scripts to perform either a pause (FLUSH TABLES WITH READ LOCK), or a mysqldump of the database (my preference as, being a conservative person I always like to have the native tool backup as well). There is tons of information and scripts on the web that can be used as examples of this since it's basically the same technique used when leveraging native Linux LVM snapshots to backup MySQL.

Of course, MySQL also offers some ACID compliant backends as well, by far the most common being InnoDB. InnoDB is a fully ACID compliant engine with transactional eollback and commit, as well as crash recovery via log replay. It should be as robust as Oracle with regards to recovery after restore from a snapshot. Once again there's plenty of information regarding MySQL and InnoDB, and their differences available on the web.

Unfortunately, MySQL on windows gives you pretty much the same issue as, at least as far as I know, MySQL still does not have a VSS writer on Windows, so all of the same issues apply when running on Windows as well.

Note that this does not mean that a MySQL backup that did not freeze isn't recoverable, but it may require some manual effort to do so, especially if there were active transactions in progress at the time of the backup. I've worked with MySQL since the 3.x versions and I've always been able to recover the data after even a "hard" crash of physical servers and I've also restored dozens of MySQL databases from Veeam backups without significant issue. I can't personally remember I time when I reverted to a native backup after a Veeam restore, although I did have to run the myisamchk utility on one fairly large MyISAM database that took about 20 minutes to repair, but after that, everything was fine.

Unfortunately I don't have a lot of experience with Groupwise, and what I do have is quite old (I'm not convinced I have run across a Groupwise system more than once in the last 10 years), but I believe that their databases are simple flat databases which are obviously not ACID compliant. Unfortunately I have no idea how they handle internal consistency from a crash, but I do know that they have a tool (GWcheck) which you can run to verify and repair any consistency issues in the database after a restore. My guess is this would be required for any restore from an "inconsistent" snapshot, i.e. a snapshot of a running system. If you want to avoid having to run this tool on a restore you'd need to shutdown the instance or otherwise provide some method of bringing the database to a consistent state (assuming such a method exist).

Note that this isn't any different than if the system crashed and you were forced to manually reboot it (for example a Windows BSOD or failure of the physical ESXi host, SAN connectivity, etc.). Restoring a VM from a "crash-consistent" state pretty much implies that, after restore, you will have to perform the same application recovery steps as if the system "crashed" from one of these causes. This is probably the biggest reason why application consistency is important, and strongly desired. It's not so much that recovery from a "crash-consistent" backup isn't possible but it can require additional manual steps that admins would need to be aware of. Note that this doesn't conflict with my statements above, for ACID compliant databases the "crash-consistent" recovery will generally be automatic because of log replay, but for other applications you have to understand what is required.

bhnguyen
Novice
Posts: 5
Liked: never
Joined: Feb 22, 2013 9:39 pm
Full Name: Bao Nguyen
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by bhnguyen » Feb 25, 2013 2:48 am

Thank you so much, that clarifies things tremendously! You do a fantastic job on this forum, it really exceeded my expectations and filled in the gaps in a much needed way. This information is greatly useful to us, thanks again!

bhnguyen
Novice
Posts: 5
Liked: never
Joined: Feb 22, 2013 9:39 pm
Full Name: Bao Nguyen
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by bhnguyen » Mar 01, 2013 8:58 pm

Hi, if we use the pre-freeze-script to stop databases before snapshots, do we need to first stop the applications that connect to those databases? Also, if the database has a remote a connection, is it okay to stop the database if there is an active connection? For example, our users open a Microsoft Access file on their client computers that establishes a remote ODBC connection to a PostgreSQL database located on a server. When we do our backups/snapshots of the virtual server, is it okay to stop PostgreSQL if there is an active ODBC connection to it?

And how about for any local applications that use databases? Another example is our Novell Vibe application that uses MySQL on the same server that Vibe runs on. If I stop MySQL on this server via pre-thaw-script, will I need to also stop Vibe beforehand? It would be nice if the connections/applications simply reconnect after the database comes back up, without us needing to stop/start the applications as well. We would love to use the pre-freeze-script as recommended to achieve consistent backups, but of course want to avoid causing more problems by doing so. Thanks!

tsightler
VP, Product Management
Posts: 5405
Liked: 2230 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by tsightler » Mar 02, 2013 1:22 am

In most cases you'd have to stop the applications as well. Very few applications will take kindly to I'd think that PostgreSQL should be OK without stopping (it's ACID compliant), and for MySQL I'd suggest not stopping the database but rather simply "FLUSH TABLES WITH READ LOCK" and then "UNLOCK TABLES" as documented in the MySQL admin guide.

http://dev.mysql.com/doc/refman/5.1/en/ ... thods.html

At the very bottom it talks about making backups using file system snapshots which is basically the same as what we are doing.

hagepat
Veeam ProPartner
Posts: 15
Liked: never
Joined: Aug 31, 2010 1:17 pm
Contact:

[MERGED] : Replica of Oracle 9.2 database server

Post by hagepat » May 21, 2013 1:46 pm

Hi there,

What is best way to replicate a windows server with oracle 9.2 database with no downtime?
I know that oracle 9.2 doesn't have the oracle vss writer.
Customer wants to replicate database during production hours 3 times a day.
Please advise.

Kind regards,

Patrick Hagen

veremin
Product Manager
Posts: 16789
Liked: 1410 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Pre-Freeze and Post-Thaw without a VCB Proxy

Post by veremin » May 21, 2013 1:55 pm

You have been merged to the existing discussion, so, kindly view the answers provided above.

In nutshell, VMs running obsolete versions of Oracle have to be backed up, using pre-freeze/ post-thaw-scripts in order to complete consistent backups of Oracle databases.

Hope this helps.
Thanks.

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 17 guests