Database quiescing on Linux

[MJeB]

We want to use Veeam to Backup Linux VMs which hold various kinds of Databases (Oracle, MySql, etc.). As these are Linux maschines we have no such things like vss.
We tried to enable VMware tools quiescing and create pre-freeze-script and post-thaw-script under /usr/sbin and set the job to "Ignore application processing failures".
But the script (a simple test that should just write a log file) wasn't executed and veeam reports the backup as succesful.
And schedule pre commands to be run at specified time is out of question, these database are not running out of joy and have to be down as short as possible, we can't just simply chop the services for 10 minutes or so just to be sure the backup was within that window.

[veremin]

What happens if you create a snapshot manually via vSphere client? Will the scripts be executed? Also, the particular directory where the scripts should be stored depends on ESX(i) version. So, can you confirm that you've chosen the directory correctly?

Thanks.

[Vitaliy S.]

Also there is no need to set "Ignore application processing failures" to true in your case.

[foggy]

In fact, you need to disable the entire application-aware image processing, if you're using pre-freeze/post-thaw scripts (not just ignore its failures).

[MJeB]

In fact, you need to disable the entire application-aware image processing, if you're using pre-freeze/post-thaw scripts (not just ignore its failures).

That was our first attempt, but scripts where not started so we looked in veeam and vmware kbs and forums and found an article regarding mixed environments where windows and linux should be backuped together, where it stated that one could use "application aware image processing" to backup the windows VMs and set the "Ignore application processing failures" in conjunction with VMware tools quiescing to backup all the VMs that don't posses VSS capabilities, so we gave that one a try but it didn't started the scripts too.

We also tried in a single job whit just the enable vmware tools quiescing option for a windows VM to test non VSS Windows Systems.
The Backup of the Windows VMs worked and the Script under C:\Program Files\VMware\VMware tools\backup.d\ got called correctly.

What happens if you create a snapshot manually via vSphere client? Will the scripts be executed? Also, the particular directory where the scripts should be stored depends on ESX(i) version. So, can you confirm that you've chosen the directory correctly?
Thanks.

According to VMware http://www.vmware.com/pdf/vsphere4/r40/ ... #G5.998318 the path differs only in the Microsoft world. For Linux it's simply /usr/sbin the paper is regarding vsphere 4 and we use vsphere 5 with esxi 5.0/5.5 but i couldn't get any papers considering v5
Besides the posted ESX(i)version link only points out script location under windows where we don't have any problems at all.

[veremin]

It's still not clear what result you get while taking snapshot manually. Are the corresponding scripts executed on given Linux VMs or not? Thanks.

[tsightler]

If you can't get your script to run when taking a snapshot via the vSphere client then they won't run when Veeam takes a snapshot either. I just set this up in my lab in just a couple of minutes and it worked fine. This was with vSphere 5.5, Linux VM was RHEL6.5 with current VMware tools installed and the script did nothing but write a date-stamp to a log file but it worked the first time. Here's the steps:

1. Create pre-freeze-script/post-thaw-script. It should be possible to execute these scripts from the bash prompt. They should have owner and group of "root", and permissions should be 0700 (-rwx------) so that the script can only be executed by root.
2. Place script in /usr/sbin
3. Job should be configured in Advanced options, vSphere tab to "Enable VMware Tools quiescence"

Other job settings shouldn't matter, since this is a Linux VM so those settings don't apply. If you still have issues after following these instructions please share more details about your environment (vSphere versions, VMware tools versions, Linux versions, scripts). Below are the simple scripts I used:

/usr/sbin/pre-freeze-script

Code: Select all

#!/bin/bash
echo "Pre-Freeze: `date`" >> /root/freeze-thaw.log

/usr/sbin/post-thaw-script

Code: Select all

#!/bin/bash
echo "Post-Thaw: `date`" >> /root/freeze-thaw.log

And the output from those scripts when running a backup:
# cat /root/freeze-thaw.log

Code: Select all

Pre-Freeze: Wed Mar 12 12:21:28 EDT 2014
Post-Thaw: Wed Mar 12 12:21:32 EDT 2014

[tsightler]

Also, just to clarify, there is no need to do anything at all with the Guest Processing options even if the job contains both Windows and Linux VMs. Current versions of Veeam are smart enough to detect if the VM is Linux and does not attempt to use the Windows only Guest Processing options for Linux VMs even if those options are enabled. If the VMware tools quiescence option is configured in the job, the Linux VMs will use this option, or otherwise their backups will just be crash consistent.

It's possible there may be things that can confuse this automatic detection, but I've not seen that on any reasonably recent version of B&R. In older versions it was common to have to override the Guest Processing settings for Linux VMs to keep it from returning errors, but if this happens in current versions it should be considered a bug.

And yes, I just tested this. The job has 7 VMs, 6 are Windows, only 1 is Linux. Both Application Aware image processing and guest filesystem indexing are enabled globally in the job, as well as VMware Tools quiescence in the advanced option. The 6 Windows VMs processed successfully with AAIP and indexing, while this was automatically skipped for the Linux VM and the pre-freeze/post-thaw scripts ran since I had VMware Tools option enabled in advanced settings.

[serverbu]

We test with a sles11 on a esxi 5.0 or 5.5 (not quite sure if we already updated this paticular blade) using vsphere 5.5
Veeam report the job as success but the script is not called. all the pre-freeze-script should do is

Code: Select all

#!/bin/bash
echo "Veeam started: " `date` >> /tmp/veeam-start.log

and the post-thaw-script is doing

Code: Select all

#!/bin/bash
echo "Veeam stopped: " `date` >> /tmp/veeam-stop.log

testet to run the script manually and they worked.

vmware tools were outdated so I reinstalled with newest vmware tools just to be sure its not an outdated version thing.
still veeam reports successful backup, but scripts don't get called.

P.S. tried to take a snapshot through vsphere, snapshot was taken, but script didn't get called.
modified vmware suspend script (just added an echo to file in main()) and this wasn't done also. seems to be a problem with our vmware infrastructure but then I can't understand why the same infrastructure is able to call vmware tools to run scripts in windows VMs??

Just took a snapshot of our windows test VM. the snapshot didn't trigger the script, veeam does.

[Vitaliy S.]

And what about the permissions? Were they correctly configured as Tom has described above?

[tsightler]

Veeam will report the job as success no matter what happens with the pre-freeze/post-thaw, it has no awareness of pre-freeze/post-thaw operations, it's just calling the API with the require flag to trigger quiescence, that's all. When you took the snapshots via vSphere did you check the "Quiesce guest file system" option? I can't reproduce this issue, it works perfectly for me with Veeam using RHEL5/6 and multiple Ubuntu versions. Unfortunately I don't have any SLES systems. Perhaps it's some type of security restriction.

I'd suggest enabling VMware tools debugging and see if that helps narrow down the cause.

http://kb.vmware.com/selfservice/micros ... Id=1007873

[MJeB]

And what about the permissions? Were they correctly configured as Tom has described above?

For testing purpose the setting were set to an even looser set of 777.

Veeam will report the job as success no matter what happens with the pre-freeze/post-thaw, it has no awareness of pre-freeze/post-thaw operations, it's just calling the API with the require flag to trigger quiescence, that's all.

Hmm, thought Veeam would check return codes.

When you took the snapshots via vSphere did you check the "Quiesce guest file system" option?

Must admit I forgot to check the option.
Did so and snapshot triggered the scripts but snapshot fails at trying to quiesce guest OS. post-thaw-script is called with parameter freezeFail.

I can't reproduce this issue, it works perfectly for me with Veeam using RHEL5/6 and multiple Ubuntu versions. Unfortunately I don't have any SLES systems. Perhaps it's some type of security restriction.

I'am going to set up a Debian to test if its a SLES issue.

I'd suggest enabling VMware tools debugging and see if that helps narrow down the cause.
http://kb.vmware.com/selfservice/micros ... Id=1007873

I'am going to activate logging, thanks for the kb source!

[tsightler]

For testing purpose the setting were set to an even looser set of 777.

I've been told that, if permission are too loose, the script might not be executed do to the potential security issue of having a world writable script executed by root. I have no idea if this is true or not, I guess I should test it, but that's why I suggested the tighter security settings. I have tested both 755 and 700.

Hmm, thought Veeam would check return codes.

What return codes? We're only calling the API to request a snapshot with quiescence, that's it. It's up to VMware to run the scripts, monitor return codes, take the snapshot, etc. VMware only reports whether the creation of the snapshot was successful or not. If VMware detects failure in pre-freeze script, then it will likely return an error and not take the actual snapshot, and our backup will fail due to no snapshot being created. But if VMware does take the snapshot, and return that success to us, we have no other info to go on.

Did so and snapshot triggered the scripts but snapshot fails at trying to quiesce guest OS. post-thaw-script is called with parameter freezeFail.

Interesting. And that was on the SLES11 machine? When you choose this option this is effectively the exact same call being made by Veeam (assuming the Quiescence options is selected in the Advanced options under the vSphere tab). I would expect it to produce the same results. Please keep us updated on your testing.


Please keep us updating on your testing.

[MJeB]

I was finally able to trigger a successful snapshot via vsphere with pre-freeze-script getting called with parameter freeze and post-thaw-script getting called with parameter thaw.
Script permission were still set to 777, so vmware-tools aren't concerning about too loose permissions, that just as a side-note. will restrict permission to root only.
I tested on one of our RHEL5 32bit test systems (SLES is 32bit too by the way, don't know if issue occurs on 64bit by now. will have to check with a 64bit machine) and had the same issue regarding snapshot not beeing able to quiesce guest OS.

The problem was caused by the vmware-tools syncdriver. As a solution I created /etc/vmware-tools/tools.conf with

Code: Select all

[vmbackup]
enableSyncDriver=false

and vsphere could take snapshot successful even with quiescing guest OS option.
Veeam could start the Backup job and started the scripts as well, although I must admit I don't know why Veeam triggered the script all of a sudden though I didn't edited the backup job since last time were the scripts didn't get called.

Thanks go out to the veeam support, which I find very good and quick responding.

A big thanks to you tsightler. Your hints and insights were very helpfull.