Linux VM backup failed with VMTools

[stevenfoo]

We have just installed the VMTools in one of Linux VM.

Before there was no VMTools and the backup works fine even with Enable VMWare tools quiescense set.

With the VMTools installed and Enable VMWare tools quiescense set, I get failed error message as below:

2 of 14 files processed

Total VM size: 40.00 GB
Processed size: 2.79 KB
Avg. performance rate: 0 KB/s
Backup mode: service console agent

Start time: 7/28/2009 4:42:43 AM
End time: 7/28/2009 4:43:04 AM

Releasing VM files
CreateSnapshot failed, vmRef 224, timeout 1800000, snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory False, quiesce True
Creating a quiesced snapshot failed because the (user-supplied) custom pre-freeze script in the virtual machine exited with a non-zero return code.

For your information, we have not created any pre-freeze script for this Linux VM. Just why it is reporting here.


I have to disable the VMWare tools quiescense to have the backup working.

Please advice how to get VMTools and enable VMWare tools quiescense backup working correctly.

Thanks
Steven

[Gostev]

Steven, most likely you have an existing snapshot on this VM. There is a VMware Tools issue with Linux servers that is causing this error if Linux VM has one or more snapshots. Please remove all snapshots and try again.

[stevenfoo]

My VM is using Redhat 3 ~ 2.6.9-42.ELsmp.

I read the forum topic. I tried to check for any snapshot. There is no vmware snapshot taken on that VM .

I just installed the VMware tools from the Vmware Infrastructure client 3.5 and run the Linux VM with VMWare tools quiescense in Veeam.

I just get the above error as mention.

For far no help and solution,

[Gostev]

Do you know if RedHat 3 is supported by VMware? Sounds like this is very old version. If it is supported, it would be best if you contact our support and let them investigate this issue further. I will appreaciate if you update this post with resolution once resolved, or provide our support with link to this topic so that they could do it.

[tsightler]

My VM is using Redhat 3 ~ 2.6.9-42.ELsmp.

This would be Redhat 4.3, not Redhat 3. Redhat 3.x was based on the 2.4.21 kernel. Redhat 4 was based on the 2.6.9 kernel and the 2.6.9-42 kernel was the initial kernel release for RHEL4 Update 3, also referred to as 4.3.

It appears that you're using network backup, not VCB. Any change of trying VCB backup?

Have you verified if there is a "pre-freeze-script" on the host? Perhaps there is one used by another application that is not working with Veeam. The message seems to indicate that there is such a script that is failing for whatever reason.

[stevenfoo]

Yes RHEL 3 is supported by VMware. I have bring up another support case with Veeam.

Veeam support is looking into it as well.

[stevenfoo]

oops my wrong information. yes, redhat 4.3.

VCB is not possible in our environment.

This is a Linux VM, I have just installed VMware tools. There was notihing installed before. I don't even know how to add a pre-freeze-script under Linux VM.

Any suggestion?

[Gostev]

Every time I've seen this issue reported before, it was due to additional snapshots present on Linux VM. I remembered this one because of completely misleading error message above, which is thrown to us by VMware API. If this is not the case, and extra snapshots are not present, then best would be to have our support and devs investigate this.

[stevenfoo]

Gostev,

I have in contact with Irina Veeam support. She mention that the issue is related to VMware. I still don't accept the answer since I tested a snapshot using VMware infra client and it works. I even delete it.

If you the error, the execution is done by Veeam.

Releasing VM files
CreateSnapshot failed, vmRef 224, timeout 1800000, snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory False, quiesce True
Creating a quiesced snapshot failed because the (user-supplied) custom pre-freeze script in the virtual machine exited with a non-zero return code.

[Gostev]

Steven, I've reviewed the case. It looks that in fact the issue can be reproduced even without Veeam Backup present, because executing snapshot operation through MOB (managed objects browser) also fails with the same error. Could you please contact VMware and have them fix the ability to create quiesced snapshots for this VM through this public API? As soon as the capability to perform snaphost via API for this VM is restored/fixed, Veeam Backup will start working normally.

[tsightler]

I think it's fair to say it's not a Veeam problem as this issue is reported by many users of various backup products, including simply using vcbMount. The solution most people have reported is to use -Q 0 but this just disables quiesce. As far as I know taking a snapshot with the VI client does not do a quiesce.

Other solutions that were reported to work for some users:

1. Vmotion the machine to another ESX host
2. Reboot the ESX host

What version of ESX and VMware tools?

[Gostev]

As far as I know taking a snapshot with the VI client does not do a quiesce.

Tom, yes, VI client does not do a quiesce when snapshotting. I've tested this before myself, when working on my VSS whitepaper last year.

Steven, please keep in mind that Veeam Backup also provides ability to disable VMware Tools quiescence via Advanced job settings dialog (which would make Veeam Backup job to create non-quiesced snapshots, just like VI Client does it).

[stevenfoo]

Issued resovled. It's some damm stupid VMware tools on Linux.

I was working from home last nite from 8pm till 11:15pm.. Finaally, it works. I am cursing the VMware tools.

Below is my reply back to Veeam support

Irina,

This is wierd. The issue have been resolved. No idea why this works now.

1) Create snapshot and delete it at Vmware level (works)
2) Force the vmtools upgrade (not working)
3) Force reinstall again (works)
4) Rerun backup with Vmware tools quiescense - manually (works)
5) Reschedule backup with Vmware tools quiescense (works)

I will try with other linux box when I have the time.

Thanks very much for the help.

Regards
Steven

[Vitaliy S.]

Hello Steven,

Great troubleshoting! and thank you for your update on the issue resolution.

[jaiswack]

I had also same issue and it has been fixed now.
check the permission on below file, this should be executable.
/opt/SYMCquiesce/bin/freeze

[Planet X]

After trying all and I really don't wanted to uninstall VMware tools I figured out that in the Options of the virtual machine in VMware Tools section there are four switches. I removed the check mark for Before suspending. The backup works fine now. You hae to stop the virtual to be able to change these settings by the way. But it's worth it .