msg.snapshot.error-QUIESCINGERROR

[tommyv]

Hi there,

I have a Linux SLES11 VM that has been backing up successfully via B&R for some time. Last week we added an application to the VM; GlusterFS to be precise and now every time a backup is performed it bombs out with the VMware error "msg.snapshot.error-QUIESCINGERROR". Disabling VMware tools quiescence allows the backup to be performed successfully. This is on vSphere 5.5 with the latest VMware tools installed and B&R v8p2.

I'm just wandering if anyone else has come across similar issues when quiescing a Linux guest OS? All of the VMware KB's with similar errors point to Windows VSS issues so aren't helpful.

Thanks

[veremin]

Are you able to reproduce this issue by taking manual snapshot with quiescence enabled via vSphere Client?

[tommyv]

Are you able to reproduce this issue by taking manual snapshot with quiescence enabled via vSphere Client?

Its a production box so I cant test it. Just rigging up a test box now to see what happens.

[veremin]

So, you can backup VM in question which also requires snapshot to be created, but cannot create that snapshot manually? The tests conducted on similar testing VM might necessarily show VM specific issue. Thanks.

[tommyv]

I forgot to mention that the quiescence snapshot causes Linux to crash and requires a hard reboot, which is why im now trying to recereate the issue on a test box.

[veremin]

So, if the quiescence snapshot conducted outside of VB&R results in VM crash, may be it's time to contact VMware support team and let investigate it. Thanks.

[tommyv]

I haven't tried a snapshot outside of VB&R, as I said its a production box so I'm working on recreating a test environment. I havent approached VMware support since they only offer break/fix assistance not fault finding. I'm not really looking for any support, just curious if anyone else has had similar experiences and under what circumstances. Thanks.

[foggy]

You seem to be the first one to report such issue, so if you want to find the reasons and prevent this behavior, contacting support looks to be the most effective way of further action.

[kirchk]

I just wanted to chime in that I am also having this issue with SLES 11 VMware virtual machines on using a different backup solution. Attempting to create a snapshot within the vSphere client and using the quiesce option does not appear to reproduce the issue. I've been working with VMware technical support for close to two months to resolve this issue. Technical support suggested to update ESX 5.5 to the recently published Patch 6, however, QUIESCINGERRORs still occur seemly at random after patching the hosts.

I'm curious to know too if anyone else has had this issue occur to them and if they were able to go about resolving it. I'm also curious to know if anyone hears anything differently from VMware technical support.

[tommyv]

Hi kirchk, thanks for the input. Its interesting that you are also seeing the same issue on SLES. Would you mind letting me know what applications and filesystems are running on the affected VM's?

On a side note, I have set up a test VM and been able to recreate the issue. Interestingly the issue occurs when performing a Veeam backup but not when taking a snapshot directly through the vSphere client.

[veremin]

Probably, I have misinterpreted your answer. If the issue is reproducible only while VB&R is present, please, open a case with our support team and let them analyze the problem directly. Thanks.

[foggy]

Guys, could you please check if the affected VMs have disk.enableUUID parameter set to true in the vmx file and if yes, try to backup with it set to false?

And yes, we would be grateful if you open cases with our support as well and provide their ID's for reference. Thanks!

[zerina]

Hello,

i found this thread while i was searching for a similiar Problem.

In Our case is also a Sles 11 SP3 (up to date patched) affected.

We are running a Vsphere 5.5 (patches up to date) but we are NOT using veeam, we use a different Product (i am not sure if saying which one is here allowed).

Most vesphere Backup solutions rely on VDDK so the process is mostly the same, that means to me, it is somehow a Sles (kernel) + Vsphere VM Tools Problem (probably VDDK related).

We activated vmtools debug log unfortunatly it is not telling usefull informations (while the vm dies) The only thing that i can guess off is, that the related Sles VM has an Sybase DB running wich Sqlanywhere performs some scheduled DB Jobs.

Every Time the VM dies while performing the backup, these jobs are running, so it looks like an overload on the VM (just a guess)

We are still searching for a solution

regards

[veremin]

Just out of curiosity - does setting disk.enableUUID parameter to false allow your backup solution to proceed any further?

[zerina]

disk.enableUUID is currently not within the .vmx file, what is the default value for disk.enableUUID.

Our Vm is not dying (during backup) ervery time, i can try that setting, but i need some days to proof that this setting helps.
I will test that and will report back.

[tommyv]

Interesting that a few people are having the same issues on SLES.

Foggy,v.Eremin, Ive just tried a few test runs with disk.enableUUID=false and it the backup went fine without any issues. When setting disk.enableUUID=true we encounter the error and the VM crashes. I imagine this setting gives us a crash consistent backup so is the equivalent of having quiesence disabled?

[foggy]

Right, "true" (default) means application-consistent backup, "false" - crash-consistent, however the setting applies to Windows guests only, so no need to bother with Linux VMs. We are still investigating the issue.

[tommyv]

Right, "true" (default) means application-consistent backup, "false" - crash-consistent, however the setting applies to Windows guests only, so no need to bother with Linux VMs. We are still investigating the issue.

Are you sure about that? The test shows very different results in Linux when toggling the flag. I can't find much documentation around disk.EnableUUID but I'm guessing it enables ID disk presentation to the OS. Can anyone elaborate?

[zerina]

In our case, the test failed, we had an crash this weekend during backup.
The Test was made with the vmx setting disk.enableUUID, false

[Wocka]

Hello,

I'm also experiencing this issue in a hosted Datacentre on 7 RedHat servers after upgrading VMware Tools to 9.0.5.21789 (build-1065307). Perviously we were on open vmware tools 8.3.7.4937 (build-381511).

The hoster sent me this error:
An error occured while taking a snapshot: msg.snapshot.error-QUIESCINGERROR
An error occured while saving a snapshot: msg.snapshot.error-QUIESCINGERROR

Environment:
Vmware: 5.5 - I don't know the patch level as it's a hosted server
OS: RedHat Enterprise Linux Server release 6.6 (Santiago)
Backup Software: CommVault 10

Manual Tests I have performed with VMWare tools logging on:
(http://kb.vmware.com/selfservice/micros ... Id=1007873 )
1. Snapshot through vCentre (no Quiesce) = no error or crash
2. Snapshot through vCentre (with Quiesce) = no error or crash
3. Snapshot through CommVault = no error or crash

I could not reproduce the error, but we are still experiencing it randomly over our 7 servers at least one server per night, sometimes up to 3.
When the issue occurs, I can still PING the server, I can SSh to it and enter in root and my password. The prompt never comes back after entering the password. A hard reset is the only way to resolve this.

Interesting, on our internal UAT environment we have not seen this issue:
Vmware: 5.1.0 no patches
OS: RedHat Enterprise Linux Server release 6.6 (Santiago)
All Yum Updates are the same between Dev, Test, UAT and Production
Backup Software: vRanger 6.1.0.35402 (still trying to get mgmt approval for my Veeam solution)

I have found the following articles for Linux, this talks about changing the "disk.enableUUID=false" as perviously mentioned.
http://kb.vmware.com/selfservice/micros ... Id=2038606
http://kb.vmware.com/selfservice/micros ... Id=2079220

I'm about to make these changes today.

[veremin]

Kindly, keep us updated about the results you get. Based on the experience of previous posters, the mentioned setting should be a way to go, indeed.

[Wocka]

A follow up from my post above.

After making the required changes to the .vmx files by our hosting provider and myself making the changes in the VMWare Tools (tools.conf) file. We have not had a server become unresponsive during a snapshot for the past 5 nights. The above KB2079220 from VMWare has resolved the issue.

[veremin]

Thank you for confirming that setting disk.EnableUUID parameter in the .vmx file to false resolves the problem for you; much appreciated.

[tommyv]

Thanks for the update Wocka. Interestingly KB2079220 claims to be fixed in ESXi5.5u2 however we still see the issue running on the latest version. Setting disk.EnableUUID to false or disabling VMware tools quiesce does work as a fix though.

[jspeicher]

we're seeing the same issue with vcenter/veeam. esxi 5.5.0 3568722

We're getting random issues as described in this thread. When can keep retrying the jobs, and eventually get the job to complete all the vm's. Was any resolution found for anyone?

[stuartmacgreen]

I had this issue and was given a fix by VMware. Here is the environment and versions I am running.

VM Guest OS: SUSE Linux Enterprise Server 11 SP4
ESXi: 5.5 U3
VirtualCenter Server: 5.5 U3e

The VM's were updated to VMware Tools 10.x from 9.x and got a load of VM backup failures "msg.snapshot.error-QUIESCINGERROR"
But the VM's would sometimes backup, and then sometimes fail. VERY inconsistent but consistent to fail on the v10.x tools compared to v9.x

I reverted back my VM's to v9.x
Logged a case with VMware to report the issue.
Several VMware Tools in the 10.x stream have been released and all applied to a sample VM but exhibited the same "msg.snapshot.error-QUIESCINGERROR" every now and again.

Now these VM's had configured in the OS via /etc/vmware-tools/tools.conf the following setting:

[vmbackup]
enableSyncDriver = false

This was set after a recommendation from VMware in v9.x tools as without it this HUNG the whole filesystem of the OS - even more severe, than the current predicament of just not quiescing and at least you got the OS back!

VMware recommended to install VMware Tools 10.x and set tools.conf

Code: Select all

[vmbackup]
enableSyncDriver = true

Now we are getting successful VM backups without any QUIESCE errors.