Host-based backup of VMware vSphere VMs.
Post Reply
tommyv
Influencer
Posts: 19
Liked: 1 time
Joined: Dec 17, 2014 10:47 am
Full Name: Tom Vernon
Location: England
Contact:

msg.snapshot.error-QUIESCINGERROR

Post by tommyv »

Hi there,

I have a Linux SLES11 VM that has been backing up successfully via B&R for some time. Last week we added an application to the VM; GlusterFS to be precise and now every time a backup is performed it bombs out with the VMware error "msg.snapshot.error-QUIESCINGERROR". Disabling VMware tools quiescence allows the backup to be performed successfully. This is on vSphere 5.5 with the latest VMware tools installed and B&R v8p2.

I'm just wandering if anyone else has come across similar issues when quiescing a Linux guest OS? All of the VMware KB's with similar errors point to Windows VSS issues so aren't helpful.

Thanks :wink:
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by veremin »

Are you able to reproduce this issue by taking manual snapshot with quiescence enabled via vSphere Client?
tommyv
Influencer
Posts: 19
Liked: 1 time
Joined: Dec 17, 2014 10:47 am
Full Name: Tom Vernon
Location: England
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by tommyv »

v.Eremin wrote:Are you able to reproduce this issue by taking manual snapshot with quiescence enabled via vSphere Client?
Its a production box so I cant test it. Just rigging up a test box now to see what happens.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by veremin »

So, you can backup VM in question which also requires snapshot to be created, but cannot create that snapshot manually? The tests conducted on similar testing VM might necessarily show VM specific issue. Thanks.
tommyv
Influencer
Posts: 19
Liked: 1 time
Joined: Dec 17, 2014 10:47 am
Full Name: Tom Vernon
Location: England
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by tommyv »

I forgot to mention that the quiescence snapshot causes Linux to crash and requires a hard reboot, which is why im now trying to recereate the issue on a test box.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by veremin »

So, if the quiescence snapshot conducted outside of VB&R results in VM crash, may be it's time to contact VMware support team and let investigate it. Thanks.
tommyv
Influencer
Posts: 19
Liked: 1 time
Joined: Dec 17, 2014 10:47 am
Full Name: Tom Vernon
Location: England
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by tommyv »

I haven't tried a snapshot outside of VB&R, as I said its a production box so I'm working on recreating a test environment. I havent approached VMware support since they only offer break/fix assistance not fault finding. I'm not really looking for any support, just curious if anyone else has had similar experiences and under what circumstances. Thanks.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by foggy »

You seem to be the first one to report such issue, so if you want to find the reasons and prevent this behavior, contacting support looks to be the most effective way of further action.
kirchk
Lurker
Posts: 1
Liked: never
Joined: May 28, 2015 2:21 pm
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by kirchk »

I just wanted to chime in that I am also having this issue with SLES 11 VMware virtual machines on using a different backup solution. Attempting to create a snapshot within the vSphere client and using the quiesce option does not appear to reproduce the issue. I've been working with VMware technical support for close to two months to resolve this issue. Technical support suggested to update ESX 5.5 to the recently published Patch 6, however, QUIESCINGERRORs still occur seemly at random after patching the hosts.

I'm curious to know too if anyone else has had this issue occur to them and if they were able to go about resolving it. I'm also curious to know if anyone hears anything differently from VMware technical support.
tommyv
Influencer
Posts: 19
Liked: 1 time
Joined: Dec 17, 2014 10:47 am
Full Name: Tom Vernon
Location: England
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by tommyv »

Hi kirchk, thanks for the input. Its interesting that you are also seeing the same issue on SLES. Would you mind letting me know what applications and filesystems are running on the affected VM's?

On a side note, I have set up a test VM and been able to recreate the issue. Interestingly the issue occurs when performing a Veeam backup but not when taking a snapshot directly through the vSphere client.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by veremin »

Probably, I have misinterpreted your answer. If the issue is reproducible only while VB&R is present, please, open a case with our support team and let them analyze the problem directly. Thanks.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by foggy »

Guys, could you please check if the affected VMs have disk.enableUUID parameter set to true in the vmx file and if yes, try to backup with it set to false?

And yes, we would be grateful if you open cases with our support as well and provide their ID's for reference. Thanks!
zerina
Novice
Posts: 3
Liked: never
Joined: Jun 02, 2015 7:39 am
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by zerina »

Hello,

i found this thread while i was searching for a similiar Problem.

In Our case is also a Sles 11 SP3 (up to date patched) affected.

We are running a Vsphere 5.5 (patches up to date) but we are NOT using veeam, we use a different Product (i am not sure if saying which one is here allowed).

Most vesphere Backup solutions rely on VDDK so the process is mostly the same, that means to me, it is somehow a Sles (kernel) + Vsphere VM Tools Problem (probably VDDK related).

We activated vmtools debug log unfortunatly it is not telling usefull informations (while the vm dies) The only thing that i can guess off is, that the related Sles VM has an Sybase DB running wich Sqlanywhere performs some scheduled DB Jobs.

Every Time the VM dies while performing the backup, these jobs are running, so it looks like an overload on the VM (just a guess)

We are still searching for a solution

regards
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by veremin »

Just out of curiosity - does setting disk.enableUUID parameter to false allow your backup solution to proceed any further?
zerina
Novice
Posts: 3
Liked: never
Joined: Jun 02, 2015 7:39 am
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by zerina »

disk.enableUUID is currently not within the .vmx file, what is the default value for disk.enableUUID.

Our Vm is not dying (during backup) ervery time, i can try that setting, but i need some days to proof that this setting helps.
I will test that and will report back.
tommyv
Influencer
Posts: 19
Liked: 1 time
Joined: Dec 17, 2014 10:47 am
Full Name: Tom Vernon
Location: England
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by tommyv »

Interesting that a few people are having the same issues on SLES.

Foggy,v.Eremin, Ive just tried a few test runs with disk.enableUUID=false and it the backup went fine without any issues. When setting disk.enableUUID=true we encounter the error and the VM crashes. I imagine this setting gives us a crash consistent backup so is the equivalent of having quiesence disabled?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by foggy »

Right, "true" (default) means application-consistent backup, "false" - crash-consistent, however the setting applies to Windows guests only, so no need to bother with Linux VMs. We are still investigating the issue.
tommyv
Influencer
Posts: 19
Liked: 1 time
Joined: Dec 17, 2014 10:47 am
Full Name: Tom Vernon
Location: England
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by tommyv »

foggy wrote:Right, "true" (default) means application-consistent backup, "false" - crash-consistent, however the setting applies to Windows guests only, so no need to bother with Linux VMs. We are still investigating the issue.
Are you sure about that? The test shows very different results in Linux when toggling the flag. I can't find much documentation around disk.EnableUUID but I'm guessing it enables ID disk presentation to the OS. Can anyone elaborate?
zerina
Novice
Posts: 3
Liked: never
Joined: Jun 02, 2015 7:39 am
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by zerina »

In our case, the test failed, we had an crash this weekend during backup.
The Test was made with the vmx setting disk.enableUUID, false
Wocka
Enthusiast
Posts: 44
Liked: 8 times
Joined: Oct 01, 2014 12:04 am
Full Name: Warwick Ferguson
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by Wocka »

Hello,

I'm also experiencing this issue in a hosted Datacentre on 7 RedHat servers after upgrading VMware Tools to 9.0.5.21789 (build-1065307). Perviously we were on open vmware tools 8.3.7.4937 (build-381511).

The hoster sent me this error:
An error occured while taking a snapshot: msg.snapshot.error-QUIESCINGERROR
An error occured while saving a snapshot: msg.snapshot.error-QUIESCINGERROR

Environment:
Vmware: 5.5 - I don't know the patch level as it's a hosted server
OS: RedHat Enterprise Linux Server release 6.6 (Santiago)
Backup Software: CommVault 10

Manual Tests I have performed with VMWare tools logging on:
(http://kb.vmware.com/selfservice/micros ... Id=1007873 )
1. Snapshot through vCentre (no Quiesce) = no error or crash
2. Snapshot through vCentre (with Quiesce) = no error or crash
3. Snapshot through CommVault = no error or crash

I could not reproduce the error, but we are still experiencing it randomly over our 7 servers at least one server per night, sometimes up to 3.
When the issue occurs, I can still PING the server, I can SSh to it and enter in root and my password. The prompt never comes back after entering the password. A hard reset is the only way to resolve this.

Interesting, on our internal UAT environment we have not seen this issue:
Vmware: 5.1.0 no patches
OS: RedHat Enterprise Linux Server release 6.6 (Santiago)
All Yum Updates are the same between Dev, Test, UAT and Production
Backup Software: vRanger 6.1.0.35402 (still trying to get mgmt approval for my Veeam solution)

I have found the following articles for Linux, this talks about changing the "disk.enableUUID=false" as perviously mentioned.
http://kb.vmware.com/selfservice/micros ... Id=2038606
http://kb.vmware.com/selfservice/micros ... Id=2079220

I'm about to make these changes today.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by veremin »

Kindly, keep us updated about the results you get. Based on the experience of previous posters, the mentioned setting should be a way to go, indeed.
Wocka
Enthusiast
Posts: 44
Liked: 8 times
Joined: Oct 01, 2014 12:04 am
Full Name: Warwick Ferguson
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by Wocka » 1 person likes this post

A follow up from my post above.

After making the required changes to the .vmx files by our hosting provider and myself making the changes in the VMWare Tools (tools.conf) file. We have not had a server become unresponsive during a snapshot for the past 5 nights. The above KB2079220 from VMWare has resolved the issue.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by veremin »

Thank you for confirming that setting disk.EnableUUID parameter in the .vmx file to false resolves the problem for you; much appreciated.
tommyv
Influencer
Posts: 19
Liked: 1 time
Joined: Dec 17, 2014 10:47 am
Full Name: Tom Vernon
Location: England
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by tommyv » 1 person likes this post

Thanks for the update Wocka. Interestingly KB2079220 claims to be fixed in ESXi5.5u2 however we still see the issue running on the latest version. Setting disk.EnableUUID to false or disabling VMware tools quiesce does work as a fix though.
jspeicher
Service Provider
Posts: 17
Liked: 1 time
Joined: May 09, 2013 7:49 pm
Full Name: Jason
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by jspeicher »

we're seeing the same issue with vcenter/veeam. esxi 5.5.0 3568722

We're getting random issues as described in this thread. When can keep retrying the jobs, and eventually get the job to complete all the vm's. Was any resolution found for anyone?
stuartmacgreen
Expert
Posts: 149
Liked: 34 times
Joined: May 01, 2012 11:56 am
Full Name: Stuart Green
Contact:

Re: msg.snapshot.error-QUIESCINGERROR

Post by stuartmacgreen » 1 person likes this post

I had this issue and was given a fix by VMware. Here is the environment and versions I am running.

VM Guest OS: SUSE Linux Enterprise Server 11 SP4
ESXi: 5.5 U3
VirtualCenter Server: 5.5 U3e

The VM's were updated to VMware Tools 10.x from 9.x and got a load of VM backup failures "msg.snapshot.error-QUIESCINGERROR"
But the VM's would sometimes backup, and then sometimes fail. VERY inconsistent but consistent to fail on the v10.x tools compared to v9.x

I reverted back my VM's to v9.x
Logged a case with VMware to report the issue.
Several VMware Tools in the 10.x stream have been released and all applied to a sample VM but exhibited the same "msg.snapshot.error-QUIESCINGERROR" every now and again.

Now these VM's had configured in the OS via /etc/vmware-tools/tools.conf the following setting:

[vmbackup]
enableSyncDriver = false

This was set after a recommendation from VMware in v9.x tools as without it this HUNG the whole filesystem of the OS - even more severe, than the current predicament of just not quiescing and at least you got the OS back!

VMware recommended to install VMware Tools 10.x and set tools.conf

Code: Select all

[vmbackup]
enableSyncDriver = true
Now we are getting successful VM backups without any QUIESCE errors.
Post Reply

Who is online

Users browsing this forum: No registered users and 67 guests