VMWare Tools not running?

[Mike]

Got this error last night, on this job is the option "Enable VMware tools quiescense" disabled.

(Its not very clear what this option does btw)

The error happend on the second last server on the job and on the last server in the job.
The servers where online, no errors there.
-----
0 of 0 files processed

Total VM size: 120,03 GB
Processed size: 0,00 KB
Avg. performance rate: 0 KB/s
Backup mode: VCB SAN

Start time: 16-3-2009 4:14:03
End time: 16-3-2009 4:42:46

Retrieving file list
Failed to perform Veeam VSS freeze: VMware tools status is "toolsNotRunning"

Failed to perform Veeam VSS freeze: VMware tools status is "toolsNotRunning"

[Gostev]

Sounds like you do not have VMware Tools installed and/or running on these servers?

[ecleppe]

Hello Mike,

This is a known issue which will be fixed in update 4 of ESX (not vCenter) which should be released soon.
To counter this problem you can run 'service mgmt-vmware restart' on your ESX host. This should solve the problem temporarily.
If it does not, restart the VM Tools service within the Windows guest. Note that after another VCB backup, you will have to restart it again.

Good luck,
Erik

[Gostev]

Erik, thank you very much for taking time to post this. I was not aware about this issue.

[beckhamk]

Can I get some more information on this?

We just upgraded and have seen this "Retrieving file list Failed to perform Veeam VSS freeze: VMware tools status is "toolsNotRunning" " on a few jobs. Is there anything to worry about?


What I find really interesting is that we did not have this issue with v2.0.1. only after we upgraded to v3.0.1.

[ecleppe]

To Beckhamk

What version of vCenter are you running now?
Well it's pretty annoying if your tools cannot be found, as the shutdown / restart guest options will also be unavailable.

Erik

[Gostev]

Erik, do you know if this bug was introduced with the latest vCenter update (Update 4)?
That would explain why this issue appeared all of a sudden...

[ecleppe]

Gostev,

I assume the problem started as of the vCenter update to update 4.
At least that's what I've read in some forum posts of other users having the same issue.

Cheers,
Erik

[Gostev]

Hmm looks like this is indeed VMware bug introduced with vCenter Update 4...
Sometimes vCenter does not return correct VMware Tools state even though the tools are running.

Take a look at this new post on nworks MP forum:
http://forum.veeam.com/forums/viewtopic.php?f=1&t=881

[Gostev]

Official description of this issue from VMware Knowledge Base (article was last modified just one week ago)
VMware Tools status shows as not running after running VMware Consolidated Backup

[Mike]

Workarounds
Perform one of the following options as a workaround:
1. Restart the mgmt-vmware service immediately after the backup job is done. This changes the Tools status to OK. You can write a cron job to do it periodically.
OR
2. Log in and log out, or log out if you are already logged in, from the virtual machine. This changes the Tools status to OK if it was showing as Not running.
OR
3. Use VCBMounter to look for virtual machine name or UUID rather than virtual machine IP. Virtual machine IP only works when the status of tools is OK, but virtual machine name and UUID works even if the Tools status shows as Not running.

Gostev,

Is Veeam Backup working on IP address or on UUID for the VM, UUID wouldnt cause this problem if I can beleive the workarounds..

Or do you have an other good idea?

[Gostev]

Yes, unlike many competitive solutions Veeam Backup uses UUID. However we still need VMware Tools to be functional because we do much more than regular VCB does. I heard this issue will be addressed in the upcoming ESX 3.5 Update 4. We are also investigating possibility to provide a temporary fix until the ESX update is available. So it would be best if everyone who is experiencing this issue opens a support case to be notified when fix becomes available.

[ecleppe]

The update 4 of ESX has been released, but the issue with VMTools is still listed as a known issue.
I'm going to test the update in our test environment and I'll see what it does. There is a VM Tools update coming with the ESX U4.

Erik

[Gostev]

Yes, we will also test this. Hopefully they have fixed an issue with the VMware Tools remaining in the "not running" state after VCB operation completes. This happened after some earlier patches. As for original issue, with VMware Tools "not running" during the snapshot operation, this has been there forever and it should not affect Veeam Backup.

[Gostev]

All, our initial testing shows that ESX 3.5 Update 4 appears to fix this issue... can anyone else confirm that you are seeing the same? Thanks!

[rjhindley]

Yes, I can confirm that update 4 fixed this issue for us.

[beckhamk]

yes - this is resolved for us in ESX update 4.

[Wil]

Good afternoon,
One of my backup jobs is failing. The message i recieved is below.

Code: Select all

Validating guest agent availability for the VM
Unable to use guest agent: VM guest OS state is 'NotRunning'

Freezing guest operating system
CreateSnapshot failed, vmRef "vm-75218", timeout "1800000", snName "VEEAM BACKUP TEMPORARY SNAPSHOT", snDescription "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
The attempted operation cannot be performed in the current state (Powered off).

The server is up and running.
Has anyone experienced this issue?
Any advice would be appreicated.

Thank you,

[Vitaliy S.]

Hi Wil, make sure VMware Tools are up-to-date and running, usually invalid tools state is the reason for these kind of messages.