vsphere 5.5 snapshot creation hang at 95% and VM is frozen

[emachabert]

Hi there,

On a fresh 5.5 vsphere setup with latest patches, we encounter a strange behaviour from times to times:

- backup fails for a VM (not always the same)
- we see that the VM is unreachable (frozen)
- we have a stuck task in the vcenter task list: Creating Snapshot stuck at 95%
- we can't stop the VM

So whate we do to recover:
- restart management agents (at this point, the snapshot creation task go from 95% to failed)
- need SSH logon to perform a force stop the vm (esxcli process vm kill --t=force.....)
- consolidate the snapshot
- restart the VM


Have you ever seen this behaviour ?

[veremin]

Can you reproduce this issue manually, without VB&R being present? I mean you can take the snapshot manually, using vSphere client, and see whether the issue shows up again.

Thanks.

[emachabert]

Since it happens from times to times on different servers and VM, it is pretty difficult to reproduce it manually !

We were having this issue more frequently before patching the ESX server with the update released on December 23rd. Also there was a ticket opened with Veeam about VSS failure, the problem was solved by upgrading the Vix dlls....

[foggy]

To investigate this behavior further, debug logs review is required. So I recommend to involve support team here as well.

[emachabert]

Ok. That's what the Customer will do

[Oletho]

Eric, did you solve this with your customer? I have a customer with same problem, not often but still annoying. As it happens to snapshot creation I guess it's a VMware issue.

[Vitaliy S.]

Ole, check out VMware logs of the affected VMs, as they might shed some light on this behavior.

[emachabert]

The problem has gone as soon as we patched to build 839.
We are still monitoring
Are you using HP servers ?

[dfehse]

We are on Build 839 and had this issue with one VM yesterday. Was totaly freezed and we hat to restart mgmt-agent and had to do a soft kill and reregister VM.

Is there any News on this issue?

Thx Daniel

[Vitaliy S.]

Daniel, have you tried to follow Vladimir's advice above?

[dfehse]

.. what advice? Can't see any usable solution in this thread.

[foggy]

Vitaliy means the attempt to take the snapshot of this VM manually, using vSphere client, to check whether the issue relates solely to VMware or Veeam B&R is also involved. Anyway, the most effective way to address this is contacting support directly with debug logs ready for review.

[Ismael]

Hi everyone,
I got the same problem. I work with a trhird party Backup tool: PHD Virtual Backup in vSphere 5.5. The backup tool is installed in a linux appliance that communicates to the vCenter.
When I schedule the job to take the BCK to a VM, first, the appliance send a command to take an Snapshot, after that the appliance attach the disk, take the Backup, des-attach the disk, and remove the snapshot.
My VM get frozen in the middle of the process, I have to run esxcli vm process kill --type=soft,hard,force --world-id=WorldNumber to recover the administration over the VM.
I got a ticket with VMware, but they couldn’t find anything yet. I will tell you when we found the solution.

[dfehse]

HI everyone,

we opened a case with Veeam and they asked us to tried to reproduce the isse by making a copy of a VM that we had this problem. So we made replication every 15min of clone of this VM. Last friday we suddendly had problems with other VMs that freezing for a while, always when they make Volume Shadow Copy (Snapshot) for previews files. We also realized that it was not possible to access the console anymore, got error "Unable to connect to the MKS: The operation is not allowed in the current state". We don't know if this all might be the same Problem. VMWare realsede a patch for that and also update 1 is available.
http://kb.vmware.com/selfservice/micros ... Id=2063262

[tarrach]

Hi everyone,

I got the same problem today. During Replication snapshot creation stops at 95%. The only way to get the machine running, was tha above hint with esxcli -kill process.
Is there a possible solution for this problem? Will be a solution to install update 1 from esxi 5.5?
Any help are welcome.

Vmware Esxi 5.5 1474528
Veeam B&R 7.0 839

[veremin]

Hi, Volker, Is the problem reproducible without Veeam presence? Thanks.

[tarrach]

Hi v.Eremin,

I didn't test it, because the problem occured the first time this night. For testing I took a snapshot from another machine, it worked without problem. In the meantime I update my hosts to U1. I hope it will solve the problem.

thanks

[veremin]

If after installing corresponding update the issue persists (reproducible only in situations when Veeam is present), then, please, open a case with our support team and let them analyze the required logs, confirm your environment, etc.

Thanks.

[Oletho]

My customer opened an SR (#00542057) with Veeam, but they refused to do anything as "they just use the VMware API".

VMware support worked for months without results and then escalated the SR. The first thing that happened after that is we got pointed back back to Veeam as "manual snapshot works, so Veeam must do something wrong with the API".

[foggy]

If manual snapshot can be created without any issues, I recommend asking your Veeam support engineer to escalate the case for further investigation.