Discussions specific to the VMware vSphere hypervisor
Post Reply
emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

vsphere 5.5 snapshot creation hang at 95% and VM is frozen

Post by emachabert » Feb 14, 2014 9:27 am

Hi there,

On a fresh 5.5 vsphere setup with latest patches, we encounter a strange behaviour from times to times:

- backup fails for a VM (not always the same)
- we see that the VM is unreachable (frozen)
- we have a stuck task in the vcenter task list: Creating Snapshot stuck at 95%
- we can't stop the VM

So whate we do to recover:
- restart management agents (at this point, the snapshot creation task go from 95% to failed)
- need SSH logon to perform a force stop the vm (esxcli process vm kill --t=force.....)
- consolidate the snapshot
- restart the VM


Have you ever seen this behaviour ?
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

veremin
Product Manager
Posts: 17005
Liked: 1455 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by veremin » Feb 14, 2014 10:23 am

Can you reproduce this issue manually, without VB&R being present? I mean you can take the snapshot manually, using vSphere client, and see whether the issue shows up again.

Thanks.

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by emachabert » Feb 14, 2014 10:59 am

Since it happens from times to times on different servers and VM, it is pretty difficult to reproduce it manually !

We were having this issue more frequently before patching the ESX server with the update released on December 23rd. Also there was a ticket opened with Veeam about VSS failure, the problem was solved by upgrading the Vix dlls....
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

foggy
Veeam Software
Posts: 18366
Liked: 1575 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by foggy » Feb 14, 2014 11:02 am

To investigate this behavior further, debug logs review is required. So I recommend to involve support team here as well.

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by emachabert » Feb 14, 2014 12:17 pm

Ok. That's what the Customer will do :-)
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

Oletho
Enthusiast
Posts: 66
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by Oletho » Mar 24, 2014 10:48 am

Eric, did you solve this with your customer? I have a customer with same problem, not often but still annoying. As it happens to snapshot creation I guess it's a VMware issue.

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by Vitaliy S. » Mar 24, 2014 11:01 am

Ole, check out VMware logs of the affected VMs, as they might shed some light on this behavior.

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by emachabert » Mar 24, 2014 9:27 pm

The problem has gone as soon as we patched to build 839.
We are still monitoring
Are you using HP servers ?
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

dfehse
Influencer
Posts: 10
Liked: never
Joined: Aug 15, 2013 7:51 am
Full Name: Daniel Fehse
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by dfehse » Mar 26, 2014 10:05 am

We are on Build 839 and had this issue with one VM yesterday. Was totaly freezed and we hat to restart mgmt-agent and had to do a soft kill and reregister VM.

Is there any News on this issue?

Thx Daniel

Vitaliy S.
Product Manager
Posts: 23070
Liked: 1582 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by Vitaliy S. » Mar 26, 2014 10:35 am

Daniel, have you tried to follow Vladimir's advice above?

dfehse
Influencer
Posts: 10
Liked: never
Joined: Aug 15, 2013 7:51 am
Full Name: Daniel Fehse
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by dfehse » Mar 31, 2014 4:27 am

.. what advice? Can't see any usable solution in this thread.

foggy
Veeam Software
Posts: 18366
Liked: 1575 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by foggy » Mar 31, 2014 5:38 am

Vitaliy means the attempt to take the snapshot of this VM manually, using vSphere client, to check whether the issue relates solely to VMware or Veeam B&R is also involved. Anyway, the most effective way to address this is contacting support directly with debug logs ready for review.

Ismael
Lurker
Posts: 1
Liked: never
Joined: Apr 01, 2014 10:08 pm
Full Name: Ismael Bustillo
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by Ismael » Apr 01, 2014 10:27 pm

Hi everyone,
I got the same problem. I work with a trhird party Backup tool: PHD Virtual Backup in vSphere 5.5. The backup tool is installed in a linux appliance that communicates to the vCenter.
When I schedule the job to take the BCK to a VM, first, the appliance send a command to take an Snapshot, after that the appliance attach the disk, take the Backup, des-attach the disk, and remove the snapshot.
My VM get frozen in the middle of the process, I have to run esxcli vm process kill --type=soft,hard,force --world-id=WorldNumber to recover the administration over the VM.
I got a ticket with VMware, but they couldn’t find anything yet. I will tell you when we found the solution.

dfehse
Influencer
Posts: 10
Liked: never
Joined: Aug 15, 2013 7:51 am
Full Name: Daniel Fehse
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by dfehse » Apr 07, 2014 5:21 am

HI everyone,

we opened a case with Veeam and they asked us to tried to reproduce the isse by making a copy of a VM that we had this problem. So we made replication every 15min of clone of this VM. Last friday we suddendly had problems with other VMs that freezing for a while, always when they make Volume Shadow Copy (Snapshot) for previews files. We also realized that it was not possible to access the console anymore, got error "Unable to connect to the MKS: The operation is not allowed in the current state". We don't know if this all might be the same Problem. VMWare realsede a patch for that and also update 1 is available.
http://kb.vmware.com/selfservice/micros ... Id=2063262

tarrach
Lurker
Posts: 2
Liked: never
Joined: Nov 11, 2010 10:44 am
Full Name: Volker Tarrach
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by tarrach » Apr 07, 2014 7:55 am

Hi everyone,

I got the same problem today. During Replication snapshot creation stops at 95%. The only way to get the machine running, was tha above hint with esxcli -kill process.
Is there a possible solution for this problem? Will be a solution to install update 1 from esxi 5.5?
Any help are welcome.

Vmware Esxi 5.5 1474528
Veeam B&R 7.0 839
Volker Tarrach
Schäferbarthold GmbH

veremin
Product Manager
Posts: 17005
Liked: 1455 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by veremin » Apr 07, 2014 8:02 am

Hi, Volker, Is the problem reproducible without Veeam presence? Thanks.

tarrach
Lurker
Posts: 2
Liked: never
Joined: Nov 11, 2010 10:44 am
Full Name: Volker Tarrach
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by tarrach » Apr 07, 2014 10:16 am

Hi v.Eremin,

I didn't test it, because the problem occured the first time this night. For testing I took a snapshot from another machine, it worked without problem. In the meantime I update my hosts to U1. I hope it will solve the problem.

thanks
Volker Tarrach
Schäferbarthold GmbH

veremin
Product Manager
Posts: 17005
Liked: 1455 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by veremin » Apr 07, 2014 10:45 am

If after installing corresponding update the issue persists (reproducible only in situations when Veeam is present), then, please, open a case with our support team and let them analyze the required logs, confirm your environment, etc.

Thanks.

Oletho
Enthusiast
Posts: 66
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by Oletho » Jun 24, 2014 10:29 am

My customer opened an SR (#00542057) with Veeam, but they refused to do anything as "they just use the VMware API".

VMware support worked for months without results and then escalated the SR. The first thing that happened after that is we got pointed back back to Veeam as "manual snapshot works, so Veeam must do something wrong with the API".

foggy
Veeam Software
Posts: 18366
Liked: 1575 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by foggy » Jun 24, 2014 10:44 am

If manual snapshot can be created without any issues, I recommend asking your Veeam support engineer to escalate the case for further investigation.

Post Reply

Who is online

Users browsing this forum: No registered users and 12 guests