Host-based backup of VMware vSphere VMs.
Post Reply
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

vsphere 5.5 snapshot creation hang at 95% and VM is frozen

Post by emachabert »

Hi there,

On a fresh 5.5 vsphere setup with latest patches, we encounter a strange behaviour from times to times:

- backup fails for a VM (not always the same)
- we see that the VM is unreachable (frozen)
- we have a stuck task in the vcenter task list: Creating Snapshot stuck at 95%
- we can't stop the VM

So whate we do to recover:
- restart management agents (at this point, the snapshot creation task go from 95% to failed)
- need SSH logon to perform a force stop the vm (esxcli process vm kill --t=force.....)
- consolidate the snapshot
- restart the VM


Have you ever seen this behaviour ?
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by veremin »

Can you reproduce this issue manually, without VB&R being present? I mean you can take the snapshot manually, using vSphere client, and see whether the issue shows up again.

Thanks.
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by emachabert »

Since it happens from times to times on different servers and VM, it is pretty difficult to reproduce it manually !

We were having this issue more frequently before patching the ESX server with the update released on December 23rd. Also there was a ticket opened with Veeam about VSS failure, the problem was solved by upgrading the Vix dlls....
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by foggy »

To investigate this behavior further, debug logs review is required. So I recommend to involve support team here as well.
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by emachabert »

Ok. That's what the Customer will do :-)
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by Oletho »

Eric, did you solve this with your customer? I have a customer with same problem, not often but still annoying. As it happens to snapshot creation I guess it's a VMware issue.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by Vitaliy S. »

Ole, check out VMware logs of the affected VMs, as they might shed some light on this behavior.
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by emachabert »

The problem has gone as soon as we patched to build 839.
We are still monitoring
Are you using HP servers ?
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
dfehse
Influencer
Posts: 14
Liked: 1 time
Joined: Aug 15, 2013 7:51 am
Full Name: Daniel Fehse
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by dfehse »

We are on Build 839 and had this issue with one VM yesterday. Was totaly freezed and we hat to restart mgmt-agent and had to do a soft kill and reregister VM.

Is there any News on this issue?

Thx Daniel
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by Vitaliy S. »

Daniel, have you tried to follow Vladimir's advice above?
dfehse
Influencer
Posts: 14
Liked: 1 time
Joined: Aug 15, 2013 7:51 am
Full Name: Daniel Fehse
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by dfehse »

.. what advice? Can't see any usable solution in this thread.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by foggy »

Vitaliy means the attempt to take the snapshot of this VM manually, using vSphere client, to check whether the issue relates solely to VMware or Veeam B&R is also involved. Anyway, the most effective way to address this is contacting support directly with debug logs ready for review.
Ismael
Lurker
Posts: 1
Liked: never
Joined: Apr 01, 2014 10:08 pm
Full Name: Ismael Bustillo
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by Ismael »

Hi everyone,
I got the same problem. I work with a trhird party Backup tool: PHD Virtual Backup in vSphere 5.5. The backup tool is installed in a linux appliance that communicates to the vCenter.
When I schedule the job to take the BCK to a VM, first, the appliance send a command to take an Snapshot, after that the appliance attach the disk, take the Backup, des-attach the disk, and remove the snapshot.
My VM get frozen in the middle of the process, I have to run esxcli vm process kill --type=soft,hard,force --world-id=WorldNumber to recover the administration over the VM.
I got a ticket with VMware, but they couldn’t find anything yet. I will tell you when we found the solution.
dfehse
Influencer
Posts: 14
Liked: 1 time
Joined: Aug 15, 2013 7:51 am
Full Name: Daniel Fehse
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by dfehse »

HI everyone,

we opened a case with Veeam and they asked us to tried to reproduce the isse by making a copy of a VM that we had this problem. So we made replication every 15min of clone of this VM. Last friday we suddendly had problems with other VMs that freezing for a while, always when they make Volume Shadow Copy (Snapshot) for previews files. We also realized that it was not possible to access the console anymore, got error "Unable to connect to the MKS: The operation is not allowed in the current state". We don't know if this all might be the same Problem. VMWare realsede a patch for that and also update 1 is available.
http://kb.vmware.com/selfservice/micros ... Id=2063262
tarrach
Lurker
Posts: 2
Liked: never
Joined: Nov 11, 2010 10:44 am
Full Name: Volker Tarrach
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by tarrach »

Hi everyone,

I got the same problem today. During Replication snapshot creation stops at 95%. The only way to get the machine running, was tha above hint with esxcli -kill process.
Is there a possible solution for this problem? Will be a solution to install update 1 from esxi 5.5?
Any help are welcome.

Vmware Esxi 5.5 1474528
Veeam B&R 7.0 839
Volker Tarrach
Schäferbarthold GmbH
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by veremin »

Hi, Volker, Is the problem reproducible without Veeam presence? Thanks.
tarrach
Lurker
Posts: 2
Liked: never
Joined: Nov 11, 2010 10:44 am
Full Name: Volker Tarrach
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by tarrach »

Hi v.Eremin,

I didn't test it, because the problem occured the first time this night. For testing I took a snapshot from another machine, it worked without problem. In the meantime I update my hosts to U1. I hope it will solve the problem.

thanks
Volker Tarrach
Schäferbarthold GmbH
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by veremin »

If after installing corresponding update the issue persists (reproducible only in situations when Veeam is present), then, please, open a case with our support team and let them analyze the required logs, confirm your environment, etc.

Thanks.
Oletho
Enthusiast
Posts: 67
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by Oletho »

My customer opened an SR (#00542057) with Veeam, but they refused to do anything as "they just use the VMware API".

VMware support worked for months without results and then escalated the SR. The first thing that happened after that is we got pointed back back to Veeam as "manual snapshot works, so Veeam must do something wrong with the API".
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vsphere 5.5 snapshot creation hang at 95% and VM is froz

Post by foggy »

If manual snapshot can be created without any issues, I recommend asking your Veeam support engineer to escalate the case for further investigation.
Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Bing [Bot], Google Feedfetcher and 91 guests