VMs crashing after update to 5.5 Update 3

VMware specific discussions

VMs crashing after update to 5.5 Update 3

Veeam Logoby JWester » Tue Sep 29, 2015 12:44 pm 6 people like this post

Hi,
after updating vSphere 5.5 to Update 3 we're encountering several VM crashes during snapshot removement.
VMware confirmed the bug and they're working on a solution. Currently they don't have a public KB entry.
You can also read more here: https://communities.vmware.com/thread/520379
So please wait with an upgrade if you still run 5.5.
Bye
Joern
JWester
Service Provider
 
Posts: 55
Liked: 7 times
Joined: Mon Apr 04, 2011 8:56 am
Full Name: Joern Westermann

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby Gostev » Tue Sep 29, 2015 10:21 pm

Hi, Joern. Thank you very very much for the heads up!
I really appreciate you letting everyone know about this issue.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby Cragdoo » Tue Sep 29, 2015 10:44 pm

Ouch sounds a sore one. As someone who was planning to upgrade to 5.5 update 3, id like to thank you for the heads up. Have shared via twitter to raise awareness.
Cragdoo
Veeam Vanguard
 
Posts: 408
Liked: 134 times
Joined: Tue Sep 27, 2011 12:17 pm
Location: Scotland
Full Name: Craig Dalrymple

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby btmaus » Wed Sep 30, 2015 12:26 am

Thanks for sharing the info Joern, you saved us alot of hassle as we were looking to update as well!
btmaus
Expert
 
Posts: 128
Liked: 9 times
Joined: Fri Jul 17, 2015 9:02 am
Full Name: Glenn L

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby dellock6 » Wed Sep 30, 2015 8:19 am 4 people like this post

So basically 6.0 has bugs, 6.0 u1 has, now also 5.5 u3. QA lately has got some vacation down in Palo Alto...
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby tinto1970 » Wed Sep 30, 2015 1:21 pm

thank you! i was about to upgrade to U3, but i had the good idea to search the forum to see if Veeam already supports that.

What an ugly bug!!! There's really something wrong in VMware, today. Ok, probably i'd never had experienced the bug because i always keep VM in "/" folder in the datastore (see posts in the vmware community), but that's really bad.
Alessandro Tinivelli aka Tinto
@tinto1970
tinto1970
Enthusiast
 
Posts: 69
Liked: 25 times
Joined: Thu Sep 26, 2013 8:40 am
Location: Bologna, Italy
Full Name: Alessandro Tinivelli

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby Patschi » Thu Oct 01, 2015 12:23 pm

Luckily I updated my ESXi about three days ago from 5.5 U2 to U3... However, I was lucky and the bug didn't occured yet on my environment, because the VMs are also saved in the root-folder of the datastore like tinto1970 does it too.

There is a new interesting post in the VMware community thread:
VMware Support-Team wrote:Last night engineering found the root cause. They will need to produce an express patch.
We have asked how long are we expecting to wait for this, I will follow up once a response has been received.

(Source: https://communities.vmware.com/message/2543893#2543893)
Patschi
Influencer
 
Posts: 12
Liked: 3 times
Joined: Sat Jan 17, 2015 7:16 pm
Location: Austria, Lower Austria
Full Name: Patrik Kernstock

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby jorgedlcruz » Thu Oct 01, 2015 10:02 pm 1 person likes this post

The VMware KB looks like a joke, but it's not. You need to rollback your ESXi Host - http://kb.vmware.com/selfservice/micros ... Id=2133118 Unless for now, I think it's the first time when I saw something like this from VMware, telling Costumers to rollback the entire ESXi.
jorgedlcruz
Veeam Vanguard
 
Posts: 105
Liked: 44 times
Joined: Fri Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz Mingo

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby guitarfish » Thu Oct 01, 2015 10:46 pm

Veeam backups use snapshots, right? So would running a backup potentially result in this error? I have 6 production hosts, all of which were upgraded to U3 over the last week. Sigh.
guitarfish
Enthusiast
 
Posts: 67
Liked: 2 times
Joined: Wed Mar 06, 2013 4:12 pm

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby Gostev » Fri Oct 02, 2015 12:52 am

That is correct.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby guitarfish » Fri Oct 02, 2015 1:05 am

Unbelievable. I'm sorry, but heads should roll for this.
guitarfish
Enthusiast
 
Posts: 67
Liked: 2 times
Joined: Wed Mar 06, 2013 4:12 pm

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby lightsout » Fri Oct 02, 2015 1:12 pm

Yeah, this is just incredible. We've got 6 u1 with the NETDEV WATCHDOG issue, and 5.5 U3 with the crashing snapshots issue. Can't believe VMware's QA is so bad. 5.5 U3 also fixes security issues too, so if you back it out, then you've got known security problems.

Although it does appear that having folder on your datastores seem to be a cause, and I don't do that, I guess like most people.

Just upgraded to U3 as well this week too. Not seen the issue so far (touch wood).
lightsout
Expert
 
Posts: 185
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby JWester » Mon Oct 05, 2015 7:29 am

We got information from VMware-support that they'll have Express Patch 8 ready on October 15th. 10 days to wait. :-/
JWester
Service Provider
 
Posts: 55
Liked: 7 times
Joined: Mon Apr 04, 2011 8:56 am
Full Name: Joern Westermann

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby tinto1970 » Mon Oct 05, 2015 8:01 am

it would be nice to know if the vCenter will be updated too or not. I guess they will release only a patch for the ESXi but i'd like to be sure before performing the annoying vCenter update ;)
Alessandro Tinivelli aka Tinto
@tinto1970
tinto1970
Enthusiast
 
Posts: 69
Liked: 25 times
Joined: Thu Sep 26, 2013 8:40 am
Location: Bologna, Italy
Full Name: Alessandro Tinivelli

Re: VMs crashing after update to 5.5 Update 3

Veeam Logoby dellock6 » Mon Oct 05, 2015 8:16 am

So far seems the patch will target only ESXi, I don't see why vCenter would be involved. But obviously confirmation will come only when the patch will be available.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Next

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot] and 13 guests