-
- Service Provider
- Posts: 63
- Liked: 7 times
- Joined: Apr 04, 2011 8:56 am
- Full Name: Joern Westermann
- Contact:
VMs crashing after update to 5.5 Update 3
Hi,
after updating vSphere 5.5 to Update 3 we're encountering several VM crashes during snapshot removement.
VMware confirmed the bug and they're working on a solution. Currently they don't have a public KB entry.
You can also read more here: https://communities.vmware.com/thread/520379
So please wait with an upgrade if you still run 5.5.
Bye
Joern
after updating vSphere 5.5 to Update 3 we're encountering several VM crashes during snapshot removement.
VMware confirmed the bug and they're working on a solution. Currently they don't have a public KB entry.
You can also read more here: https://communities.vmware.com/thread/520379
So please wait with an upgrade if you still run 5.5.
Bye
Joern
-
- Chief Product Officer
- Posts: 31753
- Liked: 7257 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Hi, Joern. Thank you very very much for the heads up!
I really appreciate you letting everyone know about this issue.
I really appreciate you letting everyone know about this issue.
-
- Veeam Vanguard
- Posts: 629
- Liked: 251 times
- Joined: Sep 27, 2011 12:17 pm
- Full Name: Craig Dalrymple
- Location: Scotland
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Ouch sounds a sore one. As someone who was planning to upgrade to 5.5 update 3, id like to thank you for the heads up. Have shared via twitter to raise awareness.
-
- Expert
- Posts: 138
- Liked: 10 times
- Joined: Jul 17, 2015 9:02 am
- Full Name: Glenn L
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Thanks for sharing the info Joern, you saved us alot of hassle as we were looking to update as well!
-
- VeeaMVP
- Posts: 6163
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: VMs crashing after update to 5.5 Update 3
So basically 6.0 has bugs, 6.0 u1 has, now also 5.5 u3. QA lately has got some vacation down in Palo Alto...
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Veeam Legend
- Posts: 126
- Liked: 38 times
- Joined: Sep 26, 2013 8:40 am
- Full Name: Alessandro T.
- Location: Bologna, Italy
- Contact:
Re: VMs crashing after update to 5.5 Update 3
thank you! i was about to upgrade to U3, but i had the good idea to search the forum to see if Veeam already supports that.
What an ugly bug!!! There's really something wrong in VMware, today. Ok, probably i'd never had experienced the bug because i always keep VM in "/" folder in the datastore (see posts in the vmware community), but that's really bad.
What an ugly bug!!! There's really something wrong in VMware, today. Ok, probably i'd never had experienced the bug because i always keep VM in "/" folder in the datastore (see posts in the vmware community), but that's really bad.
Alessandro aka Tinto | VMCE 2024 | Veeam Legend | VCP-DCV 2023 | VVSPHT2023 | vExpert 2024
blog.tinivelli.com
blog.tinivelli.com
-
- Influencer
- Posts: 12
- Liked: 3 times
- Joined: Jan 17, 2015 7:16 pm
- Full Name: Patrik Kernstock
- Location: Austria, Lower Austria
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Luckily I updated my ESXi about three days ago from 5.5 U2 to U3... However, I was lucky and the bug didn't occured yet on my environment, because the VMs are also saved in the root-folder of the datastore like tinto1970 does it too.
There is a new interesting post in the VMware community thread:
There is a new interesting post in the VMware community thread:
(Source: https://communities.vmware.com/message/2543893#2543893)VMware Support-Team wrote:Last night engineering found the root cause. They will need to produce an express patch.
We have asked how long are we expecting to wait for this, I will follow up once a response has been received.
-
- Veeam Software
- Posts: 1485
- Liked: 653 times
- Joined: Jul 17, 2015 6:54 pm
- Full Name: Jorge de la Cruz
- Contact:
Re: VMs crashing after update to 5.5 Update 3
The VMware KB looks like a joke, but it's not. You need to rollback your ESXi Host - http://kb.vmware.com/selfservice/micros ... Id=2133118 Unless for now, I think it's the first time when I saw something like this from VMware, telling Costumers to rollback the entire ESXi.
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software
@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
Senior Product Manager | Veeam ONE @ Veeam Software
@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
-
- Enthusiast
- Posts: 98
- Liked: 12 times
- Joined: Mar 06, 2013 4:12 pm
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Veeam backups use snapshots, right? So would running a backup potentially result in this error? I have 6 production hosts, all of which were upgraded to U3 over the last week. Sigh.
-
- Chief Product Officer
- Posts: 31753
- Liked: 7257 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VMs crashing after update to 5.5 Update 3
That is correct.
-
- Enthusiast
- Posts: 98
- Liked: 12 times
- Joined: Mar 06, 2013 4:12 pm
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Unbelievable. I'm sorry, but heads should roll for this.
-
- Expert
- Posts: 227
- Liked: 62 times
- Joined: Apr 10, 2014 4:13 pm
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Yeah, this is just incredible. We've got 6 u1 with the NETDEV WATCHDOG issue, and 5.5 U3 with the crashing snapshots issue. Can't believe VMware's QA is so bad. 5.5 U3 also fixes security issues too, so if you back it out, then you've got known security problems.
Although it does appear that having folder on your datastores seem to be a cause, and I don't do that, I guess like most people.
Just upgraded to U3 as well this week too. Not seen the issue so far (touch wood).
Although it does appear that having folder on your datastores seem to be a cause, and I don't do that, I guess like most people.
Just upgraded to U3 as well this week too. Not seen the issue so far (touch wood).
-
- Service Provider
- Posts: 63
- Liked: 7 times
- Joined: Apr 04, 2011 8:56 am
- Full Name: Joern Westermann
- Contact:
Re: VMs crashing after update to 5.5 Update 3
We got information from VMware-support that they'll have Express Patch 8 ready on October 15th. 10 days to wait. :-/
-
- Veeam Legend
- Posts: 126
- Liked: 38 times
- Joined: Sep 26, 2013 8:40 am
- Full Name: Alessandro T.
- Location: Bologna, Italy
- Contact:
Re: VMs crashing after update to 5.5 Update 3
it would be nice to know if the vCenter will be updated too or not. I guess they will release only a patch for the ESXi but i'd like to be sure before performing the annoying vCenter update
Alessandro aka Tinto | VMCE 2024 | Veeam Legend | VCP-DCV 2023 | VVSPHT2023 | vExpert 2024
blog.tinivelli.com
blog.tinivelli.com
-
- VeeaMVP
- Posts: 6163
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: VMs crashing after update to 5.5 Update 3
So far seems the patch will target only ESXi, I don't see why vCenter would be involved. But obviously confirmation will come only when the patch will be available.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Sep 14, 2013 7:55 pm
- Full Name: Max Hetrick
- Contact:
Re: VMs crashing after update to 5.5 Update 3
I upgraded our ESXi and vSphere Center to 5.5 update 3 last week before seeing this, but have not had any trouble so far. We run everything across iSCSI and all virtual machines are stored within the root folder of each iSCSI target folder. We also don't have Enable VMware Tools quiescence turned on either. This thread suggests trying to turn quiescence off and see if that helps. We've never had that turned on, so maybe that's why we haven't experienced any issues. https://communities.vmware.com/thread/5 ... 0&tstart=0
-
- Expert
- Posts: 227
- Liked: 62 times
- Joined: Apr 10, 2014 4:13 pm
- Contact:
Re: VMs crashing after update to 5.5 Update 3
I was told we'd see a patch this week actually.
I'm still running Veeam with ESXi 5.5 U3, and I have not seen the issue so far thankfully. But I do agree with Gostev's comments on his weekly digest that VMware's QA recently has been awful.
I'm still running Veeam with ESXi 5.5 U3, and I have not seen the issue so far thankfully. But I do agree with Gostev's comments on his weekly digest that VMware's QA recently has been awful.
-
- Service Provider
- Posts: 26
- Liked: 4 times
- Joined: Dec 09, 2009 9:59 pm
- Full Name: James Sprinkle
- Contact:
Re: VMs crashing after update to 5.5 Update 3
I'm guessing the snapshot issue is caused by something in patch ESXi550-201509201-UG. The OpenSLP Remote Code Execution security issue is fixed in patch ESXi550-201509101-SG. I would assume the security patch can be applied separately, thus avoiding the snapshot issue. Can anyone confirm this?lightsout wrote:Yeah, this is just incredible. We've got 6 u1 with the NETDEV WATCHDOG issue, and 5.5 U3 with the crashing snapshots issue. Can't believe VMware's QA is so bad. 5.5 U3 also fixes security issues too, so if you back it out, then you've got known security problems.
-
- Veteran
- Posts: 377
- Liked: 85 times
- Joined: Mar 17, 2015 9:50 pm
- Full Name: Aemilianus Kehler
- Contact:
Re: VMs crashing after update to 5.5 Update 3
UPDATE...
This Snapshot bug has been fixed in 5.5u3a
Check it out here!
http://pubs.vmware.com/Release_Notes/en ... notes.html
http://kb.vmware.com/selfservice/micros ... Id=2133824
This Snapshot bug has been fixed in 5.5u3a
Check it out here!
http://pubs.vmware.com/Release_Notes/en ... notes.html
http://kb.vmware.com/selfservice/micros ... Id=2133824
-
- Expert
- Posts: 227
- Liked: 62 times
- Joined: Apr 10, 2014 4:13 pm
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Well spotted. I wonder what other issues this patch introduces? ;(
-
- VeeaMVP
- Posts: 6163
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Be careful what you wish for...
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Sep 17, 2015 2:47 pm
- Full Name: neil clarke
- Location: Derbyshire, UK
- Contact:
Re: VMs crashing after update to 5.5 Update 3
so whos installed it then?
the bug is only affecting one of our servers at the moment so i am holding off as long as i can.
the bug is only affecting one of our servers at the moment so i am holding off as long as i can.
-
- Influencer
- Posts: 17
- Liked: 1 time
- Joined: Aug 26, 2013 1:32 pm
- Full Name: Fredrik
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Installed it late last night on our internal infrastructure and reenabled all our backups.
No issues to report.
No issues to report.
-
- Expert
- Posts: 227
- Liked: 62 times
- Joined: Apr 10, 2014 4:13 pm
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Likewise, I upgraded from U3 last night too, and no issues so far.
-
- Veteran
- Posts: 377
- Liked: 85 times
- Joined: Mar 17, 2015 9:50 pm
- Full Name: Aemilianus Kehler
- Contact:
Re: VMs crashing after update to 5.5 Update 3
I'm here to report, I have updated vCenter 5.5u3 and updated my hosts to ESXi 5.5 u3a.
I didn't make any changes to my jobs, moved one VM into the new host, test a backup job.. worked as usual.
So far so good.
I didn't make any changes to my jobs, moved one VM into the new host, test a backup job.. worked as usual.
So far so good.
-
- Veeam Legend
- Posts: 126
- Liked: 38 times
- Joined: Sep 26, 2013 8:40 am
- Full Name: Alessandro T.
- Location: Bologna, Italy
- Contact:
Re: VMs crashing after update to 5.5 Update 3
ESXi 5.5 U3a released yesterday
Alessandro aka Tinto | VMCE 2024 | Veeam Legend | VCP-DCV 2023 | VVSPHT2023 | vExpert 2024
blog.tinivelli.com
blog.tinivelli.com
-
- Enthusiast
- Posts: 52
- Liked: 8 times
- Joined: May 14, 2012 10:48 am
- Contact:
Re: VMs crashing after update to 5.5 Update 3
So is ESXi 5.5 U3a safe to upgrade to now? We've been avoiding v6 due to the constant critical bugs that are found every week
-
- Influencer
- Posts: 12
- Liked: 3 times
- Joined: Jan 17, 2015 7:16 pm
- Full Name: Patrik Kernstock
- Location: Austria, Lower Austria
- Contact:
Re: VMs crashing after update to 5.5 Update 3
ESXi 5.5 U3a should be safe, yes. I didn't heard any issues about yet. However, I'm still running ESXi 5.5 U3 (without the patched snapshot bug) and even this is still running fine in my lab.
And: Still avoiding ESXi 6.0 may be an good idea, because: http://kb.vmware.com/selfservice/micros ... Id=2136854. This should also affect Veeam - as far as I know Veeam is also using CBT (Changed Block Tracking).
And: Still avoiding ESXi 6.0 may be an good idea, because: http://kb.vmware.com/selfservice/micros ... Id=2136854. This should also affect Veeam - as far as I know Veeam is also using CBT (Changed Block Tracking).
-
- Veeam Software
- Posts: 21133
- Liked: 2140 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Yes, Veeam B&R uses CBT and there's a dedicated thread regarding this issue.
-
- Enthusiast
- Posts: 52
- Liked: 8 times
- Joined: May 14, 2012 10:48 am
- Contact:
Re: VMs crashing after update to 5.5 Update 3
Yeah, I saw that in the forum digest email - Thanks Gostev!
I guess I'll give 5.5 U3a a go then, thanks for the info
I guess I'll give 5.5 U3a a go then, thanks for the info
Who is online
Users browsing this forum: No registered users and 48 guests