Discussions specific to the VMware vSphere hypervisor
Post Reply
Gostev
SVP, Product Management
Posts: 23600
Liked: 3112 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Oct 02, 2018 10:32 am

Yes, 6.5 U2 is the only version that is known to be affected. Thanks!

evilaedmin
Enthusiast
Posts: 83
Liked: 12 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: vSphere 6.5U2 support

Post by evilaedmin » Oct 02, 2018 11:40 am

Our environment has many disks per VM and we've experienced this known issue with ESXi 6.5 Update 2: https://kb.vmware.com/s/article/2146829 Is anyone else struggling with it? Support says it will be resolved in 6.7 Update 1.

bpayne
Enthusiast
Posts: 52
Liked: 12 times
Joined: Jan 20, 2015 2:07 pm
Full Name: Brandon Payne
Contact:

Re: vSphere 6.5U2 support

Post by bpayne » Oct 03, 2018 2:33 pm 1 person likes this post

Accidentally neglected to see this forum thread before building a new large cluster of ESXi hosts at 6.5 Update2c. Also experiencing all of my Veeam Proxy VM's being marked as invalid. Manual procedure: Remove VM from inventory, Re-add VM back to inventory, Edit settings: remove hot-added disks, then I can power on proxy VM. Happening to all my proxy VM's in this cluster once every 1-2 days. Cluster size has several hundred VM's.

We can't go to vCenter/ESXi 6.7 yet. Also getting pushed to patch to Update 2c for "esx.problem.hyperthreading.unmitigated" (L1 Terminal Fault - VMM’ (L1TF - VMM) Speculative-Execution vulnerability, https://kb.vmware.com/s/article/55806). In a really tough spot.

Does anyone know if switching those Proxy VM's from Hot-Add to Network mode yield better results?
Only thing I can think of is to build many more Proxy VM's to give myself a better chance of more proxies to pick from to get through the night.

bhnelson
Lurker
Posts: 2
Liked: 2 times
Joined: Oct 01, 2018 5:32 pm
Full Name: Brian Nelson
Contact:

Re: vSphere 6.5U2 support

Post by bhnelson » Oct 03, 2018 2:35 pm 1 person likes this post

Mermaid wrote:
Sep 13, 2018 4:41 am
FYI

https://kb.vmware.com/s/article/56453
FYI, I was told this by VMware support:
Regarding the issue that is addressed in the KB 56453, the fix should be out in the next patch, which will be released by the end of November 2018.

evilaedmin
Enthusiast
Posts: 83
Liked: 12 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: vSphere 6.5U2 support

Post by evilaedmin » Oct 04, 2018 11:15 pm

bpayne wrote:
Oct 03, 2018 2:33 pm
Accidentally neglected to see this forum thread before building a new large cluster of ESXi hosts at 6.5 Update2c. Also experiencing all of my Veeam Proxy VM's being marked as invalid.
Have you looked into reverting? If you have done updates as directed and not something funky like we did (attempt the same update twice) the previous version should be right there on disk for you to select from the boot loader: https://kb.vmware.com/s/article/1033604

jamcool
Influencer
Posts: 10
Liked: 3 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: vSphere 6.5U2 support

Post by jamcool » Oct 08, 2018 12:30 am

2 months ago we switched to network mode from Hot Add for proxy servers and have been very happy. It eliminated the proxy servers going invalid and cut out backup times by as much as 75% (we had two large jobs that went from nearly 8 hours to 2 hour each). our configuration is ESXi 6.5 update2 and Veeam 9.5 with 3a patch since July. I am not certain why if you have 10 Gbps networking that is not highly utilized would not use network mode.

lroscoe
Novice
Posts: 7
Liked: 1 time
Joined: May 17, 2017 4:55 pm
Full Name: Leo Roscoe
Contact:

Re: vSphere 6.5U2 support

Post by lroscoe » Oct 08, 2018 3:14 pm

Any updates from VMware concerning this issue mention below?

"VMware vSphere 6.5 U2. This update introduces preliminary support by addressing all outstanding U2-specific compatibility issues that can be managed from the Veeam side. However, there is a major regression in ESXi 6.5 U2 code that makes the vSphere API fail randomly during high host CPU load periods, consequently impacting a variety of Veeam Backup & Replication functionality. VMware is currently troubleshooting this bug, and fixing one will most likely require the new ESXi 6.5 U2 build issued."

Gostev
SVP, Product Management
Posts: 23600
Liked: 3112 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev » Oct 09, 2018 10:10 am

No updates so far. Some senior product person from VMware has reached out to me about a month ago with the seemingly strong desire to get to the bottom of this issue, but sadly I have not heard back ever since... so, it is starting to look like upgrading to 6.7 will be the only "solution" to this issue.

mgd5
Influencer
Posts: 24
Liked: never
Joined: Oct 05, 2012 6:36 am
Full Name: Christer Sundqvist
Contact:

Re: vSphere 6.5U2 support

Post by mgd5 » Oct 10, 2018 11:58 am

Hi,

We are in the same boat with 6.5 U2 and Veeam U3. We have 6 Proxy servers but one proxy in particular had problems. When it was running jobs it failed 1-5 vm per job but also the other Proxys didnt work well. So i have turned that Proxy off and now almost all jobs run. This weekend i had only one Proxy host go in invalid state and we have atleast 100 backup jobs around 500 vm´s running every night. How can that be ?

/Christer

jcapone
Novice
Posts: 8
Liked: 1 time
Joined: Mar 01, 2016 9:28 pm
Full Name: John Capone
Contact:

Re: vSphere 6.5U2 support

Post by jcapone » Oct 10, 2018 5:45 pm

We are on Veeam 9.5 U3a and also recently upgraded to ESXi 6.5 U2 and are now kind of stuck here. Our hardware does not support 6.7, so we cannot migrate forward, so unless VMWare locates and resolves the problem with a U2 x patch, we will be forced to downgrade to 6.5 U1. Fortunately we are a small shop (40+ VMs, 2 hosts) and have not seen any problems yet. We are really trying to decide the best course of action here. Any advice? We opened our own ticket with VMWare (case #18947155010) but the engineer indicated that so far they do not see any global issue, that the problem is only seen in individual cases and as such will not likely be patched. Does anyone else have a case open with VMWare that they can share so we can ask the engineer to look at more details of the problem? As I said, since we are fortunate enough to not be experiencing the problem, I have no technical details to supply and don't know if we should stay on U2 or downgrade. Thanks for any input.

TrevorBell
Expert
Posts: 353
Liked: 7 times
Joined: Feb 13, 2009 10:13 am
Full Name: Trevor Bell
Location: Worcester UK
Contact:

Re: vSphere 6.5U2 support

Post by TrevorBell » Oct 12, 2018 7:11 am 1 person likes this post

I run 6.5U2 VMware ESXi, 6.5.0, 9298722 latest build with 200 VMs 100 per site and dont see any issues yet with backups or cross site replication. I only have 1 dedicated proxy per site win 2016 VM , with 10gig iscsi and Eqallogic 6210 boxes. ( all dell kit )

No issues soo far.. but will report back if i see anything strange

jamcool
Influencer
Posts: 10
Liked: 3 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: vSphere 6.5U2 support

Post by jamcool » Oct 15, 2018 1:46 am

jcapone wrote:
Oct 10, 2018 5:45 pm
We are on Veeam 9.5 U3a and also recently upgraded to ESXi 6.5 U2 and are now kind of stuck here. Our hardware does not support 6.7, so we cannot migrate forward, so unless VMWare locates and resolves the problem with a U2 x patch, we will be forced to downgrade to 6.5 U1. Fortunately we are a small shop (40+ VMs, 2 hosts) and have not seen any problems yet. We are really trying to decide the best course of action here. Any advice? We opened our own ticket with VMWare (case #18947155010) but the engineer indicated that so far they do not see any global issue, that the problem is only seen in individual cases and as such will not likely be patched. Does anyone else have a case open with VMWare that they can share so we can ask the engineer to look at more details of the problem? As I said, since we are fortunate enough to not be experiencing the problem, I have no technical details to supply and don't know if we should stay on U2 or downgrade. Thanks for any input.
Have you tried switching to network mode instead instead of Hot add?

jcapone
Novice
Posts: 8
Liked: 1 time
Joined: Mar 01, 2016 9:28 pm
Full Name: John Capone
Contact:

Re: vSphere 6.5U2 support

Post by jcapone » Oct 15, 2018 6:49 pm

jamcool I saw a mention of that in one of the replies, but I don't know much about that mode, so I have not tried that yet. I'll check the documentation and see if it's something we can use here. Thanks!

rmyers
Lurker
Posts: 1
Liked: 1 time
Joined: Oct 18, 2018 6:44 pm
Full Name: Ryan Myers
Contact:

Re: vSphere 6.5U2 support

Post by rmyers » Oct 18, 2018 6:59 pm 1 person likes this post

I stumbled upon this thread while researching this issue. I also have a Veeam case opened and will probably open a VMware one as well.

@jcapone - Our environment consists of a couple hundred ESXi hosts, most of which I've just upgraded to 6.5U2 from 5.5U3. Within the last couple months aligning with the upgrade project I started experiencing the exact symptoms described in https://kb.vmware.com/s/article/56453 which I have since confirmed in vkernel.log. We have 22 vproxy servers and I tend to lose anywhere from 0 to 10 a night as backups run. Fixing the proxies is a real hassle (remove/re add to inventory, disconnect hot add disks, power on and answer questions, rescan from veeam, re add proxies to Keep Apart DRS Rules since dropping out of vcenter clears those out).

I recently put a script together to do a vMotion of all the proxy servers at end of day but it's too soon to tell if that's helping although I've had no invalids pop from last nights backup. I'll continue to monitor this thread to see if any other solutions are posted.

To the folks who mentioned use network mode over hot add - I would have to better understand if there is a performance hit doing so considering the size of our environment and the number of jobs I have running. I would expect proxy hot add to be the most efficient way to complete the backups.

ctsherman
Lurker
Posts: 1
Liked: never
Joined: Oct 21, 2018 12:21 am
Full Name: Christop[her Sherman
Contact:

Re: vSphere 6.5U2 support

Post by ctsherman » Oct 21, 2018 12:29 am

Just to check/clarify on this...

Is this issue only affecting hotadd/proxyVM environments? I manage the network for a small business (three ESXi servers, one of which is offsite), and we're presently using a combination of NFS and network mode. In absence of hotadd or proxyVMs, will this regression issue affect us?

Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Brad.linch, tblime and 30 guests