Host-based backup of VMware vSphere VMs.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

No news. I will of course update this topic as soon as we get any material update from VMware.
law999
Service Provider
Posts: 10
Liked: never
Joined: Feb 20, 2017 2:34 pm
Full Name: James Law
Contact:

Re: vSphere 6.5U2 support

Post by law999 »

We will need to patch our hosts to the latest due to the "L1 Terminal Fault"

Gostev, What would be the recommendation. go straight to 6.7?
btmaus
Expert
Posts: 138
Liked: 10 times
Joined: Jul 17, 2015 9:02 am
Full Name: Glenn L
Contact:

[MERGED] A question on version compatibility

Post by btmaus »

Hello,

We are currently on vSphere Appliance 6.5 U1 and want to update to U2c.
Our ESXI is currently on 6.5 build 73886907.
We are running Veeam 9.5 3a.

We need to upgrade our VMware ESXI hosts and vCenter servers to the latest version of 6.5.
I would like to confirm whether this is compatible with vSphere vCenter Server Appliance 6.5 U2c build 9451637?
The ESXI version that we want to upgrade to along with vCenter will be 6.5 U2 build 9298722.

I found this article from Veeam:
https://www.veeam.com/kb2443

It means that there is support and compatibility however there is an exception to this.
Can you please provide more information around the exception and confirm whether the above mentioned versions are compatible and supported?

Thanks

Case ID: #03170017
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: A question on version compatibility

Post by DGrinev »

Hi Glenn,

There is still an issue with the vSphere 6.5 U2 on the VMware side, it would be better to wait.
Please read this discussion for additional information. Thanks!
btmaus
Expert
Posts: 138
Liked: 10 times
Joined: Jul 17, 2015 9:02 am
Full Name: Glenn L
Contact:

Re: vSphere 6.5U2 support

Post by btmaus »

Thanks for the reply DGrinev.

Is there any "official" KB article from Veeam or VMware, that specifics what the problem is and what issues are experienced?
Would it crash a VMware host for example, or does it only affect Veeam services?

I'm trying to find out what exactly "doesn't work" or is affected, as our management team requires the information for CAB changes.

Thanks!
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: vSphere 6.5U2 support

Post by DGrinev » 1 person likes this post

Hi Glenn,

I ran through the topic and here you can find the information about the issue:
1. Issues confirmed by tests
2. A brief explanation by Gostev
3. Another description by Gostev

Thanks!
williampu
Influencer
Posts: 14
Liked: never
Joined: Sep 08, 2017 1:26 am
Full Name: William Pu
Contact:

Re: vSphere 6.5U2 support

Post by williampu »

@jamcool
The esxi hosts were rolled back to the previous version and everything worked.
I've asked the VMware guys to delay upgrades to 6.5U2 until I get an all-clear from Veeam.
Hopefully it won't take too long until a hotfix is available.
late
Influencer
Posts: 12
Liked: never
Joined: Mar 10, 2015 12:43 pm
Full Name: Late
Contact:

Re: vSphere 6.5U2 support

Post by late »

Thanks for the clarification DGrinev. We are in a process of updating our hosts from 5.5U2 to a newer version. It would make sense for us to go to version 6.7 but the problem is that we have couple of hosts that only support ESXi up to 6.5U2. So 6.7 isn't really an option for us. About the issues described in this thread: have you been able to resolve any of them or are they all related to VMware and you are waiting for their actions? And have the issues affected multiple Veeam users or are they all more or less individual cases? Thanks!
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

Late, the issue is related to a regression in vSphere 6.5 U2, so we're waiting for VMware to recognize and fix one.
The issue is isolated to over-provisioned environments with heavily loaded hosts (so more or less individual cases).
late
Influencer
Posts: 12
Liked: never
Joined: Mar 10, 2015 12:43 pm
Full Name: Late
Contact:

Re: vSphere 6.5U2 support

Post by late »

Hi Gostev and thank you for the information!
vDuck
Enthusiast
Posts: 84
Liked: 5 times
Joined: Apr 02, 2018 4:16 am
Full Name: Satoko Takahashi
Contact:

Re: vSphere 6.5U2 support

Post by vDuck » 2 people like this post

bthorne
Novice
Posts: 5
Liked: never
Joined: Jul 31, 2017 5:27 am
Full Name: Bernard Thorne
Contact:

[MERGED] Is it safe to Upgrade vSphere 6.5U2

Post by bthorne »

Hello Veeam,

Can we please get some offical details on the current issues customers are facing if they update.

Veeam version : 9.5 update 3 and will upgrade to 3a
Vmware version: 6.5

1. since we do SAN direct are we impacted ?
2. if you dont upgrade vCentre but upgrade the hosts to 6.5 U2 are we ok?
3.Have many customers had issues with the upgrade ?
4: is there any more info out there about this ?

Thank you very much

The kb arcticle still only has the original information of https://www.veeam.com/kb2443.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vSphere 6.5U2 support

Post by foggy »

Hi Bernard, please review information in this thread for details. Thanks.
bhnelson
Enthusiast
Posts: 34
Liked: 9 times
Joined: Oct 01, 2018 5:32 pm
Full Name: Brian Nelson
Contact:

Re: vSphere 6.5U2 support

Post by bhnelson » 1 person likes this post

Mermaid wrote: Sep 13, 2018 4:41 am FYI

https://kb.vmware.com/s/article/56453
This kb seems to describe the exact problem I'm having. However, it does not sound at all like the issue(s) described in this thread (random failures based on high host load).
It also says it affects both 6.5 U2 and 6.7 while Veeam says 6.7 is fine.
Can someone comment further?

Thanks!
rimvydukas
Enthusiast
Posts: 94
Liked: 6 times
Joined: Jun 04, 2013 8:15 am
Contact:

Re: vSphere 6.5U2 support

Post by rimvydukas »

Hi,

Dear Gostev. I just started migration to 6.5 version. I choosed to use 6.5 U1g, build 7388607. Is this build safe from the CPU regression issue? Was it solely brought by 6.5U2?
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

Yes, 6.5 U2 is the only version that is known to be affected. Thanks!
evilaedmin
Expert
Posts: 176
Liked: 30 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: vSphere 6.5U2 support

Post by evilaedmin »

Our environment has many disks per VM and we've experienced this known issue with ESXi 6.5 Update 2: https://kb.vmware.com/s/article/2146829 Is anyone else struggling with it? Support says it will be resolved in 6.7 Update 1.
bpayne
Enthusiast
Posts: 55
Liked: 12 times
Joined: Jan 20, 2015 2:07 pm
Full Name: Brandon Payne
Contact:

Re: vSphere 6.5U2 support

Post by bpayne » 1 person likes this post

Accidentally neglected to see this forum thread before building a new large cluster of ESXi hosts at 6.5 Update2c. Also experiencing all of my Veeam Proxy VM's being marked as invalid. Manual procedure: Remove VM from inventory, Re-add VM back to inventory, Edit settings: remove hot-added disks, then I can power on proxy VM. Happening to all my proxy VM's in this cluster once every 1-2 days. Cluster size has several hundred VM's.

We can't go to vCenter/ESXi 6.7 yet. Also getting pushed to patch to Update 2c for "esx.problem.hyperthreading.unmitigated" (L1 Terminal Fault - VMM’ (L1TF - VMM) Speculative-Execution vulnerability, https://kb.vmware.com/s/article/55806). In a really tough spot.

Does anyone know if switching those Proxy VM's from Hot-Add to Network mode yield better results?
Only thing I can think of is to build many more Proxy VM's to give myself a better chance of more proxies to pick from to get through the night.
bhnelson
Enthusiast
Posts: 34
Liked: 9 times
Joined: Oct 01, 2018 5:32 pm
Full Name: Brian Nelson
Contact:

Re: vSphere 6.5U2 support

Post by bhnelson » 1 person likes this post

Mermaid wrote: Sep 13, 2018 4:41 am FYI

https://kb.vmware.com/s/article/56453
FYI, I was told this by VMware support:
Regarding the issue that is addressed in the KB 56453, the fix should be out in the next patch, which will be released by the end of November 2018.
evilaedmin
Expert
Posts: 176
Liked: 30 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: vSphere 6.5U2 support

Post by evilaedmin »

bpayne wrote: Oct 03, 2018 2:33 pm Accidentally neglected to see this forum thread before building a new large cluster of ESXi hosts at 6.5 Update2c. Also experiencing all of my Veeam Proxy VM's being marked as invalid.
Have you looked into reverting? If you have done updates as directed and not something funky like we did (attempt the same update twice) the previous version should be right there on disk for you to select from the boot loader: https://kb.vmware.com/s/article/1033604
jamcool
Enthusiast
Posts: 67
Liked: 11 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: vSphere 6.5U2 support

Post by jamcool »

2 months ago we switched to network mode from Hot Add for proxy servers and have been very happy. It eliminated the proxy servers going invalid and cut out backup times by as much as 75% (we had two large jobs that went from nearly 8 hours to 2 hour each). our configuration is ESXi 6.5 update2 and Veeam 9.5 with 3a patch since July. I am not certain why if you have 10 Gbps networking that is not highly utilized would not use network mode.
lroscoe
Influencer
Posts: 14
Liked: 1 time
Joined: May 17, 2017 4:55 pm
Full Name: Leo Roscoe
Contact:

Re: vSphere 6.5U2 support

Post by lroscoe »

Any updates from VMware concerning this issue mention below?

"VMware vSphere 6.5 U2. This update introduces preliminary support by addressing all outstanding U2-specific compatibility issues that can be managed from the Veeam side. However, there is a major regression in ESXi 6.5 U2 code that makes the vSphere API fail randomly during high host CPU load periods, consequently impacting a variety of Veeam Backup & Replication functionality. VMware is currently troubleshooting this bug, and fixing one will most likely require the new ESXi 6.5 U2 build issued."
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

No updates so far. Some senior product person from VMware has reached out to me about a month ago with the seemingly strong desire to get to the bottom of this issue, but sadly I have not heard back ever since... so, it is starting to look like upgrading to 6.7 will be the only "solution" to this issue.
mgd5
Enthusiast
Posts: 28
Liked: 2 times
Joined: Oct 05, 2012 6:36 am
Full Name: Christer Sundqvist
Contact:

Re: vSphere 6.5U2 support

Post by mgd5 »

Hi,

We are in the same boat with 6.5 U2 and Veeam U3. We have 6 Proxy servers but one proxy in particular had problems. When it was running jobs it failed 1-5 vm per job but also the other Proxys didnt work well. So i have turned that Proxy off and now almost all jobs run. This weekend i had only one Proxy host go in invalid state and we have atleast 100 backup jobs around 500 vm´s running every night. How can that be ?

/Christer
jcapone
Influencer
Posts: 15
Liked: 1 time
Joined: Mar 01, 2016 9:28 pm
Full Name: John Capone
Contact:

Re: vSphere 6.5U2 support

Post by jcapone »

We are on Veeam 9.5 U3a and also recently upgraded to ESXi 6.5 U2 and are now kind of stuck here. Our hardware does not support 6.7, so we cannot migrate forward, so unless VMWare locates and resolves the problem with a U2 x patch, we will be forced to downgrade to 6.5 U1. Fortunately we are a small shop (40+ VMs, 2 hosts) and have not seen any problems yet. We are really trying to decide the best course of action here. Any advice? We opened our own ticket with VMWare (case #18947155010) but the engineer indicated that so far they do not see any global issue, that the problem is only seen in individual cases and as such will not likely be patched. Does anyone else have a case open with VMWare that they can share so we can ask the engineer to look at more details of the problem? As I said, since we are fortunate enough to not be experiencing the problem, I have no technical details to supply and don't know if we should stay on U2 or downgrade. Thanks for any input.
TrevorBell
Veteran
Posts: 364
Liked: 17 times
Joined: Feb 13, 2009 10:13 am
Full Name: Trevor Bell
Location: Worcester UK
Contact:

Re: vSphere 6.5U2 support

Post by TrevorBell » 1 person likes this post

I run 6.5U2 VMware ESXi, 6.5.0, 9298722 latest build with 200 VMs 100 per site and dont see any issues yet with backups or cross site replication. I only have 1 dedicated proxy per site win 2016 VM , with 10gig iscsi and Eqallogic 6210 boxes. ( all dell kit )

No issues soo far.. but will report back if i see anything strange
jamcool
Enthusiast
Posts: 67
Liked: 11 times
Joined: Feb 02, 2018 7:56 pm
Full Name: Jason Mount
Contact:

Re: vSphere 6.5U2 support

Post by jamcool »

jcapone wrote: Oct 10, 2018 5:45 pm We are on Veeam 9.5 U3a and also recently upgraded to ESXi 6.5 U2 and are now kind of stuck here. Our hardware does not support 6.7, so we cannot migrate forward, so unless VMWare locates and resolves the problem with a U2 x patch, we will be forced to downgrade to 6.5 U1. Fortunately we are a small shop (40+ VMs, 2 hosts) and have not seen any problems yet. We are really trying to decide the best course of action here. Any advice? We opened our own ticket with VMWare (case #18947155010) but the engineer indicated that so far they do not see any global issue, that the problem is only seen in individual cases and as such will not likely be patched. Does anyone else have a case open with VMWare that they can share so we can ask the engineer to look at more details of the problem? As I said, since we are fortunate enough to not be experiencing the problem, I have no technical details to supply and don't know if we should stay on U2 or downgrade. Thanks for any input.
Have you tried switching to network mode instead instead of Hot add?
jcapone
Influencer
Posts: 15
Liked: 1 time
Joined: Mar 01, 2016 9:28 pm
Full Name: John Capone
Contact:

Re: vSphere 6.5U2 support

Post by jcapone »

jamcool I saw a mention of that in one of the replies, but I don't know much about that mode, so I have not tried that yet. I'll check the documentation and see if it's something we can use here. Thanks!
rmyers
Lurker
Posts: 1
Liked: 1 time
Joined: Oct 18, 2018 6:44 pm
Full Name: Ryan Myers
Contact:

Re: vSphere 6.5U2 support

Post by rmyers » 1 person likes this post

I stumbled upon this thread while researching this issue. I also have a Veeam case opened and will probably open a VMware one as well.

@jcapone - Our environment consists of a couple hundred ESXi hosts, most of which I've just upgraded to 6.5U2 from 5.5U3. Within the last couple months aligning with the upgrade project I started experiencing the exact symptoms described in https://kb.vmware.com/s/article/56453 which I have since confirmed in vkernel.log. We have 22 vproxy servers and I tend to lose anywhere from 0 to 10 a night as backups run. Fixing the proxies is a real hassle (remove/re add to inventory, disconnect hot add disks, power on and answer questions, rescan from veeam, re add proxies to Keep Apart DRS Rules since dropping out of vcenter clears those out).

I recently put a script together to do a vMotion of all the proxy servers at end of day but it's too soon to tell if that's helping although I've had no invalids pop from last nights backup. I'll continue to monitor this thread to see if any other solutions are posted.

To the folks who mentioned use network mode over hot add - I would have to better understand if there is a performance hit doing so considering the size of our environment and the number of jobs I have running. I would expect proxy hot add to be the most efficient way to complete the backups.
ctsherman
Lurker
Posts: 1
Liked: never
Joined: Oct 21, 2018 12:21 am
Full Name: Christop[her Sherman
Contact:

Re: vSphere 6.5U2 support

Post by ctsherman »

Just to check/clarify on this...

Is this issue only affecting hotadd/proxyVM environments? I manage the network for a small business (three ESXi servers, one of which is offsite), and we're presently using a combination of NFS and network mode. In absence of hotadd or proxyVMs, will this regression issue affect us?
Post Reply

Who is online

Users browsing this forum: No registered users and 69 guests