Host-based backup of VMware vSphere VMs.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

ctsherman wrote: Oct 21, 2018 12:29 amIn absence of hotadd or proxyVMs, will this regression issue affect us?
Yes. The only thing that seems to matters is your host load.
madhon
Lurker
Posts: 2
Liked: never
Joined: Jun 12, 2017 8:25 am
Full Name: Jonatho Douglas
Contact:

Re: vSphere 6.5U2 support

Post by madhon »

Just noticed VMware have released a new patch ESXi-6.5.0-20181004002-standard (Build 10390116) which includes the following in the release notes ..

PR 2209189: Heavy I/Os issued to a snapshot virtual machine using the SEsparse format might cause guest OS file system inconsistencies
Heavy I/Os issued to a snapshot virtual machine using the SEsparse format might cause guest OS file system inconsistencies, or data inconsistencies in applications.

Is this the esxi side fix for the regression ?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

No, it's a different regression that VMware has first disclosed just last week - we were not aware of it.
bdufour
Expert
Posts: 206
Liked: 41 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: vSphere 6.5U2 support

Post by bdufour »

im on 6.5.0 Update 2 with network mode and not experiencing these issues or any other issue related to veeam.

cheers!
bpayne
Enthusiast
Posts: 55
Liked: 12 times
Joined: Jan 20, 2015 2:07 pm
Full Name: Brandon Payne
Contact:

Re: vSphere 6.5U2 support

Post by bpayne »

rmyers wrote: Oct 18, 2018 6:59 pm To the folks who mentioned use network mode over hot add - I would have to better understand if there is a performance hit doing so considering the size of our environment and the number of jobs I have running. I would expect proxy hot add to be the most efficient way to complete the backups.
@rmyers - I have a very similar environment. I bit the bullet last week and switched all my Veeam proxies to Network Mode only. I was pretty nervous doing this given the size of my environment, but all has been well the past 7 days. I did notice a decrease in performance though as anticipated so expect that your backup window might go a bit longer.
lroscoe
Influencer
Posts: 14
Liked: 1 time
Joined: May 17, 2017 4:55 pm
Full Name: Leo Roscoe
Contact:

Re: vSphere 6.5U2 support

Post by lroscoe »

The following issue listed below just started to affect our environment. I see some response in the Veeam forums about a possible VMware patch being released by the end of November to address this issue. We've opened a case with VMware support and Veeam support but haven't gotten a concrete answer of when a patch will be released to address this issue. Any updates?

Backup Proxy VMs goes invalid and terminates during VM backup
https://kb.vmware.com/s/article/56453
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

This looks to be a different 6.5 U2 issue from the "main" one that have been discussed in this thread so far. We have not been in contact with VMware on this one, so I don't know much about it at the moment. Except that it is certainly not a common issue in our support.
anpa
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 27, 2011 8:42 am
Full Name: Andreas Cederlund
Location: Landskrona, Sweden
Contact:

Re: vSphere 6.5U2 support

Post by anpa »

Hi guys,

Have you got any updates on this?
Do you have any figures on how many customers that have 6.5U2 are experiencing problems?
We are running metro cluster setup, so our hosts are never loaded to more than 50%, so if this is triggered by high load perhaps we could be safe?

/Andreas
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

Hi - yes, you should be safe with your load. In general, I don't believe we have had many customers report being impacted by the issue. I think this is because when the issue does happen, automated retries just take care of it transparently. Thanks!
anpa
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 27, 2011 8:42 am
Full Name: Andreas Cederlund
Location: Landskrona, Sweden
Contact:

Re: vSphere 6.5U2 support

Post by anpa »

Hi again, and thanks for the prompt reply!

VMware seems to have released a new version of vCenter recently (well, yesterday);
https://docs.vmware.com/en/VMware-vSphe ... notes.html
There's also a new patched version of ESXi released:
https://docs.vmware.com/en/VMware-vSphe ... 11002.html

Seems like a couple of issues are fixed, not sure if they are affecting Veeam or not tho..

/Andreas
Ollo
Enthusiast
Posts: 32
Liked: 3 times
Joined: Jan 19, 2018 6:32 am
Contact:

Re: vSphere 6.5U2 support

Post by Ollo »

Should be fixed.

https://docs.vmware.com/en/VMware-vSphe ... d-resolved

PR 2170126: Backup proxy virtual machines might go to invalid state during backup
mdiver
Veeam Legend
Posts: 201
Liked: 33 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: vSphere 6.5U2 support

Post by mdiver »

Could someone from Veeam please comment on this most recent VMWARE patch.
The changelog really reads as if the load issue has finally been taken care of.

Thanks,
Mike
vDuck
Enthusiast
Posts: 84
Liked: 5 times
Joined: Apr 02, 2018 4:16 am
Full Name: Satoko Takahashi
Contact:

Re: vSphere 6.5U2 support

Post by vDuck »

Ollo
Enthusiast
Posts: 32
Liked: 3 times
Joined: Jan 19, 2018 6:32 am
Contact:

Re: vSphere 6.5U2 support

Post by Ollo »

Mermaid, the question was about the Heavy Load issue in overprovisioned environments in U2, not the hotadd issue.
falkob
Veeam Vanguard
Posts: 86
Liked: 31 times
Joined: Sep 28, 2017 7:47 am
Full Name: Falko Banaszak
Contact:

[MERGED] Veeam 9.5 Udpate 3a and vSphere 6.5 Update 2

Post by falkob »

Hi community,

i have customer who wants to upgrade his vSphere Infrastructure to vSphere 6.5 Update 2.
He is concerned about the "Known Issues" described in this article: https://www.veeam.com/kb2443

There are two Known Issues with VBR 9.5U3a and vSphere 6.5 Update 2

1. Supported with the exception of issues caused by a regression introduced in ESXi 6.5 U2 that causes its API to fail randomly under heavy host load.
2. There is a known issue where a Veeam Proxy running on a VM may become listed as (invalid) in vSphere. More information here: https://kb.vmware.com/s/article/56453


Issue number 2 won't have an impact on my customer, cause he is using physical Proxies with Direct Storage Access.

But in fact i wanted to know about Issue number 1. How frequent does this "occur" ? Are there any experiences on this ? What does heavy host load mean in this context ?

Thank you so far !

Best Regards

Falko
VCP6.5-DCV, VCP6-DCV, VMCE, VMCA, Veeam Vanguard, VMware vExpert
https://www.virtualhome.blog
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 9.5 Udpate 3a and vSphere 6.5 Update 2

Post by foggy » 1 person likes this post

Hi Falko, please review this thread for more information. Thanks!
Yuya
Enthusiast
Posts: 81
Liked: 2 times
Joined: Jan 30, 2013 9:32 am
Contact:

Re: vSphere 6.5U2 support

Post by Yuya »

UPDATE: Our auto-tests have exhibited one potentials issue with 6.5 U2 which may reduce reliability of a number of Veeam Backup & Replication features dependent on the corresponding protocol. Having said that, the issue is only reproducible in our stress testing lab where we emulate a heavy environment load, meaning for those who already upgraded these issue will likely exhibit "random" behavior which is in reality tied up to peak load in the environment. We're opening a support case with VMware - and I will keep you posted on the progress. Thanks!
1)Specifically, what kind of problems do you encounter?
For example, will the backup continue to fail until the load of ESXi goes down?
Or, because there is no reliability of the backup file, is there a possibility that it can not be restored?

2)I understand that VMware support recognizes that the cause of this problem is the vSphere API.
Do you have information on when to be fixed?

3)Is there a workaround for this problem?

I could not understand from the forum, so I'm glad if you tell me.

Thanks,
Yuya
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vSphere 6.5U2 support

Post by foggy » 1 person likes this post

I'll try to summarize what you can find in this thread above for you:
Yuya wrote:1)Specifically, what kind of problems do you encounter?
Like mentioned above, almost all product functionality might behave unreliably due to ESXi API failures during periods when the environment operates under heavy load.
Yuya wrote:For example, will the backup continue to fail until the load of ESXi goes down?
The issue has random nature but does not show up under normal load conditions, so if your hosts are not over-provisioned, you should be safe.
Yuya wrote:Or, because there is no reliability of the backup file, is there a possibility that it can not be restored?
This behavior does not cause any data corruptions, you just will not be able to perform some operations and, most likely, to restore from the backup files created during failed jobs. If subsequent job run doesn't produce any errors/warnings, the corresponding restore point should be restorable.
Yuya wrote:Do you have information on when to be fixed?
Currently we do not have any information in this regard.
Yuya wrote:3)Is there a workaround for this problem?
Currently the only workaround is to provide environment operation under normal load.
Yuya
Enthusiast
Posts: 81
Liked: 2 times
Joined: Jan 30, 2013 9:32 am
Contact:

Re: vSphere 6.5U2 support

Post by Yuya »

Thank you for your answer.

This issue can occur when the load on the ESXi host is high.
I understood that this is the ESXi host where the virtual machine to be backed up resides, correct?
Does the vCenter load have an impact?

For example, suppose vCenter is created in a virtual machine and this vCenter VM is in ESXi 6.5 u2.
- The load on the ESXi host (A) where the virtual machine to be backed up resides is low.
- The load on the ESXi host (B) where the vCenter VM resides is high.
In this case, is there a possibility that this issue occurs?

Even if the load of the ESXi host where the vCenter VM resides is high, if the load of the ESXi host where the virtual machine to be backed up resides is low, this issue is unlikely to occur, correct?

Thanks,
Yuya
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

Correct.
skrause
Veteran
Posts: 487
Liked: 105 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: vSphere 6.5U2 support

Post by skrause »

Is there a ballpark level of host resource consumption (CPU/RAM%) that this happens above but that would likely be safe if consumption stayed below?

We are unable to upgrade to 6.7 due to some of our storage arrays not supporting (or planning on supporting) anything above 6.5.
Steve Krause
Veeam Certified Architect
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

All I know it's very high load, this was observed in our stress testing lab.
fll
Enthusiast
Posts: 51
Liked: never
Joined: Sep 05, 2016 10:35 am
Contact:

[MERGED] Veeam Backup & Replication 9.5 Update 3a and VMware vSphere 6.5 U2.

Post by fll »

Hello,
According to KB 2646, there are some problems between Veeam Backup & Replication 9.5 U3a and VSphere 6.5 U2.
"VMware vSphere 6.5 U2." This update introduces preliminary support by addressing all outstanding U2-specific compatibility issues that can be managed from the Veeam side, however, there is a major regression in ESXi 6.5 U2 code that makes the vSphere API fail randomly during high host CPU load periods, consequently impacting a variety of Veeam Backup & Replication functionality VMware is currently troubleshooting this bug, and fixing one will most likely require the new ESXi 6.5 U2 build issued. "
Does the same thing happen in ESXi 6.5 U1?

Thank you.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: vSphere 6.5U2 support

Post by wishr »

Hi Fll,

The given issue was discovered just in 6.5 U2 release bits, so it's safe to install 6.5 U1 for use with B&R 9.5 Update 2 or later versions. You can check that thread for additional information regarding 6.5 U1 support.

I've moved your post to an existing thread covering 6.5 U2 support topic.

Thanks
perez
Lurker
Posts: 1
Liked: 1 time
Joined: Jan 14, 2019 9:14 am
Contact:

Re: vSphere 6.5U2 support

Post by perez » 1 person likes this post

- What about the possibility of the problem could be caused by the NFC limit, which it is not restricted by the moment in ESXi 6.5U2?
It looks that in ESXi 6.7, there is not created a new thread for each NFC connection, so the thread limitation won't be hit and consequently the problem looks not exist.
- Is it possible that even if VEEAM is NOT using the API, and instead performs a large number of NFC operations, more than 25 simultaneous NFC connections, the problems also occurs, because VEEAM is hitting an NFC limit?
-Has VEEAM tested if the problem persist if VEEAM client does not hit the limit of 25 simultaneous NFC connections, using the API?
-If the problem is finally related to the NFC connections, is it possible that the new VEEAM release, include any fix which could update the client in order to restrict the number of NFC connection, and solve the problem or you think that the fix would have to be in a newest ESXi release?

- And finally, could you please provide me, the SR number of the case you opened to VMWare?
Thanks
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: vSphere 6.5U2 support

Post by wishr »

Hi Perez,

Please find the answers below.
perez wrote:What about the possibility of the problem could be caused by the NFC limit, which it is not restricted by the moment in ESXi 6.5U2?
No, unfortunately, the root of the issue cause was located on the VMware VDDK end that is used by B&R along with VMware API for performing backup and restore operations within VMware hypervisors. There were some memory leak issues in one of the VDDK versions used by B&R (just to let you know we deliver several VDDK libraries in each version to ensure compatibility and good customer experience because VDDK compatibility tests performed by VMware are far from being ideal).
perez wrote:Is it possible that even if VEEAM is NOT using the API, and instead performs a large number of NFC operations, more than 25 simultaneous NFC connections, the problems also occurs, because VEEAM is hitting an NFC limit?
No, please see above. NFC connections limit is well known here and always taken into account.
perez wrote:Has VEEAM tested if the problem persist if VEEAM client does not hit the limit of 25 simultaneous NFC connections, using the API?
Sure, this is tested frequently.
perez wrote:If the problem is finally related to the NFC connections, is it possible that the new VEEAM release, include any fix which could update the client in order to restrict the number of NFC connection, and solve the problem or you think that the fix would have to be in a newest ESXi release?
Sure, from one B&R version to another we update VDDK to the most stable versions and always deliver several VDDK versions in a single product bundle. Moreover, we test them internally, because VDDK issues are NOT something unusual. Regardless of that, this particular issue required changes in VDDK itself and on the ESXi end, so our ability to deal with it on our own was extremely limited.

The initial SR had a number 18797316305. After VDDK replacement the issue was fixed, but since that, we started observing another issue that itself disappeared after a couple updates to VDDK and ESXi servers in our labs that obviously required us to extensively communicate the issues, wait until VMware release their patches, test them internally, etc.

We'll update this thread with additional useful information once have it and it should be finally good news.

BR,
Fedor
CPCS
Influencer
Posts: 17
Liked: 4 times
Joined: Mar 01, 2016 1:49 pm
Contact:

Re: vSphere 6.5U2 support

Post by CPCS »

I would like to know what we could expect now from Update4 or Veeam or Vmware? Do we rely on VMware now to fix the left issues? Do everyone of us has to open a case for that so they will fix it or will they FINALLY do this after nearly one year everyone waiting for this?

This is really frustrating, sitting on ESXi 5.5 (already out of Support!) and not able to update (cause hardware is not supporting 6.7) and just looking at vmware/veeam/whoever not getting this fixed AFTER ALL THIS TIME...
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: vSphere 6.5U2 support

Post by foggy »

U4 doesn't change anything in this regard, the ball is on VMware side. Why don't you consider going with whatever version between 5.5 and 6.2U2: 6.0/6.5/6.5U1? All of them are supported and 6.5U1 proved to be rock solid.
CPCS
Influencer
Posts: 17
Liked: 4 times
Joined: Mar 01, 2016 1:49 pm
Contact:

Re: vSphere 6.5U2 support

Post by CPCS »

Because as war as i understood, if you want to install latest security patches (which you should today) it is mandatory to update to 6.5U2 (coming from 6.5(U1), isnt it?

"the ball is on VMware side" does mean what exactly? Veeam does have an open ticket on vmware side for this issue?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: vSphere 6.5U2 support

Post by Gostev »

Yes, it is referenced by wishr in the post just above yours.
Post Reply

Who is online

Users browsing this forum: GregorS and 98 guests