vSphere 6.5U2 support

[Gostev]

In absence of hotadd or proxyVMs, will this regression issue affect us?

Yes. The only thing that seems to matters is your host load.

[madhon]

Just noticed VMware have released a new patch ESXi-6.5.0-20181004002-standard (Build 10390116) which includes the following in the release notes ..

PR 2209189: Heavy I/Os issued to a snapshot virtual machine using the SEsparse format might cause guest OS file system inconsistencies
Heavy I/Os issued to a snapshot virtual machine using the SEsparse format might cause guest OS file system inconsistencies, or data inconsistencies in applications.

Is this the esxi side fix for the regression ?

[Gostev]

No, it's a different regression that VMware has first disclosed just last week - we were not aware of it.

[bdufour]

im on 6.5.0 Update 2 with network mode and not experiencing these issues or any other issue related to veeam.

cheers!

[bpayne]

To the folks who mentioned use network mode over hot add - I would have to better understand if there is a performance hit doing so considering the size of our environment and the number of jobs I have running. I would expect proxy hot add to be the most efficient way to complete the backups.

@rmyers - I have a very similar environment. I bit the bullet last week and switched all my Veeam proxies to Network Mode only. I was pretty nervous doing this given the size of my environment, but all has been well the past 7 days. I did notice a decrease in performance though as anticipated so expect that your backup window might go a bit longer.

[lroscoe]

The following issue listed below just started to affect our environment. I see some response in the Veeam forums about a possible VMware patch being released by the end of November to address this issue. We've opened a case with VMware support and Veeam support but haven't gotten a concrete answer of when a patch will be released to address this issue. Any updates?

Backup Proxy VMs goes invalid and terminates during VM backup
https://kb.vmware.com/s/article/56453

[Gostev]

This looks to be a different 6.5 U2 issue from the "main" one that have been discussed in this thread so far. We have not been in contact with VMware on this one, so I don't know much about it at the moment. Except that it is certainly not a common issue in our support.

[anpa]

Hi guys,

Have you got any updates on this?
Do you have any figures on how many customers that have 6.5U2 are experiencing problems?
We are running metro cluster setup, so our hosts are never loaded to more than 50%, so if this is triggered by high load perhaps we could be safe?

/Andreas

[Gostev]

Hi - yes, you should be safe with your load. In general, I don't believe we have had many customers report being impacted by the issue. I think this is because when the issue does happen, automated retries just take care of it transparently. Thanks!

[anpa]

Hi again, and thanks for the prompt reply!

VMware seems to have released a new version of vCenter recently (well, yesterday);
https://docs.vmware.com/en/VMware-vSphe ... notes.html
There's also a new patched version of ESXi released:
https://docs.vmware.com/en/VMware-vSphe ... 11002.html

Seems like a couple of issues are fixed, not sure if they are affecting Veeam or not tho..

/Andreas

[Ollo]

Should be fixed.

https://docs.vmware.com/en/VMware-vSphe ... d-resolved

PR 2170126: Backup proxy virtual machines might go to invalid state during backup

[mdiver]

Could someone from Veeam please comment on this most recent VMWARE patch.
The changelog really reads as if the load issue has finally been taken care of.

Thanks,
Mike

[vDuck]

https://kb.vmware.com/s/article/56453

[Ollo]

Mermaid, the question was about the Heavy Load issue in overprovisioned environments in U2, not the hotadd issue.

[falkob]

Hi community,

i have customer who wants to upgrade his vSphere Infrastructure to vSphere 6.5 Update 2.
He is concerned about the "Known Issues" described in this article: https://www.veeam.com/kb2443

There are two Known Issues with VBR 9.5U3a and vSphere 6.5 Update 2

1. Supported with the exception of issues caused by a regression introduced in ESXi 6.5 U2 that causes its API to fail randomly under heavy host load.
2. There is a known issue where a Veeam Proxy running on a VM may become listed as (invalid) in vSphere. More information here: https://kb.vmware.com/s/article/56453

Issue number 2 won't have an impact on my customer, cause he is using physical Proxies with Direct Storage Access.

But in fact i wanted to know about Issue number 1. How frequent does this "occur" ? Are there any experiences on this ? What does heavy host load mean in this context ?

Thank you so far !

Best Regards

Falko

[foggy]

Hi Falko, please review this thread for more information. Thanks!

[Yuya]

UPDATE: Our auto-tests have exhibited one potentials issue with 6.5 U2 which may reduce reliability of a number of Veeam Backup & Replication features dependent on the corresponding protocol. Having said that, the issue is only reproducible in our stress testing lab where we emulate a heavy environment load, meaning for those who already upgraded these issue will likely exhibit "random" behavior which is in reality tied up to peak load in the environment. We're opening a support case with VMware - and I will keep you posted on the progress. Thanks!

1)Specifically, what kind of problems do you encounter?
For example, will the backup continue to fail until the load of ESXi goes down?
Or, because there is no reliability of the backup file, is there a possibility that it can not be restored?

2)I understand that VMware support recognizes that the cause of this problem is the vSphere API.
Do you have information on when to be fixed?

3)Is there a workaround for this problem?

I could not understand from the forum, so I'm glad if you tell me.

Thanks,
Yuya

[foggy]

I'll try to summarize what you can find in this thread above for you:

1)Specifically, what kind of problems do you encounter?

Like mentioned above, almost all product functionality might behave unreliably due to ESXi API failures during periods when the environment operates under heavy load.

For example, will the backup continue to fail until the load of ESXi goes down?

The issue has random nature but does not show up under normal load conditions, so if your hosts are not over-provisioned, you should be safe.

Or, because there is no reliability of the backup file, is there a possibility that it can not be restored?

This behavior does not cause any data corruptions, you just will not be able to perform some operations and, most likely, to restore from the backup files created during failed jobs. If subsequent job run doesn't produce any errors/warnings, the corresponding restore point should be restorable.

Do you have information on when to be fixed?

Currently we do not have any information in this regard.

3)Is there a workaround for this problem?

Currently the only workaround is to provide environment operation under normal load.

[Yuya]

Thank you for your answer.

This issue can occur when the load on the ESXi host is high.
I understood that this is the ESXi host where the virtual machine to be backed up resides, correct?
Does the vCenter load have an impact?

For example, suppose vCenter is created in a virtual machine and this vCenter VM is in ESXi 6.5 u2.
- The load on the ESXi host (A) where the virtual machine to be backed up resides is low.
- The load on the ESXi host (B) where the vCenter VM resides is high.
In this case, is there a possibility that this issue occurs?

Even if the load of the ESXi host where the vCenter VM resides is high, if the load of the ESXi host where the virtual machine to be backed up resides is low, this issue is unlikely to occur, correct?

Thanks,
Yuya

[Gostev]

Correct.

[skrause]

Is there a ballpark level of host resource consumption (CPU/RAM%) that this happens above but that would likely be safe if consumption stayed below?

We are unable to upgrade to 6.7 due to some of our storage arrays not supporting (or planning on supporting) anything above 6.5.

[Gostev]

All I know it's very high load, this was observed in our stress testing lab.

[fll]

Hello,
According to KB 2646, there are some problems between Veeam Backup & Replication 9.5 U3a and VSphere 6.5 U2.
"VMware vSphere 6.5 U2." This update introduces preliminary support by addressing all outstanding U2-specific compatibility issues that can be managed from the Veeam side, however, there is a major regression in ESXi 6.5 U2 code that makes the vSphere API fail randomly during high host CPU load periods, consequently impacting a variety of Veeam Backup & Replication functionality VMware is currently troubleshooting this bug, and fixing one will most likely require the new ESXi 6.5 U2 build issued. "
Does the same thing happen in ESXi 6.5 U1?

Thank you.

[wishr]

Hi Fll,

The given issue was discovered just in 6.5 U2 release bits, so it's safe to install 6.5 U1 for use with B&R 9.5 Update 2 or later versions. You can check that thread for additional information regarding 6.5 U1 support.

I've moved your post to an existing thread covering 6.5 U2 support topic.

Thanks

[perez]

- What about the possibility of the problem could be caused by the NFC limit, which it is not restricted by the moment in ESXi 6.5U2?
It looks that in ESXi 6.7, there is not created a new thread for each NFC connection, so the thread limitation won't be hit and consequently the problem looks not exist.
- Is it possible that even if VEEAM is NOT using the API, and instead performs a large number of NFC operations, more than 25 simultaneous NFC connections, the problems also occurs, because VEEAM is hitting an NFC limit?
-Has VEEAM tested if the problem persist if VEEAM client does not hit the limit of 25 simultaneous NFC connections, using the API?
-If the problem is finally related to the NFC connections, is it possible that the new VEEAM release, include any fix which could update the client in order to restrict the number of NFC connection, and solve the problem or you think that the fix would have to be in a newest ESXi release?

- And finally, could you please provide me, the SR number of the case you opened to VMWare?
Thanks

[wishr]

Hi Perez,

Please find the answers below.

What about the possibility of the problem could be caused by the NFC limit, which it is not restricted by the moment in ESXi 6.5U2?

No, unfortunately, the root of the issue cause was located on the VMware VDDK end that is used by B&R along with VMware API for performing backup and restore operations within VMware hypervisors. There were some memory leak issues in one of the VDDK versions used by B&R (just to let you know we deliver several VDDK libraries in each version to ensure compatibility and good customer experience because VDDK compatibility tests performed by VMware are far from being ideal).

Is it possible that even if VEEAM is NOT using the API, and instead performs a large number of NFC operations, more than 25 simultaneous NFC connections, the problems also occurs, because VEEAM is hitting an NFC limit?

No, please see above. NFC connections limit is well known here and always taken into account.

Has VEEAM tested if the problem persist if VEEAM client does not hit the limit of 25 simultaneous NFC connections, using the API?

Sure, this is tested frequently.

If the problem is finally related to the NFC connections, is it possible that the new VEEAM release, include any fix which could update the client in order to restrict the number of NFC connection, and solve the problem or you think that the fix would have to be in a newest ESXi release?

Sure, from one B&R version to another we update VDDK to the most stable versions and always deliver several VDDK versions in a single product bundle. Moreover, we test them internally, because VDDK issues are NOT something unusual. Regardless of that, this particular issue required changes in VDDK itself and on the ESXi end, so our ability to deal with it on our own was extremely limited.

The initial SR had a number 18797316305. After VDDK replacement the issue was fixed, but since that, we started observing another issue that itself disappeared after a couple updates to VDDK and ESXi servers in our labs that obviously required us to extensively communicate the issues, wait until VMware release their patches, test them internally, etc.

We'll update this thread with additional useful information once have it and it should be finally good news.

BR,
Fedor

[CPCS]

I would like to know what we could expect now from Update4 or Veeam or Vmware? Do we rely on VMware now to fix the left issues? Do everyone of us has to open a case for that so they will fix it or will they FINALLY do this after nearly one year everyone waiting for this?

This is really frustrating, sitting on ESXi 5.5 (already out of Support!) and not able to update (cause hardware is not supporting 6.7) and just looking at vmware/veeam/whoever not getting this fixed AFTER ALL THIS TIME...

[foggy]

U4 doesn't change anything in this regard, the ball is on VMware side. Why don't you consider going with whatever version between 5.5 and 6.2U2: 6.0/6.5/6.5U1? All of them are supported and 6.5U1 proved to be rock solid.

[CPCS]

Because as war as i understood, if you want to install latest security patches (which you should today) it is mandatory to update to 6.5U2 (coming from 6.5(U1), isnt it?

"the ball is on VMware side" does mean what exactly? Veeam does have an open ticket on vmware side for this issue?

[Gostev]

Yes, it is referenced by wishr in the post just above yours.