Community digest: vSphere 6.5 performance bug with PVSCSI

[DonZoomik]

I was wondering if the issue was huge IO latency spikes (500ms to 1s range) in Windows when UNMAP is invoked (and for a while after)?
I noticed them with my previous employer in April but left before this issue got anywhere. Then I fired up a brand new vSphere cluster last week and the same issue appeared. I was about to write a support case to VMware on Monday when the community digest arrived...

6.5, pvscsi, UNMAP - the stars seem to align.
This time I also took a look at esxtop and only symptom is that VM IOPS goes down considerably during and after UNMAP (corresponding with in-guest IO spikes), while other parameters see no major changes...

[johnlong]

With each one of the Community Digests that I read with some major flaw in vSphere 6.x, I'm constantly reminded why I'm still on 5.5. Though I know I'll eventually have to upgrade.

[DonZoomik]

True, but I was hoping that Gostev could shed some more light on the issue as I could find no other information other the basic fact in the newsletter.

[poulpreben]

Hi guys. I shared the info with Gostev, and my customer just received the official, private patch. They have been running successfully on a debug version of the patch for the past 10 days.

I will report back in this thread with the KB article, once it has been published by VMware.

[DonZoomik]

VMware support said that the public fix will probably be released on the last week of July.

[poulpreben]

The final patch has been issued to our customer, and it has been deployed on ESXi 6.5 EP2 (build 5310538). The patched build number is 5620045. As a final step in the troubleshooting process, the patch was installed on both a clean install of ESXi with patches, and an existing, previously upgraded ESXi host. The patch works successfully in both cases.

According to VMware Support, the patch is now confirmed to be included in ESXi 6.5U1 GA. We are still awaiting the KB article for detailed information about the actual root cause, and how to systematically detect the symptoms, but at least we have now successfully stabilized the environment.

[poulpreben]

I just received the following response, before they closed the ticket.

[...] they are reviewing a number of cases at present to ensure all observations are captured and correct.
The KB number will be 2150591 once published. As of now I do not have an ETA for publishing externally.

I'll keep an eye open for the published article.

[mwvme]

Someone accidentally share with me a comment that end of July might be a good time to expect a nice upgrade from VMware.

Michael

[poulpreben]

Hi everyone.

vSphere 6.5 Update 1 is now available.

Read more here > vSphere 6.5 Update 1 – Under the Hood

The KB article is also live > Performance issues on Windows virtual machine with hardware version 13 after upgrading to ESXi 6.5 (2150591)