Community digest: vSphere 6.5 performance bug with PVSCSI

VMware specific discussions

Community digest: vSphere 6.5 performance bug with PVSCSI

Veeam Logoby DonZoomik » Sun Jun 25, 2017 10:19 pm

I was wondering if the issue was huge IO latency spikes (500ms to 1s range) in Windows when UNMAP is invoked (and for a while after)?
I noticed them with my previous employer in April but left before this issue got anywhere. Then I fired up a brand new vSphere cluster last week and the same issue appeared. I was about to write a support case to VMware on Monday when the community digest arrived...

6.5, pvscsi, UNMAP - the stars seem to align. :D
This time I also took a look at esxtop and only symptom is that VM IOPS goes down considerably during and after UNMAP (corresponding with in-guest IO spikes), while other parameters see no major changes...
DonZoomik
Influencer
 
Posts: 20
Liked: 8 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Community digest: vSphere 6.5 performance bug with PVSCS

Veeam Logoby johnlong » Mon Jun 26, 2017 4:12 am 1 person likes this post

With each one of the Community Digests that I read with some major flaw in vSphere 6.x, I'm constantly reminded why I'm still on 5.5. Though I know I'll eventually have to upgrade. :shock:
johnlong
Enthusiast
 
Posts: 72
Liked: 11 times
Joined: Sat May 12, 2012 3:37 pm

Re: Community digest: vSphere 6.5 performance bug with PVSCS

Veeam Logoby DonZoomik » Mon Jun 26, 2017 6:29 pm

True, but I was hoping that Gostev could shed some more light on the issue as I could find no other information other the basic fact in the newsletter.
DonZoomik
Influencer
 
Posts: 20
Liked: 8 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Community digest: vSphere 6.5 performance bug with PVSCS

Veeam Logoby poulpreben » Tue Jun 27, 2017 9:19 am 3 people like this post

Hi guys. I shared the info with Gostev, and my customer just received the official, private patch. They have been running successfully on a debug version of the patch for the past 10 days.

I will report back in this thread with the KB article, once it has been published by VMware.
poulpreben
Expert
 
Posts: 935
Liked: 398 times
Joined: Mon Jul 23, 2012 8:16 am
Full Name: Preben Berg

Re: Community digest: vSphere 6.5 performance bug with PVSCS

Veeam Logoby DonZoomik » Thu Jun 29, 2017 1:40 pm 1 person likes this post

VMware support said that the public fix will probably be released on the last week of July.
DonZoomik
Influencer
 
Posts: 20
Liked: 8 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Community digest: vSphere 6.5 performance bug with PVSCS

Veeam Logoby poulpreben » Mon Jul 03, 2017 6:19 am 1 person likes this post

The final patch has been issued to our customer, and it has been deployed on ESXi 6.5 EP2 (build 5310538). The patched build number is 5620045. As a final step in the troubleshooting process, the patch was installed on both a clean install of ESXi with patches, and an existing, previously upgraded ESXi host. The patch works successfully in both cases.

According to VMware Support, the patch is now confirmed to be included in ESXi 6.5U1 GA. We are still awaiting the KB article for detailed information about the actual root cause, and how to systematically detect the symptoms, but at least we have now successfully stabilized the environment.
poulpreben
Expert
 
Posts: 935
Liked: 398 times
Joined: Mon Jul 23, 2012 8:16 am
Full Name: Preben Berg

Re: Community digest: vSphere 6.5 performance bug with PVSCS

Veeam Logoby poulpreben » Wed Jul 05, 2017 1:20 pm 3 people like this post

I just received the following response, before they closed the ticket.
VMware Support wrote:[...] they are reviewing a number of cases at present to ensure all observations are captured and correct.
The KB number will be 2150591 once published. As of now I do not have an ETA for publishing externally.

I'll keep an eye open for the published article.
poulpreben
Expert
 
Posts: 935
Liked: 398 times
Joined: Mon Jul 23, 2012 8:16 am
Full Name: Preben Berg

Re: Community digest: vSphere 6.5 performance bug with PVSCS

Veeam Logoby mwvme » Wed Jul 05, 2017 11:24 pm

Someone accidentally share with me a comment that end of July might be a good time to expect a nice upgrade from VMware.

Michael
mwvme
Veeam Software
 
Posts: 70
Liked: 18 times
Joined: Sat Dec 05, 2015 10:19 pm
Location: Calgary, Alberta Canada
Full Name: Michael White

Re: Community digest: vSphere 6.5 performance bug with PVSCS

Veeam Logoby poulpreben » Fri Jul 28, 2017 7:32 am

poulpreben
Expert
 
Posts: 935
Liked: 398 times
Joined: Mon Jul 23, 2012 8:16 am
Full Name: Preben Berg


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 21 guests