Management reporting and documentation
Post Reply
_james
Enthusiast
Posts: 30
Liked: 6 times
Joined: Nov 15, 2018 3:51 pm
Contact:

Maximum Datastore IOPS is lower from VM

Post by _james »

Hello everyone,

I have a confusing issue with Veeam One reports.
Every month I create custom report where I check TOP 10 VM IOPS and latency.
What confuses me if I compare TOP 10 VM numbers to Datastore highest latency and IOPS ones the datastore numbers are lower. Why is that? Shouldn't the Datastore maximum IOPS and Latency be atleast the same as the peak TOP-1 VM one (atleast because there will be other VMs that will generate IO).

Can someone advise why is that?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by wishr »

Hi _james,

Could you please provide at least a few screenshots from both reports for our better understanding?

Thanks in advance.
_james
Enthusiast
Posts: 30
Liked: 6 times
Joined: Nov 15, 2018 3:51 pm
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by _james » 1 person likes this post

Hello,

Yes, no problem.
When comparing vSphere Monitoring - Datastore Performance with vSphere Infrastructure Assessment - Datastore Performance assessment information about peak datastore IOPS the peak numbers are the same, no problem there:
Image
Image

However, as mentioned, when checking out vSphere Custom performance - Object type: Virtual Machines, Measured entities: Datastore and Read/Write IO and Latency we can see that some VMs produce higher IOs and latencies than Datastore performance and Performance assessment reported ones. Here are the write and read IOPS:
Image
Image

Timeframe - last week for top VMs (4.06 - 10.06), this month for Datastore performance and assessment.
_james
Enthusiast
Posts: 30
Liked: 6 times
Joined: Nov 15, 2018 3:51 pm
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by _james »

The only thing which is different is the timeframe. As you cannot set the same timeframes for DS performance and DS performance assessment as Custom VM IO metrics because of the the bug in V11 that shows only data for the last week (you can but only on sunday for the timeframe to be Last week.)
However this shouldn't be an issue because of the timeframes overlapping (This month for datastores / last week for the VMs)
nikolaj
Expert
Posts: 164
Liked: 57 times
Joined: Mar 22, 2021 11:19 am
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by nikolaj »

Hi _james,

Would you be able to provide a full excerpt of that report page while I investigate this on my lab environment?
In general, all performance data come directly from vSphere counters, so I don't expect there to be any surprises in terms of interpretation.
_james
Enthusiast
Posts: 30
Liked: 6 times
Joined: Nov 15, 2018 3:51 pm
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by _james »

Hello Nikolaj, sorry about previous screenshots, I created new ones.

This time - all timestamps are be equal. Range of 07.06.-13.06. for Custom VM metrics and DS performance. Interval - Current Week for DS Performance assessment (end time is in the "future", but that is the only way how to rule out timestamp inconsistencies, also it is not possible to set a specific range for DS Perf. assessment).

Heres what I found:
Lets take vSphere Datastore performance as a reference. As we can see for specified reporting period it reports max. IOPS of 6328
Image
Then we compare this result to Datastore performance assessment, timeframe is current week, but it has the same dates as the Range timeframe.
Image
However it shows much higher result:
Image

Heres what confuses me the most - VMs show higher IOPS than the DS Performance report, however this time the numbers do fall under the DS performance assessment report (DS max iops are higher than TOP1 VM IOPS which makes sense)
Image

Same applies to basically all storage metrics - rw iops, latencies etc.
Apologies for the lenghty posts, also if something is not understandable here, let me know i'll try to rephrase, provide additional data. Host, Datastore and VM names are cleared for obvious reasons.
nikolaj
Expert
Posts: 164
Liked: 57 times
Joined: Mar 22, 2021 11:19 am
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by nikolaj »

Hi _james,

Thank you for your detailed explanation.
We are already investigating it internally, so I'll get back to you in this post when there's news.
nikolaj
Expert
Posts: 164
Liked: 57 times
Joined: Mar 22, 2021 11:19 am
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by nikolaj »

Hi _james,

After some analysis we've come to the conclusion that the issue is actually two-fold.

First of all, the 'Latency' columns in 'Datastore IOPs' and 'IOPs by host' tables of Datastore Performance Assessment report should not be there as they don't actually represent latency metric. That's a minor visual artefact, though.
Another problem is the way those values are being calculated when one chooses 'Current week' as the timeframe in the report parameters.
With that option they are calculated differently (and diverge significantly from the ones in 'Datastore Performance' report) than when one picks 'Past week', for instance.

Please open a support case referencing this thread (and bug ID 321622) in order to receive a patch when it becomes available.

Thanks for bringing this to our attention!
nikolaj
Expert
Posts: 164
Liked: 57 times
Joined: Mar 22, 2021 11:19 am
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by nikolaj » 1 person likes this post

Hi _james,

There's been a new discovery in this case. Turns out, the values between two reports differ because of the specific logic in the Datastore Performance Assessment report. We reflected those details in the user guide but here's the explanation of how it works:

When you set the Interval parameter that is less than or equal to 7 days, the IOPS counters of datastores in this report will be generated based on the more accurate (more frequently gathered) values. The ones that are collected every 5 min as opposed to the ones that are obtained every 2 hours.
As a consequence, the absolute values for the IOPS metric in this report will differ from the values found in the Datastore Performance report for the same datastore objects, since the latter lacks this logic. It will also take more time to generate the report because there would be much more points to process.

I hope this helps.
_james
Enthusiast
Posts: 30
Liked: 6 times
Joined: Nov 15, 2018 3:51 pm
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by _james »

Hello @nikolaj, thank you for the update and explanation.

On the another note I have noticed another bug - vSphere Custom Performance is not working with Virtual Machimes - CPU and Memory usage (Percent) atleast for me, let me know if I should make another thread about this, as this is somewhat off topic.
_james
Enthusiast
Posts: 30
Liked: 6 times
Joined: Nov 15, 2018 3:51 pm
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by _james »

Not sure if it was intended but charts are working, data can be seen if only one type of resource is selected (CPU or RAM). (data is only for last 3-4 days, this is a known bug in v11.)
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Maximum Datastore IOPS is lower from VM

Post by Vitaliy S. »

_james wrote:On the another note I have noticed another bug - vSphere Custom Performance is not working with Virtual Machimes - CPU and Memory usage (Percent) atleast for me, let me know if I should make another thread about this, as this is somewhat off topic.
No need to open another thread. To get it fixed, I would recommend contacting our technical support team.
Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests