Maximum Datastore IOPS is lower from VM

[_james]

Hello everyone,

I have a confusing issue with Veeam One reports.
Every month I create custom report where I check TOP 10 VM IOPS and latency.
What confuses me if I compare TOP 10 VM numbers to Datastore highest latency and IOPS ones the datastore numbers are lower. Why is that? Shouldn't the Datastore maximum IOPS and Latency be atleast the same as the peak TOP-1 VM one (atleast because there will be other VMs that will generate IO).

Can someone advise why is that?

[wishr]

Hi _james,

Could you please provide at least a few screenshots from both reports for our better understanding?

Thanks in advance.

[_james]

Hello,

Yes, no problem.
When comparing vSphere Monitoring - Datastore Performance with vSphere Infrastructure Assessment - Datastore Performance assessment information about peak datastore IOPS the peak numbers are the same, no problem there:



However, as mentioned, when checking out vSphere Custom performance - Object type: Virtual Machines, Measured entities: Datastore and Read/Write IO and Latency we can see that some VMs produce higher IOs and latencies than Datastore performance and Performance assessment reported ones. Here are the write and read IOPS:



Timeframe - last week for top VMs (4.06 - 10.06), this month for Datastore performance and assessment.

[_james]

The only thing which is different is the timeframe. As you cannot set the same timeframes for DS performance and DS performance assessment as Custom VM IO metrics because of the the bug in V11 that shows only data for the last week (you can but only on sunday for the timeframe to be Last week.)
However this shouldn't be an issue because of the timeframes overlapping (This month for datastores / last week for the VMs)

[nikolaj]

Hi _james,

Would you be able to provide a full excerpt of that report page while I investigate this on my lab environment?
In general, all performance data come directly from vSphere counters, so I don't expect there to be any surprises in terms of interpretation.

[_james]

Hello Nikolaj, sorry about previous screenshots, I created new ones.

This time - all timestamps are be equal. Range of 07.06.-13.06. for Custom VM metrics and DS performance. Interval - Current Week for DS Performance assessment (end time is in the "future", but that is the only way how to rule out timestamp inconsistencies, also it is not possible to set a specific range for DS Perf. assessment).

Heres what I found:
Lets take vSphere Datastore performance as a reference. As we can see for specified reporting period it reports max. IOPS of 6328

Then we compare this result to Datastore performance assessment, timeframe is current week, but it has the same dates as the Range timeframe.

However it shows much higher result:


Heres what confuses me the most - VMs show higher IOPS than the DS Performance report, however this time the numbers do fall under the DS performance assessment report (DS max iops are higher than TOP1 VM IOPS which makes sense)


Same applies to basically all storage metrics - rw iops, latencies etc.
Apologies for the lenghty posts, also if something is not understandable here, let me know i'll try to rephrase, provide additional data. Host, Datastore and VM names are cleared for obvious reasons.

[nikolaj]

Hi _james,

Thank you for your detailed explanation.
We are already investigating it internally, so I'll get back to you in this post when there's news.

[nikolaj]

Hi _james,

After some analysis we've come to the conclusion that the issue is actually two-fold.

First of all, the 'Latency' columns in 'Datastore IOPs' and 'IOPs by host' tables of Datastore Performance Assessment report should not be there as they don't actually represent latency metric. That's a minor visual artefact, though.
Another problem is the way those values are being calculated when one chooses 'Current week' as the timeframe in the report parameters.
With that option they are calculated differently (and diverge significantly from the ones in 'Datastore Performance' report) than when one picks 'Past week', for instance.

Please open a support case referencing this thread (and bug ID 321622) in order to receive a patch when it becomes available.

Thanks for bringing this to our attention!

[nikolaj]

Hi _james,

There's been a new discovery in this case. Turns out, the values between two reports differ because of the specific logic in the Datastore Performance Assessment report. We reflected those details in the user guide but here's the explanation of how it works:

When you set the Interval parameter that is less than or equal to 7 days, the IOPS counters of datastores in this report will be generated based on the more accurate (more frequently gathered) values. The ones that are collected every 5 min as opposed to the ones that are obtained every 2 hours.
As a consequence, the absolute values for the IOPS metric in this report will differ from the values found in the Datastore Performance report for the same datastore objects, since the latter lacks this logic. It will also take more time to generate the report because there would be much more points to process.

I hope this helps.

[_james]

Hello @nikolaj, thank you for the update and explanation.

On the another note I have noticed another bug - vSphere Custom Performance is not working with Virtual Machimes - CPU and Memory usage (Percent) atleast for me, let me know if I should make another thread about this, as this is somewhat off topic.

[_james]

Not sure if it was intended but charts are working, data can be seen if only one type of resource is selected (CPU or RAM). (data is only for last 3-4 days, this is a known bug in v11.)

[Vitaliy S.]

On the another note I have noticed another bug - vSphere Custom Performance is not working with Virtual Machimes - CPU and Memory usage (Percent) atleast for me, let me know if I should make another thread about this, as this is somewhat off topic.

No need to open another thread. To get it fixed, I would recommend contacting our technical support team.