VM Disk metrics for Hyper-V

[coolf124]

Hi!

I am testing Veeam ONE 11 with Hyper-V enviroment. My regular job task is to find the VMs that create most IOPS on HDD datastores to reallocate them to SSD tier.
However I can't find enough metrics at VM level in Customs Reports - Hyper-V Custom Perfomance or Raw Perfomance
It seems that there are more metrics in same report for VMware then there are for Hyper-V. We used Veeam ONE with vSphere at my old job.

I see Disk IO, Writes/s, Reads/s, Virtual Storage Write/Read bytes/c. I see no latency metrics, no throughput.

This raises 3 questions:
1) Are theses the only metric for VM Disk in Hyper-V or am I missing something? Maybe some data is not gathered or maybe not processed
2) If there are additional metrics like Latency, Disk Que lengh how can I see them and add to report
3) If there aren't additional metrics, then how should I analyze the ones that exists to find VM that overload the datastores

[wishr]

Hi Artem,

We collect and process most of the useful metrics available in Hyper-V. For example, in Hyper-V disk latency is present via "Avg. Disk sec/Read" and "Avg. Disk sec/Write" performance monitor counters which are collected by Veeam ONE and can be populated in the reports you mentioned. At the same time, you can find these metrics in the Client UI on the "Disk" tab. The same about Avg. Disk Queue Length. These metrics are attributes of the host system and are not related directly to the VMs. Please, keep in mind that Hyper-V is not the same as vSphere, and differences in the list and the size of the list of available metrics are expected.

You can see the full list of metrics we collect via the Veeam ONE setting utility, scalability section, performance metrics tab. Also, please make sure the required metrics are ticked and thus collected.

If the end goal is to determine what VMs are causing load on a datastore you may even use predefined reports, such as Local Datastore Performance, for example.

Thanks

[coolf124]

I dont' see that the suggested metrics, such as "Avg. Disk sec/Read" and "Avg. Disk sec/Write" are available at VM level
If there are a lot of VMs on the datastore, how can I find out which VMs are experiencing problems and which are not, and which are causing the problem?

[wishr]

Hi Artem,

As mentioned, these metrics are available only at the host level in Hyper-V and are there to help you find out if the storage is experiencing a significant overhead.

What particular VM issues you are trying to discover? We have lots of reports and I'm sure there are several that cover your use case.

Thanks

[coolf124]

Veeam reports alarms on the datastore such as Cluster Shared Volume write latency threshold exceeded.
The datastore has many VMs on it. Also there are many datastores on the same disk group. So at the moment of trouble there are many datastores, hosts and VM's impacted.

I want to find which VMs are doing excess IO at that moment and that are causing latency issues.

[wishr]

In this case, you may run the Windows Cluster Shared Volume Performance report which is similar to the report I linked above. It helps to identify CSV trends and highlights VMs that causing the most overhead on the storage. Depending on the outcome, you may decide to relocate these VMs or investigate the excessive resource consumption on the VM side with the help of other reports like VM Performance or custom reports.

[coolf124]

What is the difference betwen Disk/CSV: Latency (Millisecond) and Disk/CSV: Direct Latency (Millisecond)
Direct Latency seems to be much less then just Latency. Why are they different, while there are no reditected IOPS on the volume

[wishr]

Please, take a look at this article. You can also view these metrics natively in Hyper-V if concerned about the correctness of the values.

[coolf124]

Hi!

I am comparing veeam Disk/CSV: Latency (Millisecond) and Disk/CSV: Direct Latency (Millisecond) for Hyper-V CSV datastores.
The datastores are from FC SAN, formatted in NTFS and is direct mode. The redirected IO is 0.

From Veeam Hyper-V Custom Performance I notice very different values for the same datastores. Disk/CSV: Latency (Millisecond) is 2-3 higher then Direct Latency for the same datastores.
Why and what is this difference between the 2 metrics for the same datastore in my case? Which metric is the real latency for VMs on this datastore ?

[wishr]

Hi Artem,

These are two different metrics based on two separate Hyper-V counters.

The first metric (Direct Latency) represents average latency between the time a request is sent to the disk using Direct IO and when its response is received. The second metric (Latency) represents average latency between the time a request arrived to the file system and when it was completed. Check that article by Microsoft for more info.

P.S. I'm merging this topic with our previous discussion.

Thanks

[coolf124]

@wishr Thank you, for the reply.

Studied hard article provided , but still don't get why they differ so much in my case. So the Direct IO from disk is 8 ms on average and IO spends an additional 20 ms on average somewhere in CSVFS ?
Any clues on where to look to find the cause or what else to look for?

[wishr]

Unfortunately, it's hard to say what is the root cause for that in your particular case without comprehensively analyzing all the details of the environment. I would suggest checking Perfmon for the corresponding counters and then, based on the results, look for an answer on Microsoft resources or even open a case directly with their technical support.

Thanks