Real-time performance monitoring and troubleshooting
Post Reply
max0664
Influencer
Posts: 17
Liked: never
Joined: Sep 04, 2012 4:30 pm
Full Name: Max
Contact:

Host disk write latency

Post by max0664 »

Hi,
i am using Veeam one v9 and have 4x UCS B200 M4 Server with 2x 300 GB 10K 64MB Cache (Raid 1) for a Hyper-V Cluster.
I have also 6x UCS B200 M3 Server with 2x 300 GB 10k 32MB Cache (Raid 1) for a other Hyper-V Cluster.
But i get often this error on all UCS B200 M4 Server. Sometimes 16 Milliseconds sometimes 17 Milliseconds.

Code: Select all

Host disk write latency
Alarm details
Description
"Disk/Host: Avg Disk sec/Write" (16.0 Milliseconds) is above a defined threshold (15.0 Milliseconds) for 'C:';
Knowledge
This host disk (logical disk) has exceeded the threshold for total write latency. This performance monitor counter measures the amount of time that write operations take to respond to the operating system
Cause
As a general rule of thumb, average response times greater than 15ms are considered sub-optimal. This is based on the typical seek time of a single 7200 RPM disk drive without cache. Physical disks installed on a non-virtualized environment offer better performance than disks accessed through a Hyper-V host operating system
Resolution
View the host disk latency for historical information. If your workload is disk-intensive, consider using SCSI controllers for the VMs running on this host as they can reduce CPU overhead and enable a virtual SCSI bus to provide multiple transactions simultaneously. Be aware that not all guest OS supports that configuration. If that's not possible, add additional SCSI-connected VHDs (preferably ones that are stored on separate physical spindles or arrays on the host server). Also if you have any concurrently running resource intensive tasks (backup jobs, antivirus checks etc.), consider using different time schedule in order to offload you host disks
I update the LSI Raid controller Driver. I use the same OS 2012 R2 and Virusscanner on both Type of Server.
Someone have some idee to fix it?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Host disk write latency

Post by Shestakov »

Hi Max,
Is the alarm usually being triggered during some backup or any other periodic activity?
You may want to leverage IO Control options.
max0664
Influencer
Posts: 17
Liked: never
Joined: Sep 04, 2012 4:30 pm
Full Name: Max
Contact:

Re: Host disk write latency

Post by max0664 »

Hey Shestakov,
my backup start at 22:00 and need around 30 min. Active full need longer. I dont have some periodic activity that i know. Only the virus scanner start every Wednesday at 12:00 a quick scan and a MS sccm client installed on alle Server.

Here the time frequenze of the error
Image

The problem is on the C: drive. I use for the VMs a CSV with fiber channel.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Host disk write latency

Post by Shestakov »

Looks like backup activity is not the main reason of the alarm.
Have you considered using SCSI controllers for the VMs running on the host as stated in the alarm description?
Do you notice low performance when the alarms are triggered?
max0664
Influencer
Posts: 17
Liked: never
Joined: Sep 04, 2012 4:30 pm
Full Name: Max
Contact:

Re: Host disk write latency

Post by max0664 »

I have 12 VMs in the Cluster, 2 Windows 7 and 10 Windows Server 2012 R2. They all have one SCSI Controller. Only the Win 7 have one IDE and one SCSI Controller.
I dont work with the VMs. But i dont get any feedback that the VMs are low on performance or something.
The Host work fine. Dont see any performance impact.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Host disk write latency

Post by Shestakov »

I would suggest to run Hyper-V Configuration and Performance Assessment reports to find out potential flaws.

By the way, since your warnings are just a bit above the threshold and looks like the issue is hardware you may just shift the threshold a bit not to be bothered constantly.
CezarL
Lurker
Posts: 2
Liked: never
Joined: Oct 26, 2015 12:11 am
Full Name: Cezar L D Santiago
Contact:

Re: Host disk write latency

Post by CezarL »

hi guys!
I have exactly the same problem. Someone managed to get around? Data Stores CSV not accuse anything .. the error points to C: \ only
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Host disk write latency

Post by Vitaliy S. »

Hi Cezar,

When you see these alarms triggered? During some special activity on the server?

Thanks!
CezarL
Lurker
Posts: 2
Liked: never
Joined: Oct 26, 2015 12:11 am
Full Name: Cezar L D Santiago
Contact:

Re: Host disk write latency

Post by CezarL »

Not a specific time, all time. randomly in all 3 hosts.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Host disk write latency

Post by Vitaliy S. »

Did you check latency performance graphs for this disk? That would be helpful in correlating what activity might be causing this.
Post Reply

Who is online

Users browsing this forum: No registered users and 15 guests