Host disk write latency

Real-time performance monitoring and troubleshooting

Host disk write latency

Veeam Logoby max0664 » Sat Mar 12, 2016 3:34 pm

Hi,
i am using Veeam one v9 and have 4x UCS B200 M4 Server with 2x 300 GB 10K 64MB Cache (Raid 1) for a Hyper-V Cluster.
I have also 6x UCS B200 M3 Server with 2x 300 GB 10k 32MB Cache (Raid 1) for a other Hyper-V Cluster.
But i get often this error on all UCS B200 M4 Server. Sometimes 16 Milliseconds sometimes 17 Milliseconds.

Code: Select all
Host disk write latency
Alarm details
Description
"Disk/Host: Avg Disk sec/Write" (16.0 Milliseconds) is above a defined threshold (15.0 Milliseconds) for 'C:';
Knowledge
This host disk (logical disk) has exceeded the threshold for total write latency. This performance monitor counter measures the amount of time that write operations take to respond to the operating system
Cause
As a general rule of thumb, average response times greater than 15ms are considered sub-optimal. This is based on the typical seek time of a single 7200 RPM disk drive without cache. Physical disks installed on a non-virtualized environment offer better performance than disks accessed through a Hyper-V host operating system
Resolution
View the host disk latency for historical information. If your workload is disk-intensive, consider using SCSI controllers for the VMs running on this host as they can reduce CPU overhead and enable a virtual SCSI bus to provide multiple transactions simultaneously. Be aware that not all guest OS supports that configuration. If that's not possible, add additional SCSI-connected VHDs (preferably ones that are stored on separate physical spindles or arrays on the host server). Also if you have any concurrently running resource intensive tasks (backup jobs, antivirus checks etc.), consider using different time schedule in order to offload you host disks


I update the LSI Raid controller Driver. I use the same OS 2012 R2 and Virusscanner on both Type of Server.
Someone have some idee to fix it?
max0664
Influencer
 
Posts: 17
Liked: never
Joined: Tue Sep 04, 2012 4:30 pm
Full Name: Max

Re: Host disk write latency

Veeam Logoby Shestakov » Sat Mar 12, 2016 10:34 pm

Hi Max,
Is the alarm usually being triggered during some backup or any other periodic activity?
You may want to leverage IO Control options.
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Host disk write latency

Veeam Logoby max0664 » Sun Mar 13, 2016 1:42 am

Hey Shestakov,
my backup start at 22:00 and need around 30 min. Active full need longer. I dont have some periodic activity that i know. Only the virus scanner start every Wednesday at 12:00 a quick scan and a MS sccm client installed on alle Server.

Here the time frequenze of the error
Image

The problem is on the C: drive. I use for the VMs a CSV with fiber channel.
max0664
Influencer
 
Posts: 17
Liked: never
Joined: Tue Sep 04, 2012 4:30 pm
Full Name: Max

Re: Host disk write latency

Veeam Logoby Shestakov » Sun Mar 13, 2016 7:35 am

Looks like backup activity is not the main reason of the alarm.
Have you considered using SCSI controllers for the VMs running on the host as stated in the alarm description?
Do you notice low performance when the alarms are triggered?
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Host disk write latency

Veeam Logoby max0664 » Sun Mar 13, 2016 3:53 pm

I have 12 VMs in the Cluster, 2 Windows 7 and 10 Windows Server 2012 R2. They all have one SCSI Controller. Only the Win 7 have one IDE and one SCSI Controller.
I dont work with the VMs. But i dont get any feedback that the VMs are low on performance or something.
The Host work fine. Dont see any performance impact.
max0664
Influencer
 
Posts: 17
Liked: never
Joined: Tue Sep 04, 2012 4:30 pm
Full Name: Max

Re: Host disk write latency

Veeam Logoby Shestakov » Sun Mar 13, 2016 7:39 pm

I would suggest to run Hyper-V Configuration and Performance Assessment reports to find out potential flaws.

By the way, since your warnings are just a bit above the threshold and looks like the issue is hardware you may just shift the threshold a bit not to be bothered constantly.
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Host disk write latency

Veeam Logoby CezarL » Tue Mar 29, 2016 11:12 am

hi guys!
I have exactly the same problem. Someone managed to get around? Data Stores CSV not accuse anything .. the error points to C: \ only
CezarL
Lurker
 
Posts: 2
Liked: never
Joined: Mon Oct 26, 2015 12:11 am
Full Name: Cezar L D Santiago

Re: Host disk write latency

Veeam Logoby Vitaliy S. » Tue Mar 29, 2016 4:31 pm

Hi Cezar,

When you see these alarms triggered? During some special activity on the server?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Host disk write latency

Veeam Logoby CezarL » Tue Mar 29, 2016 7:07 pm

Not a specific time, all time. randomly in all 3 hosts.
CezarL
Lurker
 
Posts: 2
Liked: never
Joined: Mon Oct 26, 2015 12:11 am
Full Name: Cezar L D Santiago

Re: Host disk write latency

Veeam Logoby Vitaliy S. » Tue Mar 29, 2016 8:44 pm

Did you check latency performance graphs for this disk? That would be helpful in correlating what activity might be causing this.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov


Return to Monitoring



Who is online

Users browsing this forum: No registered users and 2 guests