VM Disk Errors on 2012

[JimTech18]

I get several VM Disk Errors reported in monitoring every time I restart my VM but can't find corresponding errors in the 2012 logs. Can someone help me locate the correct location to find the log errors so I can troubleshoot and resolve this issue.

Thanks

[Vitaliy S.]

Hello Jim,

Do you see these VM disk errors for every VM you restart?

Thank you!

[KaterKarlo99]

Hi,

i got the same issue un a W2K12 hyper-v cluster. Some of my VM's have this "VM Disk Errors" Error.
I'm interested too, to localized and fix this.

Anyone an idea?

[JimTech18]

I have just the one VM at this point. I plan to move several others from an old server but hesitate to do so until I have diagnosed these errors.

One item of note is that the errors only occur on the 3 mounted vhdx drives via SCSI controller.

Thanks

[veremin]

Some of my VM's have this "VM Disk Errors" Error.

I’m wondering whether these disks have something in common. Or it’s just random disks that are affected by this error? Thanks.

[Vitaliy S.]

One item of note is that the errors only occur on the 3 mounted vhdx drives via SCSI controller.

Can you please take a look at other VM events happened at that time?

[JimTech18]

Can you please take a look at other VM events happened at that time?

Where would you like me to look? I have reviewed the event logs on both the Host and Guest and see no errors or warnings before or after the restart. But then the Veeam One monitor shows another alert. I always get a status of resolved at the time that the restart begins with "Virtual Machine was powered off" then a while after it is back up I get an alert for each .vhdx. Restarting the Host does not seem to generate the errors. I am planning on doing some additional testing, changing to IDE controller; mounting a new .vhd file both as IDE and as SCSI to see what happens. I will report back.

[JimTech18]

I changed one of the 3 SCSI mounted .vhdx to IDE and restarted the server. Now I only get errors on the SCSI drives, not the IDE drive. Also errors reported for each drive dropped from (9) to (7).

[Vitaliy S.]

I was referring to events tab on the VM. Can you please tell me wha values do you have for these disks if you take a look here: PerfMon -> Hyper-V Virtual Storage Device\Error Count?

[JimTech18]

I was referring to events tab on the VM. Can you please tell me wha values do you have for these disks if you take a look here: PerfMon -> Hyper-V Virtual Storage Device\Error Count?

On the host I found that monitor and the error count on both SCSI drives is 9 and 0 on the IDE drives.

Now I just have to figure where they are coming from, any ideas?

[Vitaliy S.]

Are these pass-through disks by any chance? I have found this KB article from Microsoft: http://support.microsoft.com/kb/2624495

Even though this article refers to Windows Server 2008R2, it still does look that it can be applied to Windows Server 2012.

[JimTech18]

Are these pass-through disks by any chance? I have found this KB article from Microsoft: http://support.microsoft.com/kb/2624495

Even though this article refers to Windows Server 2008R2, it still does look that it can be applied to Windows Server 2012.

No they are VHDX disks not pass through disks. Though the end result seems to be exactly what is being described here. Since VHDX disks did not exist for 2008 HyperV I wonder if it is a similar issue with the new disk type.

Looks like I will have to raise the threshold to (10) to eliminate the false indication while still allowing indication of more serious problems. Does that sound like the best solution to this issue?

[Vitaliy S.]

Currently it seems like an acceptable workaround, however the number of errors will keep rising every time, as this counter only resets itself after you reboot the VM. To address this issue, I have asked our dev team to measure the number of errors per collection interval, so in v7 we will report on errors rate (number of errors per time period) and not only on the actual number of errors occurred.

[JimTech18]

Currently it seems like an acceptable workaround, however the number of errors will keep rising every time, as this counter only resets itself after you reboot the VM. To address this issue, I have asked our dev team to measure the number of errors per collection interval, so in v7 we will report on errors rate (number of errors per time period) and not only on the actual number of errors occurred.

This should not be a problem if the errors only occur on restart of the VM also, since the counters clear each time I am only seeing the same number of errors after each restart. It would be great if at least it was reported to Microsoft as a problem, doesn't mean they will ever fix it but still worth reporting. It would be great if your dev team could confirm they get the same results when mounting a VHDX as SCSI. Then we could at least know it is not just my configuration.

[Vitaliy S.]

Yes, this does make sense.

[davida]

I am getting the same error with a Windows 2012 R2 Cluster with 2012 VM reporting VM disk errors. On my end it happens with both IDE and SCSI drives.

[Vitaliy S.]

Hi David,

Do you see it for all VMs? Does it happen on a particular time or on the action you perform?

Thanks!

[davida]

Hi Vitaly,

Not on all VM, for now I saw the error on 1 Windows 2012 with SCSI controller, 1 Windows 2012 with IDE controller and 1 2008R2 with IDE controller.

No particular time, this is the last one from the 2008R2

Object: RDS
Object Type: Hyper-V VM
Parent Object: Resources
Status: Warning
Alarm: VM disk errors
Time: 12/03/2014 7:53:11 PM
Details: "Error Count" (2) is above a defined threshold (1)



Object: RDS
Object Type: Hyper-V VM
Parent Object: Resources
Status: Reset/resolved
Alarm: VM disk errors
Time: 12/03/2014 7:54:11 PM
Details: All alarm metrics are back to normal

[Vitaliy S.]

David,

We are trying to verify all situations when these errors can occur, for us even a reboot of the VM generates these errors in perf monitor. Can you please adjust the threshold and monitor the behavior of this counter?

Thanks!

[davida]

Hi Vitaly,

I will increase the counter and monitor. Does the above reply means that you are also experiencing this issue on your end? If yes does it appear on both idr and scsi disks? We would like to exclude the possibility of disk errors and this particular alert is the worse one to get

[Vitaliy S.]

In our QC environment we see these errors for some VMs, but the number doesn't go higher than 2-4. I believe this is still something to do with the fact the some SCSI commands are either not recognized by the host or something else. Not sure why you see it with IDE controllers...

[Paul W.]

Hello everyone,

As I've seen this particular issue manifest itself several times, I just wanted to offer any information I have on this:

This particular alarm is monitoring the perfmon metric Hyper-V Virtual Storage Device > Error Count on the host server housing the VM (with each instance of this counter relating to a VM running on the host). From what I've seen myself, powering off or migrating the VMs and then rebooting the host server makes the error count reset, however, you can usually observe the error count in perfmon climb right back to where it was previously as soon as the host and VMs are powered back on so it seems that, in at least most cases, the commands causing the errors are happening during the boot sequence but I have no way of confirming this.

Unfortunately, as you've all noted in this thread, the environmental details surrounding this are often quite random. I've seen it happen with both SCSI and IDE disk controllers. The only strict correlation I've noticed is that this always seems to happen with Windows Server 2012+ or Windows 8+ VM's that are running on Hyper-V 2012+ hosts. I hope this helps in some way but I've not found any information out there yet documenting a cause or a resolution.

[avrham_g]

any fix for this issue?
i get this error on windows 2012 r2 cluster no pass-through disk,disks are vhd and vhdx...

THX

[Vitaliy S.]

Hi Avi,

These counters are reported by Hyper-V, so I do not see how we can fix that on our side. Do you see this on random VMs or on particular ones?

Thanks!

[avrham_g]

i see this error on random vm's.

[Vitaliy S.]

Usually these errors happen on VM reboot and SCSI controllers. Let me check if I can ask Microsoft guys on why perfmon is reporting these errors. BTW, is there any correlation in the Guest OS installed in the VMs?

[teachdire]

Sorry for posting in this older Thread , but i have a similar problem , i get the same error counts as mentioned above.

Server 2012 R2 fully patched as of today.

What i found is :

When i get the message from veeam one :

Error
VM disk errors
21.05.2015 15:31:29
"Errors/min" (50) is above a defined threshold (2)

I find the following entry in my system logfile on the guest machine:

21.05.2015 15:30:51
Im System wurde ein Dienst installiert.

Dienstname: gfiark
Dienstdateiname: system32\drivers\gfiark.sys
Diensttyp: Kernelmodustreiber
Dienststarttyp: Manuell starten
Dienstkonto:

happen after the definiton update of the AV scanner:
21.05.2015 15:30:52
Definitions update downloaded.

Definitions version: 40430

The error occurs only then and sometimes when the veeam backups starts.

Maybe it helps to find out where these errors come from exactly.

[Vitaliy S.]

Hello,

Don't be sorry for posting into existing topic, actually this is the way all users should do

In order to help us with troubleshooting, can you please say what Veeam ONE version you're currently at?

Thank you!

[teachdire]

Veeam One 8.0.0.1669

Just checked over night VM Disk errors occur everytime the AV Scanner gets an update on the guest machines, but only 2 of 3 machines on the same cluster volume.

Just uninstalled the AV Scanner for testing.

2 Questions:

Just trying to understand what exactly this errors means i cant find anything in my logfiles execpt i trace it in perfmon.

My servers running fine and smooth, backups are sucessfull inkl Recovery tests.

Any ideas in the meantime where this error comes from ?

[Vitaliy S.]

What type of virtual disk is this? The reason for my question is that for Pass-through Disk this is expected to have these kind of errors, which can be disregarded according to Microsoft > http://support.microsoft.com/en-us/kb/2624495