VM Disk Errors on 2012

Real-time performance monitoring and troubleshooting

Re: VM Disk Errors on 2012

Veeam Logoby davida » Wed Mar 12, 2014 12:47 pm

I am getting the same error with a Windows 2012 R2 Cluster with 2012 VM reporting VM disk errors. On my end it happens with both IDE and SCSI drives.
davida
Influencer
 
Posts: 20
Liked: 6 times
Joined: Fri May 04, 2012 5:48 am
Full Name: David Avice

Re: VM Disk Errors on 2012

Veeam Logoby Vitaliy S. » Wed Mar 12, 2014 3:08 pm

Hi David,

Do you see it for all VMs? Does it happen on a particular time or on the action you perform?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VM Disk Errors on 2012

Veeam Logoby davida » Wed Mar 12, 2014 3:44 pm

Hi Vitaly,

Not on all VM, for now I saw the error on 1 Windows 2012 with SCSI controller, 1 Windows 2012 with IDE controller and 1 2008R2 with IDE controller.

No particular time, this is the last one from the 2008R2

Object:RDS
Object Type:Hyper-V VM
Parent Object:Resources
Status:Warning
Alarm:VM disk errors
Time:12/03/2014 7:53:11 PM
Details:"Error Count" (2) is above a defined threshold (1)



Object:RDS
Object Type:Hyper-V VM
Parent Object:Resources
Status:Reset/resolved
Alarm:VM disk errors
Time:12/03/2014 7:54:11 PM
Details:All alarm metrics are back to normal
davida
Influencer
 
Posts: 20
Liked: 6 times
Joined: Fri May 04, 2012 5:48 am
Full Name: David Avice

Re: VM Disk Errors on 2012

Veeam Logoby Vitaliy S. » Wed Mar 12, 2014 3:59 pm

David,

We are trying to verify all situations when these errors can occur, for us even a reboot of the VM generates these errors in perf monitor. Can you please adjust the threshold and monitor the behavior of this counter?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VM Disk Errors on 2012

Veeam Logoby davida » Thu Mar 13, 2014 4:00 pm

Hi Vitaly,

I will increase the counter and monitor. Does the above reply means that you are also experiencing this issue on your end? If yes does it appear on both idr and scsi disks? We would like to exclude the possibility of disk errors and this particular alert is the worse one to get :-)
davida
Influencer
 
Posts: 20
Liked: 6 times
Joined: Fri May 04, 2012 5:48 am
Full Name: David Avice

Re: VM Disk Errors on 2012

Veeam Logoby Vitaliy S. » Thu Mar 13, 2014 4:06 pm

In our QC environment we see these errors for some VMs, but the number doesn't go higher than 2-4. I believe this is still something to do with the fact the some SCSI commands are either not recognized by the host or something else. Not sure why you see it with IDE controllers...
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VM Disk Errors on 2012

Veeam Logoby Paul W. » Tue Mar 25, 2014 10:56 pm

Hello everyone,

As I've seen this particular issue manifest itself several times, I just wanted to offer any information I have on this:

This particular alarm is monitoring the perfmon metric Hyper-V Virtual Storage Device > Error Count on the host server housing the VM (with each instance of this counter relating to a VM running on the host). From what I've seen myself, powering off or migrating the VMs and then rebooting the host server makes the error count reset, however, you can usually observe the error count in perfmon climb right back to where it was previously as soon as the host and VMs are powered back on so it seems that, in at least most cases, the commands causing the errors are happening during the boot sequence but I have no way of confirming this.

Unfortunately, as you've all noted in this thread, the environmental details surrounding this are often quite random. I've seen it happen with both SCSI and IDE disk controllers. The only strict correlation I've noticed is that this always seems to happen with Windows Server 2012+ or Windows 8+ VM's that are running on Hyper-V 2012+ hosts. I hope this helps in some way but I've not found any information out there yet documenting a cause or a resolution.
Paul W.
Enthusiast
 
Posts: 87
Liked: 6 times
Joined: Tue May 29, 2012 7:33 pm
Full Name: Paul Wallace

Re: VM Disk Errors on 2012

Veeam Logoby avrham_g » Tue May 20, 2014 9:36 am

any fix for this issue?
i get this error on windows 2012 r2 cluster no pass-through disk,disks are vhd and vhdx...

THX
avrham_g
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Thu Dec 29, 2011 9:41 pm
Full Name: Avi

Re: VM Disk Errors on 2012

Veeam Logoby Vitaliy S. » Tue May 20, 2014 9:58 am

Hi Avi,

These counters are reported by Hyper-V, so I do not see how we can fix that on our side. Do you see this on random VMs or on particular ones?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VM Disk Errors on 2012

Veeam Logoby avrham_g » Tue May 20, 2014 10:47 am

i see this error on random vm's.
avrham_g
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Thu Dec 29, 2011 9:41 pm
Full Name: Avi

Re: VM Disk Errors on 2012

Veeam Logoby Vitaliy S. » Tue May 20, 2014 4:11 pm

Usually these errors happen on VM reboot and SCSI controllers. Let me check if I can ask Microsoft guys on why perfmon is reporting these errors. BTW, is there any correlation in the Guest OS installed in the VMs?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VM Disk Errors on 2012

Veeam Logoby teachdire » Thu May 21, 2015 2:06 pm

Sorry for posting in this older Thread , but i have a similar problem , i get the same error counts as mentioned above.

Server 2012 R2 fully patched as of today.

What i found is :

When i get the message from veeam one :

Error
VM disk errors
21.05.2015 15:31:29
"Errors/min" (50) is above a defined threshold (2)

I find the following entry in my system logfile on the guest machine:

21.05.2015 15:30:51
Im System wurde ein Dienst installiert.

Dienstname: gfiark
Dienstdateiname: system32\drivers\gfiark.sys
Diensttyp: Kernelmodustreiber
Dienststarttyp: Manuell starten
Dienstkonto:

happen after the definiton update of the AV scanner:
21.05.2015 15:30:52
Definitions update downloaded.

Definitions version: 40430

The error occurs only then and sometimes when the veeam backups starts.

Maybe it helps to find out where these errors come from exactly.
teachdire
Lurker
 
Posts: 2
Liked: never
Joined: Thu May 21, 2015 1:57 pm

Re: VM Disk Errors on 2012

Veeam Logoby Vitaliy S. » Thu May 21, 2015 4:58 pm

Hello,

Don't be sorry for posting into existing topic, actually this is the way all users should do :)

In order to help us with troubleshooting, can you please say what Veeam ONE version you're currently at?

Thank you!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VM Disk Errors on 2012

Veeam Logoby teachdire » Fri May 22, 2015 5:51 am

Veeam One 8.0.0.1669

Just checked over night VM Disk errors occur everytime the AV Scanner gets an update on the guest machines, but only 2 of 3 machines on the same cluster volume.

Just uninstalled the AV Scanner for testing.

2 Questions:

Just trying to understand what exactly this errors means i cant find anything in my logfiles execpt i trace it in perfmon.

My servers running fine and smooth, backups are sucessfull inkl Recovery tests.

Any ideas in the meantime where this error comes from ?
teachdire
Lurker
 
Posts: 2
Liked: never
Joined: Thu May 21, 2015 1:57 pm

Re: VM Disk Errors on 2012

Veeam Logoby Vitaliy S. » Fri May 22, 2015 12:23 pm

What type of virtual disk is this? The reason for my question is that for Pass-through Disk this is expected to have these kind of errors, which can be disregarded according to Microsoft > http://support.microsoft.com/en-us/kb/2624495
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

PreviousNext

Return to Monitoring



Who is online

Users browsing this forum: No registered users and 3 guests