Real-time performance monitoring and troubleshooting
JimTech18
Novice
Posts: 7
Liked: never
Joined: Feb 14, 2013 4:52 pm
Contact:

VM Disk Errors on 2012

Post by JimTech18 » Jun 28, 2013 9:03 pm

I get several VM Disk Errors reported in monitoring every time I restart my VM but can't find corresponding errors in the 2012 logs. Can someone help me locate the correct location to find the log errors so I can troubleshoot and resolve this issue.

Thanks

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » Jun 30, 2013 11:44 am

Hello Jim,

Do you see these VM disk errors for every VM you restart?

Thank you!

KaterKarlo99
Lurker
Posts: 1
Liked: never
Joined: Jul 09, 2013 10:16 am
Full Name: Rainer Kreuz
Contact:

Re: VM Disk Errors on 2012

Post by KaterKarlo99 » Jul 09, 2013 10:19 am

Hi,

i got the same issue un a W2K12 hyper-v cluster. Some of my VM's have this "VM Disk Errors" Error.
I'm interested too, to localized and fix this.

Anyone an idea?

JimTech18
Novice
Posts: 7
Liked: never
Joined: Feb 14, 2013 4:52 pm
Contact:

Re: VM Disk Errors on 2012

Post by JimTech18 » Jul 10, 2013 1:41 am

I have just the one VM at this point. I plan to move several others from an old server but hesitate to do so until I have diagnosed these errors.

One item of note is that the errors only occur on the 3 mounted vhdx drives via SCSI controller.

Thanks

veremin
Product Manager
Posts: 16901
Liked: 1438 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VM Disk Errors on 2012

Post by veremin » Jul 10, 2013 8:10 am

Some of my VM's have this "VM Disk Errors" Error.
I’m wondering whether these disks have something in common. Or it’s just random disks that are affected by this error? Thanks.

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » Jul 10, 2013 9:38 am

JimTech18 wrote:One item of note is that the errors only occur on the 3 mounted vhdx drives via SCSI controller.
Can you please take a look at other VM events happened at that time?

JimTech18
Novice
Posts: 7
Liked: never
Joined: Feb 14, 2013 4:52 pm
Contact:

Re: VM Disk Errors on 2012

Post by JimTech18 » Jul 11, 2013 2:18 pm

Vitaliy S. wrote: Can you please take a look at other VM events happened at that time?
Where would you like me to look? I have reviewed the event logs on both the Host and Guest and see no errors or warnings before or after the restart. But then the Veeam One monitor shows another alert. I always get a status of resolved at the time that the restart begins with "Virtual Machine was powered off" then a while after it is back up I get an alert for each .vhdx. Restarting the Host does not seem to generate the errors. I am planning on doing some additional testing, changing to IDE controller; mounting a new .vhd file both as IDE and as SCSI to see what happens. I will report back.

JimTech18
Novice
Posts: 7
Liked: never
Joined: Feb 14, 2013 4:52 pm
Contact:

Re: VM Disk Errors on 2012

Post by JimTech18 » Jul 11, 2013 2:47 pm

I changed one of the 3 SCSI mounted .vhdx to IDE and restarted the server. Now I only get errors on the SCSI drives, not the IDE drive. Also errors reported for each drive dropped from (9) to (7).

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » Jul 11, 2013 9:27 pm

I was referring to events tab on the VM. Can you please tell me wha values do you have for these disks if you take a look here: PerfMon -> Hyper-V Virtual Storage Device\Error Count?

JimTech18
Novice
Posts: 7
Liked: never
Joined: Feb 14, 2013 4:52 pm
Contact:

Re: VM Disk Errors on 2012

Post by JimTech18 » Jul 11, 2013 9:46 pm

Vitaliy S. wrote:I was referring to events tab on the VM. Can you please tell me wha values do you have for these disks if you take a look here: PerfMon -> Hyper-V Virtual Storage Device\Error Count?
On the host I found that monitor and the error count on both SCSI drives is 9 and 0 on the IDE drives.

Now I just have to figure where they are coming from, any ideas?

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » Jul 11, 2013 10:02 pm

Are these pass-through disks by any chance? I have found this KB article from Microsoft: http://support.microsoft.com/kb/2624495

Even though this article refers to Windows Server 2008R2, it still does look that it can be applied to Windows Server 2012.

JimTech18
Novice
Posts: 7
Liked: never
Joined: Feb 14, 2013 4:52 pm
Contact:

Re: VM Disk Errors on 2012

Post by JimTech18 » Jul 12, 2013 2:10 pm

Vitaliy S. wrote:Are these pass-through disks by any chance? I have found this KB article from Microsoft: http://support.microsoft.com/kb/2624495

Even though this article refers to Windows Server 2008R2, it still does look that it can be applied to Windows Server 2012.
No they are VHDX disks not pass through disks. Though the end result seems to be exactly what is being described here. Since VHDX disks did not exist for 2008 HyperV I wonder if it is a similar issue with the new disk type.

Looks like I will have to raise the threshold to (10) to eliminate the false indication while still allowing indication of more serious problems. Does that sound like the best solution to this issue?

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » Jul 12, 2013 2:41 pm

Currently it seems like an acceptable workaround, however the number of errors will keep rising every time, as this counter only resets itself after you reboot the VM. To address this issue, I have asked our dev team to measure the number of errors per collection interval, so in v7 we will report on errors rate (number of errors per time period) and not only on the actual number of errors occurred.

JimTech18
Novice
Posts: 7
Liked: never
Joined: Feb 14, 2013 4:52 pm
Contact:

Re: VM Disk Errors on 2012

Post by JimTech18 » Jul 12, 2013 2:48 pm

Vitaliy S. wrote:Currently it seems like an acceptable workaround, however the number of errors will keep rising every time, as this counter only resets itself after you reboot the VM. To address this issue, I have asked our dev team to measure the number of errors per collection interval, so in v7 we will report on errors rate (number of errors per time period) and not only on the actual number of errors occurred.
This should not be a problem if the errors only occur on restart of the VM also, since the counters clear each time I am only seeing the same number of errors after each restart. It would be great if at least it was reported to Microsoft as a problem, doesn't mean they will ever fix it but still worth reporting. It would be great if your dev team could confirm they get the same results when mounting a VHDX as SCSI. Then we could at least know it is not just my configuration.

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » Jul 12, 2013 3:10 pm

Yes, this does make sense.

davida
Influencer
Posts: 20
Liked: 6 times
Joined: May 04, 2012 5:48 am
Full Name: David Avice
Contact:

Re: VM Disk Errors on 2012

Post by davida » Mar 12, 2014 12:47 pm

I am getting the same error with a Windows 2012 R2 Cluster with 2012 VM reporting VM disk errors. On my end it happens with both IDE and SCSI drives.

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » Mar 12, 2014 3:08 pm

Hi David,

Do you see it for all VMs? Does it happen on a particular time or on the action you perform?

Thanks!

davida
Influencer
Posts: 20
Liked: 6 times
Joined: May 04, 2012 5:48 am
Full Name: David Avice
Contact:

Re: VM Disk Errors on 2012

Post by davida » Mar 12, 2014 3:44 pm

Hi Vitaly,

Not on all VM, for now I saw the error on 1 Windows 2012 with SCSI controller, 1 Windows 2012 with IDE controller and 1 2008R2 with IDE controller.

No particular time, this is the last one from the 2008R2

Object: RDS
Object Type: Hyper-V VM
Parent Object: Resources
Status: Warning
Alarm: VM disk errors
Time: 12/03/2014 7:53:11 PM
Details: "Error Count" (2) is above a defined threshold (1)



Object: RDS
Object Type: Hyper-V VM
Parent Object: Resources
Status: Reset/resolved
Alarm: VM disk errors
Time: 12/03/2014 7:54:11 PM
Details: All alarm metrics are back to normal

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » Mar 12, 2014 3:59 pm

David,

We are trying to verify all situations when these errors can occur, for us even a reboot of the VM generates these errors in perf monitor. Can you please adjust the threshold and monitor the behavior of this counter?

Thanks!

davida
Influencer
Posts: 20
Liked: 6 times
Joined: May 04, 2012 5:48 am
Full Name: David Avice
Contact:

Re: VM Disk Errors on 2012

Post by davida » Mar 13, 2014 4:00 pm

Hi Vitaly,

I will increase the counter and monitor. Does the above reply means that you are also experiencing this issue on your end? If yes does it appear on both idr and scsi disks? We would like to exclude the possibility of disk errors and this particular alert is the worse one to get :-)

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » Mar 13, 2014 4:06 pm

In our QC environment we see these errors for some VMs, but the number doesn't go higher than 2-4. I believe this is still something to do with the fact the some SCSI commands are either not recognized by the host or something else. Not sure why you see it with IDE controllers...

Paul W.
Veeam Software
Posts: 87
Liked: 6 times
Joined: May 29, 2012 7:33 pm
Full Name: Paul Wallace
Contact:

Re: VM Disk Errors on 2012

Post by Paul W. » Mar 25, 2014 10:56 pm

Hello everyone,

As I've seen this particular issue manifest itself several times, I just wanted to offer any information I have on this:

This particular alarm is monitoring the perfmon metric Hyper-V Virtual Storage Device > Error Count on the host server housing the VM (with each instance of this counter relating to a VM running on the host). From what I've seen myself, powering off or migrating the VMs and then rebooting the host server makes the error count reset, however, you can usually observe the error count in perfmon climb right back to where it was previously as soon as the host and VMs are powered back on so it seems that, in at least most cases, the commands causing the errors are happening during the boot sequence but I have no way of confirming this.

Unfortunately, as you've all noted in this thread, the environmental details surrounding this are often quite random. I've seen it happen with both SCSI and IDE disk controllers. The only strict correlation I've noticed is that this always seems to happen with Windows Server 2012+ or Windows 8+ VM's that are running on Hyper-V 2012+ hosts. I hope this helps in some way but I've not found any information out there yet documenting a cause or a resolution.

avrham_g
Enthusiast
Posts: 34
Liked: 1 time
Joined: Dec 29, 2011 9:41 pm
Full Name: Avi
Contact:

Re: VM Disk Errors on 2012

Post by avrham_g » May 20, 2014 9:36 am

any fix for this issue?
i get this error on windows 2012 r2 cluster no pass-through disk,disks are vhd and vhdx...

THX

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » May 20, 2014 9:58 am

Hi Avi,

These counters are reported by Hyper-V, so I do not see how we can fix that on our side. Do you see this on random VMs or on particular ones?

Thanks!

avrham_g
Enthusiast
Posts: 34
Liked: 1 time
Joined: Dec 29, 2011 9:41 pm
Full Name: Avi
Contact:

Re: VM Disk Errors on 2012

Post by avrham_g » May 20, 2014 10:47 am

i see this error on random vm's.

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » May 20, 2014 4:11 pm

Usually these errors happen on VM reboot and SCSI controllers. Let me check if I can ask Microsoft guys on why perfmon is reporting these errors. BTW, is there any correlation in the Guest OS installed in the VMs?

teachdire
Lurker
Posts: 2
Liked: never
Joined: May 21, 2015 1:57 pm
Contact:

Re: VM Disk Errors on 2012

Post by teachdire » May 21, 2015 2:06 pm

Sorry for posting in this older Thread , but i have a similar problem , i get the same error counts as mentioned above.

Server 2012 R2 fully patched as of today.

What i found is :

When i get the message from veeam one :

Error
VM disk errors
21.05.2015 15:31:29
"Errors/min" (50) is above a defined threshold (2)

I find the following entry in my system logfile on the guest machine:

21.05.2015 15:30:51
Im System wurde ein Dienst installiert.

Dienstname: gfiark
Dienstdateiname: system32\drivers\gfiark.sys
Diensttyp: Kernelmodustreiber
Dienststarttyp: Manuell starten
Dienstkonto:

happen after the definiton update of the AV scanner:
21.05.2015 15:30:52
Definitions update downloaded.

Definitions version: 40430

The error occurs only then and sometimes when the veeam backups starts.

Maybe it helps to find out where these errors come from exactly.

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » May 21, 2015 4:58 pm

Hello,

Don't be sorry for posting into existing topic, actually this is the way all users should do :)

In order to help us with troubleshooting, can you please say what Veeam ONE version you're currently at?

Thank you!

teachdire
Lurker
Posts: 2
Liked: never
Joined: May 21, 2015 1:57 pm
Contact:

Re: VM Disk Errors on 2012

Post by teachdire » May 22, 2015 5:51 am

Veeam One 8.0.0.1669

Just checked over night VM Disk errors occur everytime the AV Scanner gets an update on the guest machines, but only 2 of 3 machines on the same cluster volume.

Just uninstalled the AV Scanner for testing.

2 Questions:

Just trying to understand what exactly this errors means i cant find anything in my logfiles execpt i trace it in perfmon.

My servers running fine and smooth, backups are sucessfull inkl Recovery tests.

Any ideas in the meantime where this error comes from ?

Vitaliy S.
Product Manager
Posts: 23001
Liked: 1557 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VM Disk Errors on 2012

Post by Vitaliy S. » May 22, 2015 12:23 pm

What type of virtual disk is this? The reason for my question is that for Pass-through Disk this is expected to have these kind of errors, which can be disregarded according to Microsoft > http://support.microsoft.com/en-us/kb/2624495

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest