VEEAM Hyper-V CSV Backup - Blue Screen

Hyper-V specific discussions

VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby fulostix » Mon Dec 26, 2011 10:27 pm

[UPDATE] Based on further investigation, most typically the issue is caused by:
• CSV ownership transfer operation taking too long with certain storage devices. Patch #3 mitigates this issue.
• The presence of 3rd party antivirus software which purposely "brake" some native OS functionality for obfuscation purposes, to make it harder for viruses to penetrate the system. This logic also affects legitimate low-level applications such as our change tracking driver. Excluding your CSV volumes from monitoring, or at least disabling real-time protection on those volumes usually helps.


We have a 2 node csv cluster running Hyper-V. I am having an intermitent problem backing up our csv cluster which causes our resource disk to fail...and then the opposite node blue screens and reboots. For example if I am backing up VMs that are on NODE1...NODE2 will display an error that the cluster disk has failed. Windows event ids 1230 and 1069 will display and then NODE 2 will blue screen and reboot.

This never happened before with our previous hyper-v backup software. Does anyone have any ideas? Thank you
fulostix
Influencer
 
Posts: 13
Liked: never
Joined: Mon Dec 26, 2011 10:20 pm

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Gostev » Tue Dec 27, 2011 7:31 am

If you are using software VSS, I would guess there are some issue with the redirected I/O mode on this cluster.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby fulostix » Wed Dec 28, 2011 1:48 am

That's what it sounds like, but we never had a problem with our previous backup solution. It also used redirected I/O mode for backing up the CSV and it ran without any problems. I'm wondering if VEEAM backs up a little differently because of the changed block technology
fulostix
Influencer
 
Posts: 13
Liked: never
Joined: Mon Dec 26, 2011 10:20 pm

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby foggy » Wed Dec 28, 2011 10:39 am

Do you probably use some kind of virtual storage for your VMs?
foggy
Veeam Software
 
Posts: 14742
Liked: 1079 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Gostev » Wed Dec 28, 2011 10:53 am

Hi, please include your support case number when you post here next time.

Couple of possible reasons:
- Using non-native file system to run your Hyper-V cluster on (there are a few solutions like that, for example Virsto)
- Missing some patches for Hyper-V, see here > Hyper-v R2 Sp1 - CSV Backups causes disk to fail and node to blue screen

To say for sure, we would need our engineers to have a closer log and our product logs, crash dump file, etc.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby aantoniani » Fri Jan 06, 2012 8:33 am

I have experienced the same on a test Veeam installation and I got the following feedback from Veeam support

"This is currently an issue I have been seeing recently. The workaround we currently have in place is to add the Hyper-V as a standalone instead of adding as a cluster in Veeam. Please try adding your Hyper-V on the left side of the console under "Servers" as a stand alone and point your jobs to this instead of the cluster"

I haven't tested the suggested solution yet.
Alessandro Antoniani
aantoniani
Novice
 
Posts: 8
Liked: 2 times
Joined: Fri Jan 06, 2012 8:29 am
Full Name: Alessandro Antoniani

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Vitaliy S. » Sat Jan 07, 2012 7:25 pm

Alessandro, what is your support case number?
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Gostev » Sun Jan 08, 2012 4:31 pm

There is Veeam support KB article on this issue
http://www.veeam.com/kb_articles.html/KB1468

There is also one known Microsoft issue around CSV that can cause BSOD. This might be possible reason for BSOD during backup as well, because CSV ownership transfer will happen automatically during the pre-backup phase in cases when the backed up VM's node is not a CSV owner.
http://social.technet.microsoft.com/For ... 4dd5ebe2e7
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby fulostix » Fri Jan 20, 2012 2:02 pm

Ok, so we have all those patches installed. We went back to using our other Backup Software which is CommVault. We have no problems at all backing up the VMs with CommVault. This problem with the blue screen has to be related to VEEAM itself...unfortunately, I can't do much more testing with VEEAM because it keeps causing our cluster to fail....if anyone has any more info...that would be great and very helpful.
fulostix
Influencer
 
Posts: 13
Liked: never
Joined: Mon Dec 26, 2011 10:20 pm

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Gostev » Fri Jan 20, 2012 5:50 pm

If you could post your support case number, I may be able to provide more info on possible resolution. Thanks.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby aantoniani » Sun Jan 22, 2012 10:18 pm

sorry, I missed your answer: the support case # is 5163923.
Thanks, Alessandro
Alessandro Antoniani
aantoniani
Novice
 
Posts: 8
Liked: 2 times
Joined: Fri Jan 06, 2012 8:29 am
Full Name: Alessandro Antoniani

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby fulostix » Mon Jan 23, 2012 12:51 pm

...and my support case number is 5161625
fulostix
Influencer
 
Posts: 13
Liked: never
Joined: Mon Dec 26, 2011 10:20 pm

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby rlrizzo3 » Fri Jan 27, 2012 6:10 pm

I am having a similar issue and I opened a support ticket about a month ago. Veeam is causing our CSV to fail but none of our hyper-v hosts BSOD. Has any progress been made on this?
rlrizzo3
Lurker
 
Posts: 1
Liked: never
Joined: Fri Jan 27, 2012 6:00 pm
Full Name: Ron

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby Gostev » Sat Jan 28, 2012 12:48 am

Hi Ron - from your description, I cannot be sure if your issue is the same as discussed in this topic.

However, regarding the main issue discussed in this topic, yes - it has been fully understood. This appeared to be known Microsoft issue related to using software VSS provider. One required step of Hyper-V backup process sometimes takes very long time with certain storage devices (for unknown storage-specific reasons), and when such operation takes longer than 60 seconds, it causes the STOP error (as explained in this article). Immediate workaround is to change HangRecoveryAction to 1.

Interestingly enough, this problematic operation never takes more than a few seconds in slowest test labs we have -- so, we could not reproduce this issue to date. However, one of the customers let us see and research this in his own environment, so this certainly helped to actually see the issue.

We cannot really speed up the storage reaction to Hyper-V commands since this is not something we can control, but we are looking at placing some workarounds from our side to reduce the chance of STOP errors happening. We hope that reducing the amount of CSV ownership transfer operations (which is when STOP happens) during the job by optimizing the VM processing order, as well as adding timeouts in a few places to reduce the load on storage should help with this.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Veeam Logoby qed » Fri Feb 24, 2012 9:05 am

we're having this issue, too (hotfixes are installed, etc.). for the others that have reported it in this thread: what kind of storage are you using? we're using a dell md3220 direct attached to two hosts running Hyper-V Server 2008 R2. i was unaware there was even such a thing as a hardware VSS driver until Veeam recommended installing/utilizing it (after these blue screens, i get why), but it turns out that the installer for the VSS hardware provider available from dell fails to install on Hyper-V Server 2008 R2 as due to an "unsupported operating system" error, which i'm sure is just the result of the stupid installer not having "Hyper-V Server R2" in the list of valid OSes. Anyway, just throwing this out there in case anyone else is in a similar situation...
qed
Lurker
 
Posts: 1
Liked: never
Joined: Tue Dec 20, 2011 10:44 pm
Full Name: qed

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 3 guests