Host-based backup of Microsoft Hyper-V VMs.
fulostix
Influencer
Posts: 13
Liked: never
Joined: Dec 26, 2011 10:20 pm
Contact:

VEEAM Hyper-V CSV Backup - Blue Screen

Post by fulostix »

[UPDATE] Based on further investigation, most typically the issue is caused by:
• CSV ownership transfer operation taking too long with certain storage devices. Patch #3 mitigates this issue.
• The presence of 3rd party antivirus software which purposely "brake" some native OS functionality for obfuscation purposes, to make it harder for viruses to penetrate the system. This logic also affects legitimate low-level applications such as our change tracking driver. Excluding your CSV volumes from monitoring, or at least disabling real-time protection on those volumes usually helps.


We have a 2 node csv cluster running Hyper-V. I am having an intermitent problem backing up our csv cluster which causes our resource disk to fail...and then the opposite node blue screens and reboots. For example if I am backing up VMs that are on NODE1...NODE2 will display an error that the cluster disk has failed. Windows event ids 1230 and 1069 will display and then NODE 2 will blue screen and reboot.

This never happened before with our previous hyper-v backup software. Does anyone have any ideas? Thank you
Gostev
Chief Product Officer
Posts: 31746
Liked: 7250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

If you are using software VSS, I would guess there are some issue with the redirected I/O mode on this cluster.
fulostix
Influencer
Posts: 13
Liked: never
Joined: Dec 26, 2011 10:20 pm
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by fulostix »

That's what it sounds like, but we never had a problem with our previous backup solution. It also used redirected I/O mode for backing up the CSV and it ran without any problems. I'm wondering if VEEAM backs up a little differently because of the changed block technology
foggy
Veeam Software
Posts: 21133
Liked: 2140 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by foggy »

Do you probably use some kind of virtual storage for your VMs?
Gostev
Chief Product Officer
Posts: 31746
Liked: 7250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

Hi, please include your support case number when you post here next time.

Couple of possible reasons:
- Using non-native file system to run your Hyper-V cluster on (there are a few solutions like that, for example Virsto)
- Missing some patches for Hyper-V, see here > Hyper-v R2 Sp1 - CSV Backups causes disk to fail and node to blue screen

To say for sure, we would need our engineers to have a closer log and our product logs, crash dump file, etc.
aantoniani
Influencer
Posts: 11
Liked: 2 times
Joined: Jan 06, 2012 8:29 am
Full Name: Alessandro Antoniani
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by aantoniani »

I have experienced the same on a test Veeam installation and I got the following feedback from Veeam support

"This is currently an issue I have been seeing recently. The workaround we currently have in place is to add the Hyper-V as a standalone instead of adding as a cluster in Veeam. Please try adding your Hyper-V on the left side of the console under "Servers" as a stand alone and point your jobs to this instead of the cluster"

I haven't tested the suggested solution yet.
Alessandro Antoniani
Vitaliy S.
VP, Product Management
Posts: 27343
Liked: 2784 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Vitaliy S. »

Alessandro, what is your support case number?
Gostev
Chief Product Officer
Posts: 31746
Liked: 7250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

There is Veeam support KB article on this issue
http://www.veeam.com/kb_articles.html/KB1468

There is also one known Microsoft issue around CSV that can cause BSOD. This might be possible reason for BSOD during backup as well, because CSV ownership transfer will happen automatically during the pre-backup phase in cases when the backed up VM's node is not a CSV owner.
http://social.technet.microsoft.com/For ... 4dd5ebe2e7
fulostix
Influencer
Posts: 13
Liked: never
Joined: Dec 26, 2011 10:20 pm
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by fulostix »

Ok, so we have all those patches installed. We went back to using our other Backup Software which is CommVault. We have no problems at all backing up the VMs with CommVault. This problem with the blue screen has to be related to VEEAM itself...unfortunately, I can't do much more testing with VEEAM because it keeps causing our cluster to fail....if anyone has any more info...that would be great and very helpful.
Gostev
Chief Product Officer
Posts: 31746
Liked: 7250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

If you could post your support case number, I may be able to provide more info on possible resolution. Thanks.
aantoniani
Influencer
Posts: 11
Liked: 2 times
Joined: Jan 06, 2012 8:29 am
Full Name: Alessandro Antoniani
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by aantoniani »

sorry, I missed your answer: the support case # is 5163923.
Thanks, Alessandro
Alessandro Antoniani
fulostix
Influencer
Posts: 13
Liked: never
Joined: Dec 26, 2011 10:20 pm
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by fulostix »

...and my support case number is 5161625
rlrizzo3
Lurker
Posts: 1
Liked: never
Joined: Jan 27, 2012 6:00 pm
Full Name: Ron
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by rlrizzo3 »

I am having a similar issue and I opened a support ticket about a month ago. Veeam is causing our CSV to fail but none of our hyper-v hosts BSOD. Has any progress been made on this?
Gostev
Chief Product Officer
Posts: 31746
Liked: 7250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

Hi Ron - from your description, I cannot be sure if your issue is the same as discussed in this topic.

However, regarding the main issue discussed in this topic, yes - it has been fully understood. This appeared to be known Microsoft issue related to using software VSS provider. One required step of Hyper-V backup process sometimes takes very long time with certain storage devices (for unknown storage-specific reasons), and when such operation takes longer than 60 seconds, it causes the STOP error (as explained in this article). Immediate workaround is to change HangRecoveryAction to 1.

Interestingly enough, this problematic operation never takes more than a few seconds in slowest test labs we have -- so, we could not reproduce this issue to date. However, one of the customers let us see and research this in his own environment, so this certainly helped to actually see the issue.

We cannot really speed up the storage reaction to Hyper-V commands since this is not something we can control, but we are looking at placing some workarounds from our side to reduce the chance of STOP errors happening. We hope that reducing the amount of CSV ownership transfer operations (which is when STOP happens) during the job by optimizing the VM processing order, as well as adding timeouts in a few places to reduce the load on storage should help with this.
qed
Lurker
Posts: 1
Liked: never
Joined: Dec 20, 2011 10:44 pm
Full Name: qed
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by qed »

we're having this issue, too (hotfixes are installed, etc.). for the others that have reported it in this thread: what kind of storage are you using? we're using a dell md3220 direct attached to two hosts running Hyper-V Server 2008 R2. i was unaware there was even such a thing as a hardware VSS driver until Veeam recommended installing/utilizing it (after these blue screens, i get why), but it turns out that the installer for the VSS hardware provider available from dell fails to install on Hyper-V Server 2008 R2 as due to an "unsupported operating system" error, which i'm sure is just the result of the stupid installer not having "Hyper-V Server R2" in the list of valid OSes. Anyway, just throwing this out there in case anyone else is in a similar situation...
Gostev
Chief Product Officer
Posts: 31746
Liked: 7250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

Hi, actually the issue was addressed in Patch #3 that was released couple of weeks ago, please see a sticky topic in the main forum.
smurteira
Enthusiast
Posts: 26
Liked: never
Joined: Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by smurteira »

I'm not so sure about that. I installed patch 3 and one of my hosts bluescreened yesterday
Gostev
Chief Product Officer
Posts: 31746
Liked: 7250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

Please open a support case then. Reading through my response above, since this issue is not on the Veeam side, but in the Hyper-V design, patch only introduces some workarounds which dramatically reduce the chance of issue appearing, but does not really "fix" anything. There is, however, a setting that you can change on Hyper-V side that will actually prevent the issue (by disabling bugcheck response to a certain event) - our support will be able to share it with you. But first, it would also be good for us to make sure that the patch was deployed correctly, because frankly, in the past few week we have closed few dozens of similar support cases by deploying patch 3 - and you seem to be the only one for whom the patch did not help... please let me know your support case ID as well. Thanks!
collinp
Expert
Posts: 239
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by collinp »

I am having the same issues as you describe with the same event id's.
foggy
Veeam Software
Posts: 21133
Liked: 2140 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by foggy »

Colin, do you have Patch 3 installed? If yes, it would be great if you could also let our support know about that. Thanks!
collinp
Expert
Posts: 239
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by collinp »

Yes, I have Veeam patch 3 installed as well as the 2 hyper-v hotfixes mentioned earlier in this thread. Case # is 5177545. In the case I uploaded the event logs from both Hyper-v hosts. I get this in the eventlog: The computer has rebooted from a bugcheck. The bugcheck was: 0x0000009e (0xfffffa8062b01b30, 0x00000000000004b0, 0x0000000000000000, 0x0000000000000000). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 030712-28002-01.

Hopefully Veeam support will respond to my case. I am still on the trial edition. I would purchase the license if I could just keep my hyper-v hosts online!
Gostev
Chief Product Officer
Posts: 31746
Liked: 7250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

For what it's worth, if nothing else helps, there is a really simple way to disable bug check response to the Hyper-V event that is causing this.
collinp
Expert
Posts: 239
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by collinp »

Are you referring to setting this setting to 1? http://blogs.technet.com/b/askcore/arch ... 0009e.aspx

If so, I am worried that setting this to 1 will have worse consequences then a simple reboot of the Hyper-v host.
hopways
Novice
Posts: 6
Liked: 4 times
Joined: Mar 07, 2012 3:36 pm
Full Name: john waugh
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by hopways »

having the same issue, licensed 14 cpu sockets for a client
do not set cluster setting HangRecovery=1 as it freezes the cluster CSV volume of machine being backed up goes into deadlock and you have to find the host that owned that
particulat csv volume and then restart its cluster service the remaining vm's on other hosts will then respond and the vm's on the failed deadlocked node will restart on remaining hosts

i have now gone through a process of updating all possible firmware and driver updates and have set repository to 1 concurrent job and selected single hyper v hosts and not the cluster

will update this when i get results back
veeam support just say install patch 3 - did that !! still doesnt work
Gostev
Chief Product Officer
Posts: 31746
Liked: 7250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by Gostev »

collinp wrote:Are you referring to setting this setting to 1? http://blogs.technet.com/b/askcore/arch ... 0009e.aspx
If so, I am worried that setting this to 1 will have worse consequences then a simple reboot of the Hyper-v host.
You can just increase the timeout instead, that also helps. Again, the root cause of this issue is CSV ownership transfer (which is quite standard cluster operation) taking long time with some storage devices, causing this bugcheck to go off. While backup of VMs from CSV requires that CSV ownership is transferred to the host running the backed up VM.
smurteira
Enthusiast
Posts: 26
Liked: never
Joined: Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by smurteira »

I'm glad to see I'm not the only one having this problem. I also get the 0x0000009e bugcheck. My Luns are on a NetApp storage by the way. Wondering what storage everyone else is using?
collinp
Expert
Posts: 239
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by collinp »

I am using a fiber attached EMC CX4. Hopefully there is a patch 4 that will address this? The funny thing is that I never saw these problems when backing up our test Hyper-v cluster with 5 vm's. But once I started trying to back up our production Hyper-v cluster with 25 VM's (and got in big trouble for taking the environment down night after night), we started seeing these issues.
collinp
Expert
Posts: 239
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by collinp »

I went back and looked at the logs and noticed that the Hyper-v host blue screens usually happen when trying to backup a particular vm. It sometimes occurs on other vm's but most of the time it is this particular vm. There isn't anything unusual about this vm. There are larger vm's in the cluster on slower storage than this one. Are any of you seeing the same thing? Here is the error message that I see on the particular vm when the Hyper-v BSOD occurs.

Code: Select all

2/29/2012 3:26:25 AM :: Unable to create snapshot (Microsoft Software Shadow Copy provider 1.0). Details: This operation returned because the timeout period expired
Failed to prepare cluster shared volume for backup. File name: [C:\ClusterStorage\Volume1\VirtualMachine1\Virtual Machines\37E31A18-C436-4AF1-8641-59899BFFBADD.xml]
--tr:Failed to obtain shadow path for backup item 'C:\ClusterStorage\Volume1\VirtualMachine1\Virtual Machines\37E31A18-C436-4AF1-8641-59899BFFBADD.xml'.
--tr:Failed to perform pre-backup tasks.

2/29/2012 3:59:32 AM :: Unable to allocate processing resources. Error: Unable to allocate resources for taskId: '1a06a110-b78f-4901-9ed6-fc2d21bddcc2'

2/29/2012 3:59:32 AM :: Processing finished with errors at 2/29/2012 3:59:32 AM
collinp
Expert
Posts: 239
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by collinp »

The only thing that is different about this VM is that in the VMM console its CPU average is 19% whereas the other vm's show a CPU average between 0% and 4%
smurteira
Enthusiast
Posts: 26
Liked: never
Joined: Feb 22, 2012 1:45 pm
Full Name: Sergio Murteira
Contact:

Re: VEEAM Hyper-V CSV Backup - Blue Screen

Post by smurteira »

I'll have to pay attention to see if this happens on the same one all the time too. When you added your cluster, did you do it by adding the vmm server or did you just add the cluster to veeam?
Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests