v7 killed one of my WS2012 nodes!!!

Hyper-V specific discussions

v7 killed one of my WS2012 nodes!!!

Veeam Logoby jakesterpdx » Thu Aug 15, 2013 6:17 pm

It won't even boot at the moment. Just a warning to anyone considering upgrading to v7 - proceed at your own risk. This only hit one of my 5 nodes - the others seem fine - but this one is hosed. Aaaaarggggh
jakesterpdx
Enthusiast
 
Posts: 28
Liked: 3 times
Joined: Fri Jul 23, 2010 10:10 pm
Full Name: Jake H

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby chrisdearden » Thu Aug 15, 2013 6:48 pm

I see support are already on the case. Please let us know the outcome though.
chrisdearden
Expert
 
Posts: 1529
Liked: 225 times
Joined: Wed Jul 21, 2010 9:47 am
Full Name: Chris Dearden

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby jakesterpdx » Thu Aug 15, 2013 7:38 pm

Yes, support has been great - I was able to boot to Last Known Good. Strange thing is that all the components seemed to have installed successfully, so it's still a mystery why the node failed so spectacularly.
jakesterpdx
Enthusiast
 
Posts: 28
Liked: 3 times
Joined: Fri Jul 23, 2010 10:10 pm
Full Name: Jake H

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby jrfolkman » Thu Aug 15, 2013 8:00 pm

So you did the upgrade to the Host and then rebooted and had the problem or you upgraded and then the server crashed or...? Please follow up if support can determine what the issue was.

Thanks,
Jason
jrfolkman
Novice
 
Posts: 7
Liked: never
Joined: Thu Sep 13, 2012 3:58 pm
Full Name: Jason Folkman

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby foggy » Thu Aug 15, 2013 8:03 pm

Also would be appreciated if you could post your support case ID for reference. Thanks.
foggy
Veeam Software
 
Posts: 14716
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby jakesterpdx » Thu Aug 15, 2013 8:09 pm

Case # 00429568
I upgraded the host, it said no reboot was necessary, and then the host mysteriously crashed a couple minutes later. One thing that might be related: when I was trying to figure out what happened,I noticed that I had not updated my management network with the new subnet mask that I implemented about a week ago.
jakesterpdx
Enthusiast
 
Posts: 28
Liked: 3 times
Joined: Fri Jul 23, 2010 10:10 pm
Full Name: Jake H

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby Gostev » Thu Aug 15, 2013 8:40 pm

Moved the topic to Hyper-V forum.

Vast majority of BSOD reports received in support were caused by poorly written antivirus crashing the system due to the presence of another filter driver, so this is generally my first suspect. Most commonly reported in conjunction with file level recovery (this process also installs the driver in run-time).

Although in this case, since the version is new, I would not rule out a bug in the code either... except as far as I know, filter driver did not have any changes comparing 6.5 patch 3. Also, it is quite strange the host did not crash immediately after components upgrade, but after some time.

Anyway, support will need to collect crash dump, and have our developers take a look. From this, it will be obvious which process has crashed the host... no point to be guessing.
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby jakesterpdx » Thu Aug 15, 2013 8:51 pm

I don't have antivirus running on the host, so that's not it. I've uploaded the logs just now.
jakesterpdx
Enthusiast
 
Posts: 28
Liked: 3 times
Joined: Fri Jul 23, 2010 10:10 pm
Full Name: Jake H

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby Gostev » Fri Aug 16, 2013 10:55 am

According to the crash dump, this BSOD was brought to you by memcpy function of Windows kernel. Check the physical memory (RAM) on the crashed Hyper-V host.
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby jakesterpdx » Mon Aug 19, 2013 3:39 am

The Hyper-V host has been running in production for over 6 months without the hint of a problem, so I'm skeptical that it's the RAM on the host.
jakesterpdx
Enthusiast
 
Posts: 28
Liked: 3 times
Joined: Fri Jul 23, 2010 10:10 pm
Full Name: Jake H

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby Gostev » Mon Aug 19, 2013 8:39 am

I feel much more skeptical about your belief in the complete and eternal reliability of modern electronics ;) ever heard about CPU wear effect?

In any case, no one can cheat a crash dump. Whatever is there, is the ultimate truth. And when the operating system fails at the most basic low level operation of copying a block of memory, physical RAM is naturally the first thing to suspect. Although of course, things like power glitch, CPU or motherboard's memory controller issue and such are also a possibility.
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby jakesterpdx » Mon Aug 19, 2013 3:41 pm

Please don't condescend to me, Gostev. Yes, I know about CPU wear, and if the crash had happened a day after the installation of Veeam, I'd be prone to believe that. It happened ONE MINUTE after a perfectly-running cluster had been flawless for 6+ months. Since the installation of v7 I have been in a world of hurt - VMs constantly crashing, and I was promised a tech would be available at midnight on Thursday night and he was a no-show, then I asked for a callback on my ticket at 10am on Friday and again at 2pm on Friday and only just got a callback this morning for the first time since this disaster started. I'm on thin ice here, so while I appreciate anything constructive to figure out the situation, blaming it on my hardware is a huge stretch, in my opinion.
jakesterpdx
Enthusiast
 
Posts: 28
Liked: 3 times
Joined: Fri Jul 23, 2010 10:10 pm
Full Name: Jake H

Re: v7 killed one of my WS2012 nodes!!!

Veeam Logoby Gostev » Mon Aug 19, 2013 3:57 pm

Hi Jake, since this thread is going nowhere, and my attempts to help you and explain facts about the BSOD seem to only cause grief and frustration on your side, I am locking this topic down. I will reopen it in case of substantial updates on this issue that will be worth sharing with the community.

I'd like to remind that this is not a support forum, as explained when you click New Topic. When discussing technical issues, the team behind this forum provides constructive suggestions based on our own knowledge and information we receive in direct communication with R&D.

Also note that this forum is not paid or essential Veeam service, but rather an open community that everyone is free to use (or not to use) at own discretion. So, if you are not happy with the format of these discussions, or you feel that you are being condescended here (which is never intended), then I recommend you communicate with our technical support only, and not contribute to these forums at all.

Thanks!
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland


Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 7 guests