v7 killed one of my WS2012 nodes!!!

[jakesterpdx]

It won't even boot at the moment. Just a warning to anyone considering upgrading to v7 - proceed at your own risk. This only hit one of my 5 nodes - the others seem fine - but this one is hosed. Aaaaarggggh

[chrisdearden]

I see support are already on the case. Please let us know the outcome though.

[jakesterpdx]

Yes, support has been great - I was able to boot to Last Known Good. Strange thing is that all the components seemed to have installed successfully, so it's still a mystery why the node failed so spectacularly.

[jrfolkman]

So you did the upgrade to the Host and then rebooted and had the problem or you upgraded and then the server crashed or...? Please follow up if support can determine what the issue was.

Thanks,
Jason

[foggy]

Also would be appreciated if you could post your support case ID for reference. Thanks.

[jakesterpdx]

Case # 00429568
I upgraded the host, it said no reboot was necessary, and then the host mysteriously crashed a couple minutes later. One thing that might be related: when I was trying to figure out what happened,I noticed that I had not updated my management network with the new subnet mask that I implemented about a week ago.

[Gostev]

Moved the topic to Hyper-V forum.

Vast majority of BSOD reports received in support were caused by poorly written antivirus crashing the system due to the presence of another filter driver, so this is generally my first suspect. Most commonly reported in conjunction with file level recovery (this process also installs the driver in run-time).

Although in this case, since the version is new, I would not rule out a bug in the code either... except as far as I know, filter driver did not have any changes comparing 6.5 patch 3. Also, it is quite strange the host did not crash immediately after components upgrade, but after some time.

Anyway, support will need to collect crash dump, and have our developers take a look. From this, it will be obvious which process has crashed the host... no point to be guessing.

[jakesterpdx]

I don't have antivirus running on the host, so that's not it. I've uploaded the logs just now.

[Gostev]

According to the crash dump, this BSOD was brought to you by memcpy function of Windows kernel. Check the physical memory (RAM) on the crashed Hyper-V host.

[jakesterpdx]

The Hyper-V host has been running in production for over 6 months without the hint of a problem, so I'm skeptical that it's the RAM on the host.

[Gostev]

I feel much more skeptical about your belief in the complete and eternal reliability of modern electronics ever heard about CPU wear effect?

In any case, no one can cheat a crash dump. Whatever is there, is the ultimate truth. And when the operating system fails at the most basic low level operation of copying a block of memory, physical RAM is naturally the first thing to suspect. Although of course, things like power glitch, CPU or motherboard's memory controller issue and such are also a possibility.

[jakesterpdx]

Please don't condescend to me, Gostev. Yes, I know about CPU wear, and if the crash had happened a day after the installation of Veeam, I'd be prone to believe that. It happened ONE MINUTE after a perfectly-running cluster had been flawless for 6+ months. Since the installation of v7 I have been in a world of hurt - VMs constantly crashing, and I was promised a tech would be available at midnight on Thursday night and he was a no-show, then I asked for a callback on my ticket at 10am on Friday and again at 2pm on Friday and only just got a callback this morning for the first time since this disaster started. I'm on thin ice here, so while I appreciate anything constructive to figure out the situation, blaming it on my hardware is a huge stretch, in my opinion.

[Gostev]

Hi Jake, since this thread is going nowhere, and my attempts to help you and explain facts about the BSOD seem to only cause grief and frustration on your side, I am locking this topic down. I will reopen it in case of substantial updates on this issue that will be worth sharing with the community.

I'd like to remind that this is not a support forum, as explained when you click New Topic. When discussing technical issues, the team behind this forum provides constructive suggestions based on our own knowledge and information we receive in direct communication with R&D.

Also note that this forum is not paid or essential Veeam service, but rather an open community that everyone is free to use (or not to use) at own discretion. So, if you are not happy with the format of these discussions, or you feel that you are being condescended here (which is never intended), then I recommend you communicate with our technical support only, and not contribute to these forums at all.

Thanks!