Comprehensive data protection for all workloads
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Server 2016 BSOD Bugcheck

Post by Delo123 »

Ah... you see in the KB they tell you to disable interrupt remapping in ESX, thus disabling it good and for all in the bios should be good also. What actually happens is that most pci-express devices (hba's, nics etc...) will stop responding thus crashing the server. This issue has however been known since 2010 orso, so i wonder why it starts popping op now again
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: Server 2016 BSOD Bugcheck

Post by kubimike »

Day 4 and no daily crashes.
lepphce1
Enthusiast
Posts: 31
Liked: 2 times
Joined: Jun 28, 2016 4:40 pm
Contact:

Re: Server 2016 BSOD Bugcheck

Post by lepphce1 » 1 person likes this post

@kubimike Thanks for the instructions, I've enabled the verifier so we will see what happens. I am doing a minor update to the idrac-lifecycle controller right now (sounds the same as what you describe on the HP side), we will see if it helps (which I kind of doubt). I will also check on the interrupt remapping.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: Server 2016 BSOD Bugcheck

Post by kubimike »

Right on :)
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Server 2016 BSOD Bugcheck

Post by Gostev »

I've seen Verifier "fixing" issues before by preventing deadlocks via simply slowing things down. For example, we could never reproduce that nasty Windows Server 2012 data corruption issue I like to talk about at conferences under Verifier, which is why it is still out there (and I wonder if Windows Server 2016 has inherited it too).
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: Server 2016 BSOD Bugcheck

Post by kubimike » 1 person likes this post

Great so, I'll just run verifier forever lol. I see if my server came equipped with the "Turbo" button from back in the day. Maybe I can drop it down to 12mHZ
lepphce1
Enthusiast
Posts: 31
Liked: 2 times
Joined: Jun 28, 2016 4:40 pm
Contact:

Re: Server 2016 BSOD Bugcheck

Post by lepphce1 »

Is there any sense that this issue might be ReFS related? I'm thinking about installing the March update, turning off verifier and seeing if the issue persists.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Server 2016 BSOD Bugcheck

Post by Gostev »

It is very unlikely the issue is related to ReFS, as way too many people are using ReFS for their backup repositories by now.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: Server 2016 BSOD Bugcheck

Post by kubimike »

Really? I think this is an ReFS issue. I read your post wrong so Im reposting. Yes I was going to try just patching it but its stable with verifier turned on. Too scared to patch!
lepphce1
Enthusiast
Posts: 31
Liked: 2 times
Joined: Jun 28, 2016 4:40 pm
Contact:

Re: Server 2016 BSOD Bugcheck

Post by lepphce1 »

Verifier is really slowing down my system, and the BSOD is happening sporadically enough that I figure it's worth a shot. I can always re-enable it. Who knows maybe MS fixed something else?? I'll report back on my findings.
lepphce1
Enthusiast
Posts: 31
Liked: 2 times
Joined: Jun 28, 2016 4:40 pm
Contact:

Re: Server 2016 BSOD Bugcheck

Post by lepphce1 »

@kubimike, my server bombed last night while trying to install the patch. Even though I turned off verifier I had not yet rebooted so it was still running. It looks like there is additional information in the dmp file. What do you need from me to submit to Microsoft? We can sort it out via PM. Thanks!
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: Server 2016 BSOD Bugcheck

Post by kubimike »

@leppche1 crap, this is why I didn't want to install the patch christ! PM SENT! Anything that we make from this I'll post in the forums so everyone can see.
lepphce1
Enthusiast
Posts: 31
Liked: 2 times
Joined: Jun 28, 2016 4:40 pm
Contact:

Re: Server 2016 BSOD Bugcheck

Post by lepphce1 »

@kubimike, I sent you a link to the file on PM. Unfortunately I didn't realize that I was a couple of months behind on patching this thing, so I *hope* we are seeing the same issue to resolve it. My patch management solution sometimes has a mind of it's own...
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: Server 2016 BSOD Bugcheck

Post by kubimike »

Got some good data out of @lepphche1's machine. Seems like msiscsi.sys, is causing these BSODs for him. I was using iSCSI for awhile and since haven't reconfigured it when I reinstalled the OS (Just using HOTADD mode). That probably would explain why I am seeing less crashes as well since its not in use. His crash code with verifier turned on is 0xD5. I've submitted this to Microsoft under my ticket #

Code: Select all

FAILURE_BUCKET_ID:  0xD5_VRF_R_INVALID_msiscsi!iSpProcessIScsiTaskTimeout

BUCKET_ID:  0xD5_VRF_R_INVALID_msiscsi!iSpProcessIScsiTaskTimeout

PRIMARY_PROBLEM_CLASS:  0xD5_VRF_R_INVALID_msiscsi!iSpProcessIScsiTaskTimeout
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: Server 2016 BSOD Bugcheck

Post by kubimike »

@lepphche I just have KBKB3211320 installed at the moment. Aside from SQL patches and Windows Defender what else do you have ? Drop into powershell issue a get-hotfix see whats up
lepphce1
Enthusiast
Posts: 31
Liked: 2 times
Joined: Jun 28, 2016 4:40 pm
Contact:

Re: Server 2016 BSOD Bugcheck

Post by lepphce1 »

Update KB3176936 NT AUTHORITY\SYSTEM 12/21/2016 12:00:00 AM
Update KB3192137 NT AUTHORITY\SYSTEM 9/12/2016 12:00:00 AM
Update KB3199986 NT AUTHORITY\SYSTEM 1/18/2017 12:00:00 AM
Update KB3213522 NT AUTHORITY\SYSTEM 1/30/2017 12:00:00 AM
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: Server 2016 BSOD Bugcheck

Post by kubimike »

OK i was wrong, not only do I have KB3211320 installed but KB3192137 & KB3199986
I dont have KB3176936 & KB3213522 must be included in the updates I have .. :?:
lepphce1
Enthusiast
Posts: 31
Liked: 2 times
Joined: Jun 28, 2016 4:40 pm
Contact:

Re: Server 2016 BSOD Bugcheck

Post by lepphce1 »

Yeah they might be superseded?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Server 2016 BSOD Bugcheck

Post by Mike Resseler »

Theoretically (yeah I know, you can shoot / laugh / ...) if you would install the latest one you should have them all. The way the new update system works is that you can install the latest monthly rollup and it will detect first which you already have installed and what your are missing. Based on that query, it will download only the missing and install those.

So if you have done the latest of March, you should have them all
BLC
Influencer
Posts: 13
Liked: 1 time
Joined: Mar 01, 2017 4:56 pm
Full Name: Bryan Chapnick
Contact:

Re: Server 2016 BSOD Bugcheck

Post by BLC »

I am having the same issue with the same server. Has anyone figured out what the cause is?

Code: Select all

SYSTEM_PRODUCT_NAME:  ProLiant DL380 Gen9

SYSTEM_SKU:  777338-S01

BIOS_VENDOR:  HP

BIOS_VERSION:  P89

BIOS_DATE:  02/17/2017

BASEBOARD_MANUFACTURER:  HP

BASEBOARD_PRODUCT:  ProLiant DL380 Gen9

DUMP_TYPE:  1

BUGCHECK_P1: 1

BUGCHECK_P2: 1e00

BUGCHECK_P3: 0

BUGCHECK_P4: 0

DPC_TIMEOUT_TYPE:  DPC_QUEUE_EXECUTION_TIMEOUT_EXCEEDED

CPU_COUNT: 20

CPU_MHZ: a25

CPU_VENDOR:  GenuineIntel
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Server 2016 BSOD Bugcheck

Post by Mike Resseler »

Bryan,

Have you updated drivers before the BSOD? Looks like a driver is causing this issue
BLC
Influencer
Posts: 13
Liked: 1 time
Joined: Mar 01, 2017 4:56 pm
Full Name: Bryan Chapnick
Contact:

Re: Server 2016 BSOD Bugcheck

Post by BLC »

I have. HP replaced the system board 5 days ago. It crashed again last night twice. It always happens in the middle of the backup window.
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: Server 2016 BSOD Bugcheck

Post by ejenner »

Got the same problem as was reported at the beginning of this topic. All installed about 6-8 weeks ago so much more up to date... but the same server model. QLogic network adapter.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Server 2016 BSOD Bugcheck

Post by foggy »

Have you opened a case with MS?
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: Server 2016 BSOD Bugcheck

Post by ejenner »

No. I don't think our server has enough memory. So I wouldn't want to open a case with MS while there are weaknesses in our configuration. We're also only at the testing phase so there's no urgency to resolve as we're not depending on this installation yet.

We're upgrading the memory very soon, if we keep seeing BSOD after the memory increase then we can make it official and log a ticket.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Server 2016 BSOD Bugcheck

Post by foggy »

Got it, keep us updated.
alfredo@eurex.it
Lurker
Posts: 1
Liked: never
Joined: Oct 17, 2018 3:31 pm
Full Name: Alfredo Barbieri
Contact:

Re: Server 2016 BSOD Bugcheck

Post by alfredo@eurex.it »

Hi, someone have any news, I have an HP Proliant DL 380 Gen9 with same issue. Windows 2016 standard, veeam installed and HPe support says that is not an HW issue so contact MS directly.

Maybe ejenner have some news?

Thanks to everybody
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Server 2016 BSOD Bugcheck

Post by Mike Resseler »

Hi Alfredo,

We haven't received any update from ejenner. However, did you install all updates? Both Windows and HPe firmware and driver updates? In previous cases it mostly was a driver causing this, but it could be something different also
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: Server 2016 BSOD Bugcheck

Post by ejenner » 1 person likes this post

We upgraded our RAM but the servers continued crashing. We didn't have enough so it was worth eliminating.

After some more testing we found that the problem was ccm.exe (Microsoft Configuration Manager Client) which was causing the crashes. It's incompatible when running the combination of Veeam repository role, ReFS formatted disk and Windows Server 2016.

Obviously all the products in isolation work fine or else there would've been widespread hand wringing all over the place. It's only when you put them all together and start running backups that the crashes start to happen.

So for the moment our solution is to take ccm.exe off the repository servers with the incompatible configuration.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Server 2016 BSOD Bugcheck

Post by Mike Resseler »

Ah, thanks for letting us know EJ,

PS: Does that happen to be still SCCM 2012 R2? I heard that the issue was REFS with that version but thought they (SCCM) had fixed it in later rollup updates...
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 262 guests