Server 2016 BSOD Bugcheck

[Delo123]

Ah... you see in the KB they tell you to disable interrupt remapping in ESX, thus disabling it good and for all in the bios should be good also. What actually happens is that most pci-express devices (hba's, nics etc...) will stop responding thus crashing the server. This issue has however been known since 2010 orso, so i wonder why it starts popping op now again

[kubimike]

Day 4 and no daily crashes.

[lepphce1]

@kubimike Thanks for the instructions, I've enabled the verifier so we will see what happens. I am doing a minor update to the idrac-lifecycle controller right now (sounds the same as what you describe on the HP side), we will see if it helps (which I kind of doubt). I will also check on the interrupt remapping.

[kubimike]

Right on

[Gostev]

I've seen Verifier "fixing" issues before by preventing deadlocks via simply slowing things down. For example, we could never reproduce that nasty Windows Server 2012 data corruption issue I like to talk about at conferences under Verifier, which is why it is still out there (and I wonder if Windows Server 2016 has inherited it too).

[kubimike]

Great so, I'll just run verifier forever lol. I see if my server came equipped with the "Turbo" button from back in the day. Maybe I can drop it down to 12mHZ

[lepphce1]

Is there any sense that this issue might be ReFS related? I'm thinking about installing the March update, turning off verifier and seeing if the issue persists.

[Gostev]

It is very unlikely the issue is related to ReFS, as way too many people are using ReFS for their backup repositories by now.

[kubimike]

Really? I think this is an ReFS issue. I read your post wrong so Im reposting. Yes I was going to try just patching it but its stable with verifier turned on. Too scared to patch!

[lepphce1]

Verifier is really slowing down my system, and the BSOD is happening sporadically enough that I figure it's worth a shot. I can always re-enable it. Who knows maybe MS fixed something else?? I'll report back on my findings.

[lepphce1]

@kubimike, my server bombed last night while trying to install the patch. Even though I turned off verifier I had not yet rebooted so it was still running. It looks like there is additional information in the dmp file. What do you need from me to submit to Microsoft? We can sort it out via PM. Thanks!

[kubimike]

@leppche1 crap, this is why I didn't want to install the patch christ! PM SENT! Anything that we make from this I'll post in the forums so everyone can see.

[lepphce1]

@kubimike, I sent you a link to the file on PM. Unfortunately I didn't realize that I was a couple of months behind on patching this thing, so I *hope* we are seeing the same issue to resolve it. My patch management solution sometimes has a mind of it's own...

[kubimike]

Got some good data out of @lepphche1's machine. Seems like msiscsi.sys, is causing these BSODs for him. I was using iSCSI for awhile and since haven't reconfigured it when I reinstalled the OS (Just using HOTADD mode). That probably would explain why I am seeing less crashes as well since its not in use. His crash code with verifier turned on is 0xD5. I've submitted this to Microsoft under my ticket #

Code: Select all

FAILURE_BUCKET_ID:  0xD5_VRF_R_INVALID_msiscsi!iSpProcessIScsiTaskTimeout

BUCKET_ID:  0xD5_VRF_R_INVALID_msiscsi!iSpProcessIScsiTaskTimeout

PRIMARY_PROBLEM_CLASS:  0xD5_VRF_R_INVALID_msiscsi!iSpProcessIScsiTaskTimeout

[kubimike]

@lepphche I just have KBKB3211320 installed at the moment. Aside from SQL patches and Windows Defender what else do you have ? Drop into powershell issue a get-hotfix see whats up

[lepphce1]

Update KB3176936 NT AUTHORITY\SYSTEM 12/21/2016 12:00:00 AM
Update KB3192137 NT AUTHORITY\SYSTEM 9/12/2016 12:00:00 AM
Update KB3199986 NT AUTHORITY\SYSTEM 1/18/2017 12:00:00 AM
Update KB3213522 NT AUTHORITY\SYSTEM 1/30/2017 12:00:00 AM

[kubimike]

OK i was wrong, not only do I have KB3211320 installed but KB3192137 & KB3199986
I dont have KB3176936 & KB3213522 must be included in the updates I have ..

[lepphce1]

Yeah they might be superseded?

[Mike Resseler]

Theoretically (yeah I know, you can shoot / laugh / ...) if you would install the latest one you should have them all. The way the new update system works is that you can install the latest monthly rollup and it will detect first which you already have installed and what your are missing. Based on that query, it will download only the missing and install those.

So if you have done the latest of March, you should have them all

[BLC]

I am having the same issue with the same server. Has anyone figured out what the cause is?

Code: Select all

SYSTEM_PRODUCT_NAME:  ProLiant DL380 Gen9

SYSTEM_SKU:  777338-S01

BIOS_VENDOR:  HP

BIOS_VERSION:  P89

BIOS_DATE:  02/17/2017

BASEBOARD_MANUFACTURER:  HP

BASEBOARD_PRODUCT:  ProLiant DL380 Gen9

DUMP_TYPE:  1

BUGCHECK_P1: 1

BUGCHECK_P2: 1e00

BUGCHECK_P3: 0

BUGCHECK_P4: 0

DPC_TIMEOUT_TYPE:  DPC_QUEUE_EXECUTION_TIMEOUT_EXCEEDED

CPU_COUNT: 20

CPU_MHZ: a25

CPU_VENDOR:  GenuineIntel

[Mike Resseler]

Bryan,

Have you updated drivers before the BSOD? Looks like a driver is causing this issue

[BLC]

I have. HP replaced the system board 5 days ago. It crashed again last night twice. It always happens in the middle of the backup window.

[ejenner]

Got the same problem as was reported at the beginning of this topic. All installed about 6-8 weeks ago so much more up to date... but the same server model. QLogic network adapter.

[foggy]

Have you opened a case with MS?

[ejenner]

No. I don't think our server has enough memory. So I wouldn't want to open a case with MS while there are weaknesses in our configuration. We're also only at the testing phase so there's no urgency to resolve as we're not depending on this installation yet.

We're upgrading the memory very soon, if we keep seeing BSOD after the memory increase then we can make it official and log a ticket.

[foggy]

Got it, keep us updated.

[alfredo@eurex.it]

Hi, someone have any news, I have an HP Proliant DL 380 Gen9 with same issue. Windows 2016 standard, veeam installed and HPe support says that is not an HW issue so contact MS directly.

Maybe ejenner have some news?

Thanks to everybody

[Mike Resseler]

Hi Alfredo,

We haven't received any update from ejenner. However, did you install all updates? Both Windows and HPe firmware and driver updates? In previous cases it mostly was a driver causing this, but it could be something different also

[ejenner]

We upgraded our RAM but the servers continued crashing. We didn't have enough so it was worth eliminating.

After some more testing we found that the problem was ccm.exe (Microsoft Configuration Manager Client) which was causing the crashes. It's incompatible when running the combination of Veeam repository role, ReFS formatted disk and Windows Server 2016.

Obviously all the products in isolation work fine or else there would've been widespread hand wringing all over the place. It's only when you put them all together and start running backups that the crashes start to happen.

So for the moment our solution is to take ccm.exe off the repository servers with the incompatible configuration.

[Mike Resseler]

Ah, thanks for letting us know EJ,

PS: Does that happen to be still SCCM 2012 R2? I heard that the issue was REFS with that version but thought they (SCCM) had fixed it in later rollup updates...