Server 2016 BSOD Bugcheck

Availability for the Always-On Enterprise

Server 2016 BSOD Bugcheck

Veeam Logoby kubimike » Tue Feb 07, 2017 3:18 pm 2 people like this post

Just a heads up to anyone that is running Server 2016 w/SQL Express 2012 and they install SQL 2016. My Veeam box became very unstable and would only stay up for an hour Bugcheck: 0x00000133 (0x0000000000000001, 0x0000000000001e00, 0x0000000000000000, 0x0000000000000000) . Mind you this was on brand new hardware, HP DL380 G9 + Veeam 9.5. I do have a ticket open with Microsoft. They're doing a crash dump analysis to find out why exactly.
After installing the following updates it stopped:

Code: Select all
Installation Successful: Windows successfully installed the following update: Security Update for SQL Server 2012 Service Pack 3 GDR (KB3194721)

Installation Successful: Windows successfully installed the following update: Security Update for SQL Server 2016 RTM GDR (KB3194716)

Installation Successful: Windows successfully installed the following update: Definition Update for Windows Defender - KB2267602 (Definition 1.235.2160.0)

Installation Successful: Windows successfully installed the following update: Update for Windows Server 2016 for x64-based Systems (KB3211320)

Code: Select all
0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DPC_WATCHDOG_VIOLATION (133)
The DPC watchdog detected a prolonged run time at an IRQL of DISPATCH_LEVEL
or above.
Arguments:
Arg1: 0000000000000001, The system cumulatively spent an extended period of time at
DISPATCH_LEVEL or above. The offending component can usually be
identified with a stack trace.
Arg2: 0000000000001e00, The watchdog period.
Arg3: 0000000000000000
Arg4: 0000000000000000
kubimike
Expert
 
Posts: 196
Liked: 22 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: Server 2016 BSOD Bugcheck

Veeam Logoby kubimike » Fri Feb 10, 2017 2:47 pm

Hello, I know Veeam has fingers into back channels for Microsoft. Could someone on your staff point out Microsoft Case ID #117020615276541 , Windows 2016 crashes. Not sure if its related to the other threads I see about ReFS / Dedupes etc but not having a stable box for backups isn't fun. They do have a memory.dump from me and I've been waiting since Monday to hear about their findings. Thank you.
kubimike
Expert
 
Posts: 196
Liked: 22 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: Server 2016 BSOD Bugcheck

Veeam Logoby Gostev » Fri Feb 10, 2017 3:30 pm

Mike, I'd love to help - but as soon as I start leveraging my back channel for all sorts of support case escalations, they will obviously cut me out in no time ;) so, I don't escalate even our very own support cases with Microsoft, and keep this channel for internally confirmed critical issues impacting large amount of customers only.
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Server 2016 BSOD Bugcheck

Veeam Logoby kubimike » Fri Feb 10, 2017 4:32 pm

@Gostev OK we wouldn't want that. Figured I'd ask. As a thread update, I've reinstalled the OS w/o SQL 2016. Box still freezes and dies after about 14+ hours of use. HPs embedded tools turned up no hardware issues.
kubimike
Expert
 
Posts: 196
Liked: 22 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: Server 2016 BSOD Bugcheck

Veeam Logoby Gostev » Fri Feb 10, 2017 7:07 pm

Did you check if some process might be leaking memory?
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Server 2016 BSOD Bugcheck

Veeam Logoby kubimike » Fri Feb 10, 2017 7:55 pm

Well MSFT called back said they indeed found a problem they thought was fixed in Win 2016 https://support.microsoft.com/en-us/help/3140219/-0x00000133-stop-error-after-you-install-hotfix-3061460-in-windows-server-2012-r2

So in order to write a new fix we went into verifier.exe and turn on a bunch of switches as well as making sure we get a kernal mode dump this time too. Now we have to sit back and wait for it to dump again in order for them to collect that data and write a new fix. Joys of 2016 I guess :roll: :( :D
kubimike
Expert
 
Posts: 196
Liked: 22 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: Server 2016 BSOD Bugcheck

Veeam Logoby golmic » Tue Feb 21, 2017 12:36 pm

Hi,
have you heard anything from MS?
I think we have nearly the same problem.
golmic
Lurker
 
Posts: 2
Liked: never
Joined: Fri Nov 25, 2016 6:02 am
Full Name: Michael Goll

Re: Server 2016 BSOD Bugcheck

Veeam Logoby Delo123 » Tue Feb 21, 2017 3:55 pm 1 person likes this post

We have Server 2016 running with SQL 2016 and 9.5 latest patch & ms updates running for just over a month now on physical intel barebone, Not a single issue. YOu guys seeing issues when backuping / restoring or seomthing else? Or also when idle?
Delo123
Expert
 
Posts: 334
Liked: 93 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Server 2016 BSOD Bugcheck

Veeam Logoby kubimike » Tue Feb 21, 2017 7:15 pm 1 person likes this post

@Golmic, what type of network adapters do you have? Are they teamed ? Which bugcheck error are you getting ? Looks like my Broadcom adapter drivers are the issue. Bought so HP/Intel 10GB adapters to team with now. So far so good. I have another HP box with integrated Broadcom adapters running Windows 2012 R2 that is also teamed and it crashes as well.

@Delo123 Same question what type of Network adapter do you have? And is it teamed ?
kubimike
Expert
 
Posts: 196
Liked: 22 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: Server 2016 BSOD Bugcheck

Veeam Logoby Gostev » Tue Feb 21, 2017 7:50 pm

kubimike wrote:Broadcom adapters

Ah, our technical support's favorite. I like to say that instead of building complex "Chaos Monkey" type of tests for your networks, you could just swap a few NICs to ones from Broadcom, and you're good to go.
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Server 2016 BSOD Bugcheck

Veeam Logoby kubimike » Tue Feb 21, 2017 7:57 pm

Why did HP move away from QLogic and Intel on for onboard stuff? Its not that their servers got any CHEAPER! . So far so good on the Intel 561t's!
kubimike
Expert
 
Posts: 196
Liked: 22 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: Server 2016 BSOD Bugcheck

Veeam Logoby nmdange » Tue Feb 21, 2017 8:00 pm

I would definitely agree with using Intel NICs over Broadcom! If you are looking to do SMB Direct, then Mellanox is the way to go.
nmdange
Expert
 
Posts: 158
Liked: 48 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: Server 2016 BSOD Bugcheck

Veeam Logoby Delo123 » Wed Feb 22, 2017 8:19 am

We also use only Intel Nics (X540-T2 and onboard X540-AT2) Intel drivers but teamed with windows in Server 2016 (LACP) and Qlogic FC HBA's.
Not sure about the Broadcoms but maybe there things like offloading to check for issues.
Delo123
Expert
 
Posts: 334
Liked: 93 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Server 2016 BSOD Bugcheck

Veeam Logoby kubimike » Wed Feb 22, 2017 3:13 pm

Came into the office this morning, the veeam box was hosed again. Even though the iLO the screen was black. Power cycled it an now Im up again. Opening the case again with Microsoft, thinking even though I've disabled on the onboard Broadcom NICs perhaps the driver is still loading? I could try reinstalling the OS again. What a hassle.
kubimike
Expert
 
Posts: 196
Liked: 22 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: Server 2016 BSOD Bugcheck

Veeam Logoby lepphce1 » Tue Feb 28, 2017 8:08 pm

@kubimike
I am seeing this issue as well, always happens during backups. Intel i350-t NICs, latest drivers from Intel. I do have MS LACP teaming enabled on the access side. But the major throughput is via SAN iSCSI NICs that are not teamed. Server was performing flawlessly on 2008R2 and 2012R2 (Dell R720XD).

Is there reason to believe that teaming is the issue? Because I can easily turn this off while we wait for a fix...
lepphce1
Enthusiast
 
Posts: 27
Liked: 2 times
Joined: Tue Jun 28, 2016 4:40 pm

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot], mcz, ncapponi and 53 guests