Veeam services crash after installing KB3216755, KB4010672

Availability for the Always-On Enterprise

Re: Veeam services crash after installing KB3216755

Veeam Logoby Gostev » Mon Feb 06, 2017 7:49 pm 2 people like this post

Never mind, we seem to have found the root cause. With a high probability, leaking is System.Data.dll, which KB3216755 happens to replace of course.

We were even able to reproduce the issue without Veeam in the picture, by creating a test tool that does nothing except bombing SQL Server with large queries from multiple threads.This tool has also consumed all available system memory very fast and produced a crash dump that is identical to ones submitted by our customers. After uninstalling the update, memory consumption of the test tool remains steady at 26-28MB.

Microsoft has been notified.
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Veeam services crash after installing KB3216755

Veeam Logoby mkaec » Mon Feb 06, 2017 7:50 pm

Awesome work!
mkaec
Expert
 
Posts: 173
Liked: 43 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Veeam services crash after installing KB3216755

Veeam Logoby rendest » Mon Feb 06, 2017 8:08 pm

Nice work! Microsoft could learn something from that!
rendest
Influencer
 
Posts: 18
Liked: 5 times
Joined: Wed Feb 01, 2017 8:36 pm
Full Name: Stef

Re: Veeam services crash after installing KB3216755

Veeam Logoby vanhovejens » Mon Feb 06, 2017 8:18 pm

Hi, Please keep us posted to with the progress from Microsoft (hopefully a quick & solid fix)
We've also go an open Veeam case to keep us notified as we still experience impact on our environment: Veeam Case # 02060809
Thanks for the quick analysis!
vanhovejens
Service Provider
 
Posts: 7
Liked: never
Joined: Fri Apr 05, 2013 4:35 pm
Full Name: Jens Van Hove

Re: Veeam services crash after installing KB3216755

Veeam Logoby Gostev » Mon Feb 06, 2017 10:47 pm

OK, so people I know put me through to people I don't know, but I assume they are the right people to know what's going on. Long story short, it would appear I was spot on above regarding the current build being a pre-release preview as it comes to Windows Server 2016 specifically:

Someone at Microsoft wrote:It's interesting that KB 3216755 was only made available for client builds and people are installing it on WS 2016 Server. It wasn't meant to be installed on Server but I've confirmed it does not block installation.

Nevertheless, it does not mean they were aware about the issue - we may very well have prevented a worldwide apocalypse following the next week's Patch Tuesday! ;)
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Veeam services crash after installing KB3216755

Veeam Logoby Gostev » Mon Feb 06, 2017 10:56 pm

Microsoft Update Catalog listing actually indicates this build is for Windows 10 only
http://www.catalog.update.microsoft.com ... =KB3216755
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Veeam services crash after installing KB3216755

Veeam Logoby Gostev » Mon Feb 06, 2017 11:14 pm

OK, it would appear there is some confusion within Microsoft about the applicability of this update.

Check out this January 30th post on the official blog of the Windows Server storage engineering teams:
Windows Server 2016 Data Deduplication users: please install KB3216755!

Pretty clear call to action this... so hard to blame Microsoft support for recommending you the same.
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Veeam services crash after installing KB3216755

Veeam Logoby mkretzer » Tue Feb 07, 2017 5:42 am

@Gostev Yes, local SQL 2016
mkretzer
Expert
 
Posts: 275
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby mih » Tue Feb 07, 2017 6:54 am

same, local SQL.
mih
Novice
 
Posts: 4
Liked: never
Joined: Mon Nov 03, 2014 11:25 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby vanhovejens » Tue Feb 07, 2017 7:32 am

Remote SQL 2014 Enterprise, still affected or will it only popup with local SQL installs ?
vanhovejens
Service Provider
 
Posts: 7
Liked: never
Joined: Fri Apr 05, 2013 4:35 pm
Full Name: Jens Van Hove

Re: Veeam services crash after installing KB3216755

Veeam Logoby mkretzer » Tue Feb 07, 2017 8:25 am

@vanhovejens I think that is the question that Gostev asked here...
mkretzer
Expert
 
Posts: 275
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby vanhovejens » Tue Feb 07, 2017 8:50 am

:-) could be, we haven't gambled installing the KB as soon as we've read this thread.
Probably our environment shouldn't be impacted as long as we don't apply the patch on our Veeam management server?
We've got dedicated 'veeam repository servers' these shouldn't really rely on high SQL Load so we might take the risk to apply the patch.
vanhovejens
Service Provider
 
Posts: 7
Liked: never
Joined: Fri Apr 05, 2013 4:35 pm
Full Name: Jens Van Hove

Re: Veeam services crash after installing KB3216755

Veeam Logoby mih » Tue Feb 07, 2017 9:28 am

i have KB3216755 installed on our remote repository server, that gives no problems. Also, windows dedupe has stopped corrupting our 4 TB vbk files. So that part of the KB seems to be working.
Its on our local "all in one" veeam server, that the memory problem shows.
mih
Novice
 
Posts: 4
Liked: never
Joined: Mon Nov 03, 2014 11:25 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby SeandG » Tue Feb 07, 2017 9:53 am

We have locally installed SQL Express, the one installed by Veeam B&R at setup time.
We are also planning on going back to 2012R2, we can't use the fix because of the memory issues and we can't not use it because of the deduplication corruption.

Gostev wrote:Never mind, we seem to have found the root cause. With a high probability, leaking is System.Data.dll, which KB3216755 happens to replace of course.

We were even able to reproduce the issue without Veeam in the picture, by creating a test tool that does nothing except bombing SQL Server with large queries from multiple threads.This tool has also consumed all available system memory very fast and produced a crash dump that is identical to ones submitted by our customers. After uninstalling the update, memory consumption of the test tool remains steady at 26-28MB.

Microsoft has been notified.

Great find!

Does this mean that if we move the Veeam DB to an remote SQL server and uninstall SQL from the B&R server the issue will be resolved for us?

Thanks
Sean
SeandG
Influencer
 
Posts: 14
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby Gostev » Tue Feb 07, 2017 12:31 pm

@Sean the current version of the update should not be installed on Windows Server 2016 at all, as it is intended for Windows 10 only (see my following posts after the one you have quoted). But if you are ready to ignore this fact - then basically, you should not install the update on the Veeam backup server, as it will screw up our communication with the SQL Server (and whether SQL is local or remote does not matter). For example, you should be OK installing it or remote backup repository servers, as mih mentioned in his post right above yours.
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot] and 28 guests