Comprehensive data protection for all workloads
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Veeam services crash after installing KB3216755, KB4010672

Post by mkretzer »

Hello,

Case 02059399

we migrated all our backups back from REFS to NTFS but still installed the MS hotfix KB 3216755.

Now our Veeam services crash after ~12 hours with System.OutOfMemoryException.

There were no actions done on the REFS since the last Reboot and the Server has 128 GB of RAM. But there was at least one Veeam.Backup.Service.exe which used 32 GB RAM...

Markus
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by mkretzer »

More info: at the time of the last crash there were only a few copy jobs and one big tape job running...

It happened 2 times in 24 hours.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by mkretzer »

After talking to Veeam support we uninstalled the MS update. System behaves normal again but REFS will be much more unstable.

Gladly our move back to NTFS was already finished...
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam services crash after installing KB3216755

Post by Gostev »

As I have already warned in the corresponding thread, per Microsoft the current KB3216755 package seems to be in the "pre-release preview" state, so I would not recommending deploying it in production environments until it is officially released through Windows Update.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by mkretzer » 1 person likes this post

@Gostev thank you - i just wonder why we are the only ones with this issue. I just uploaded a 40 GB process dump of the veeam backup manager process.
Maybe your team finds something as well :-)
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam services crash after installing KB3216755

Post by Gostev » 1 person likes this post

Fantastic, thanks for doing this.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by mkretzer »

Whats really interesting is that the Veeam Services which took the 40 GB now after the removal of the update and several backups only takes 94 MB
mih
Novice
Posts: 8
Liked: never
Joined: Nov 03, 2014 11:25 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by mih »

Just a small "me too" about veeam.backup.manager processes taking over 60 GB of ram since applying KB2316755.
i have uninstalled it. hopefully the info from mkretzer can help you :)
JvZanten
Novice
Posts: 8
Liked: 1 time
Joined: Apr 01, 2015 12:50 pm
Full Name: Johan van Zanten
Contact:

Re: Veeam services crash after installing KB3216755

Post by JvZanten »

We experience the same issue, after installing the kb3216755 hotfix the veeam services run out of memory and crash
SeandG
Influencer
Posts: 15
Liked: 1 time
Joined: Jan 30, 2017 10:38 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by SeandG »

Same issue for us too, after installing kb3216755 the Veeam services/server run out of memory.

We get several Resource-Exhaustion-Detector events:
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: Veeam.Backup.Service.exe (8916) consumed 107299299328 bytes, Veeam.Backup.EnterpriseService.exe (9236) consumed 2399756288 bytes, and Veeam.Backup.Manager.exe (6908) consumed 1611759616 bytes.

We have a case open with Veeam and the memory dumps are being analysed.

Without kb3216755 we get deduplication data corruption.
SeandG
Influencer
Posts: 15
Liked: 1 time
Joined: Jan 30, 2017 10:38 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by SeandG »

I don't know why Veeam think KB3216755 is pre-release, the MS article https://support.microsoft.com/en-gb/hel ... -kb3216755 doesn't mention pre-release or preview. Can someone clarify?

Thanks
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by mkretzer » 1 person likes this post

SeandG wrote:I don't know why Veeam think KB3216755 is pre-release, the MS article https://support.microsoft.com/en-gb/hel ... -kb3216755 doesn't mention pre-release or preview. Can someone clarify?
The info "This release is only available on the Microsoft Update Catalog website" is kind of a hint. Also it is not generally avaiable in microsoft update....
rendest
Influencer
Posts: 20
Liked: 6 times
Joined: Feb 01, 2017 8:36 pm
Full Name: Stef
Contact:

Re: Veeam services crash after installing KB3216755

Post by rendest » 2 people like this post

SeandG wrote:I don't know why Veeam think KB3216755 is pre-release, the MS article https://support.microsoft.com/en-gb/hel ... -kb3216755 doesn't mention pre-release or preview. Can someone clarify?
math.random()
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam services crash after installing KB3216755

Post by Gostev »

@SeandG click the link in my post for the explanation.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam services crash after installing KB3216755

Post by Gostev »

OK, so the crash dumps were not immediately helpful because the leak is happening outside of the managed code. For now, it does not look like the issue is in our code - but it is really hard to diagnose since leaking is unmanaged memory. However, the leaked memory content seem to contain lots of SQL Server stuff (column names, stored procedure parameters and such), which prompts to ask all of you the following question - do you use local or remote SQL Server?

I know most of you in this thread are affected by the issue, but I also saw one confirmation on the other thread that someone else did not have any problems after installing the update. So, if we can confirm that all of you affected have local SQL Server installed on your backup server, and those who are not have remote SQL Server - this will be a big step forward in the investigation.

Thanks!
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam services crash after installing KB3216755

Post by Gostev » 2 people like this post

Never mind, we seem to have found the root cause. With a high probability, leaking is System.Data.dll, which KB3216755 happens to replace of course.

We were even able to reproduce the issue without Veeam in the picture, by creating a test tool that does nothing except bombing SQL Server with large queries from multiple threads.This tool has also consumed all available system memory very fast and produced a crash dump that is identical to ones submitted by our customers. After uninstalling the update, memory consumption of the test tool remains steady at 26-28MB.

Microsoft has been notified.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Veeam services crash after installing KB3216755

Post by mkaec »

Awesome work!
rendest
Influencer
Posts: 20
Liked: 6 times
Joined: Feb 01, 2017 8:36 pm
Full Name: Stef
Contact:

Re: Veeam services crash after installing KB3216755

Post by rendest »

Nice work! Microsoft could learn something from that!
vanhovejens
Service Provider
Posts: 7
Liked: never
Joined: Apr 05, 2013 4:35 pm
Full Name: Jens Van Hove
Contact:

Re: Veeam services crash after installing KB3216755

Post by vanhovejens »

Hi, Please keep us posted to with the progress from Microsoft (hopefully a quick & solid fix)
We've also go an open Veeam case to keep us notified as we still experience impact on our environment: Veeam Case # 02060809
Thanks for the quick analysis!
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam services crash after installing KB3216755

Post by Gostev »

OK, so people I know put me through to people I don't know, but I assume they are the right people to know what's going on. Long story short, it would appear I was spot on above regarding the current build being a pre-release preview as it comes to Windows Server 2016 specifically:
Someone at Microsoft wrote:It's interesting that KB 3216755 was only made available for client builds and people are installing it on WS 2016 Server. It wasn't meant to be installed on Server but I've confirmed it does not block installation.
Nevertheless, it does not mean they were aware about the issue - we may very well have prevented a worldwide apocalypse following the next week's Patch Tuesday! ;)
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam services crash after installing KB3216755

Post by Gostev »

Microsoft Update Catalog listing actually indicates this build is for Windows 10 only
http://www.catalog.update.microsoft.com ... =KB3216755
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam services crash after installing KB3216755

Post by Gostev »

OK, it would appear there is some confusion within Microsoft about the applicability of this update.

Check out this January 30th post on the official blog of the Windows Server storage engineering teams:
Windows Server 2016 Data Deduplication users: please install KB3216755!

Pretty clear call to action this... so hard to blame Microsoft support for recommending you the same.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by mkretzer »

@Gostev Yes, local SQL 2016
mih
Novice
Posts: 8
Liked: never
Joined: Nov 03, 2014 11:25 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by mih »

same, local SQL.
vanhovejens
Service Provider
Posts: 7
Liked: never
Joined: Apr 05, 2013 4:35 pm
Full Name: Jens Van Hove
Contact:

Re: Veeam services crash after installing KB3216755

Post by vanhovejens »

Remote SQL 2014 Enterprise, still affected or will it only popup with local SQL installs ?
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by mkretzer »

@vanhovejens I think that is the question that Gostev asked here...
vanhovejens
Service Provider
Posts: 7
Liked: never
Joined: Apr 05, 2013 4:35 pm
Full Name: Jens Van Hove
Contact:

Re: Veeam services crash after installing KB3216755

Post by vanhovejens »

:-) could be, we haven't gambled installing the KB as soon as we've read this thread.
Probably our environment shouldn't be impacted as long as we don't apply the patch on our Veeam management server?
We've got dedicated 'veeam repository servers' these shouldn't really rely on high SQL Load so we might take the risk to apply the patch.
mih
Novice
Posts: 8
Liked: never
Joined: Nov 03, 2014 11:25 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by mih »

i have KB3216755 installed on our remote repository server, that gives no problems. Also, windows dedupe has stopped corrupting our 4 TB vbk files. So that part of the KB seems to be working.
Its on our local "all in one" veeam server, that the memory problem shows.
SeandG
Influencer
Posts: 15
Liked: 1 time
Joined: Jan 30, 2017 10:38 am
Contact:

Re: Veeam services crash after installing KB3216755

Post by SeandG »

We have locally installed SQL Express, the one installed by Veeam B&R at setup time.
We are also planning on going back to 2012R2, we can't use the fix because of the memory issues and we can't not use it because of the deduplication corruption.
Gostev wrote:Never mind, we seem to have found the root cause. With a high probability, leaking is System.Data.dll, which KB3216755 happens to replace of course.

We were even able to reproduce the issue without Veeam in the picture, by creating a test tool that does nothing except bombing SQL Server with large queries from multiple threads.This tool has also consumed all available system memory very fast and produced a crash dump that is identical to ones submitted by our customers. After uninstalling the update, memory consumption of the test tool remains steady at 26-28MB.

Microsoft has been notified.
Great find!

Does this mean that if we move the Veeam DB to an remote SQL server and uninstall SQL from the B&R server the issue will be resolved for us?

Thanks
Sean
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam services crash after installing KB3216755

Post by Gostev »

@Sean the current version of the update should not be installed on Windows Server 2016 at all, as it is intended for Windows 10 only (see my following posts after the one you have quoted). But if you are ready to ignore this fact - then basically, you should not install the update on the Veeam backup server, as it will screw up our communication with the SQL Server (and whether SQL is local or remote does not matter). For example, you should be OK installing it or remote backup repository servers, as mih mentioned in his post right above yours.
Post Reply

Who is online

Users browsing this forum: FelixW, MartinO, Regnor and 153 guests