Veeam services crash after installing KB3216755, KB4010672

Availability for the Always-On Enterprise

Veeam services crash after installing KB3216755, KB4010672

Veeam Logoby mkretzer » Sat Feb 04, 2017 7:01 am

Hello,

Case 02059399

we migrated all our backups back from REFS to NTFS but still installed the MS hotfix KB 3216755.

Now our Veeam services crash after ~12 hours with System.OutOfMemoryException.

There were no actions done on the REFS since the last Reboot and the Server has 128 GB of RAM. But there was at least one Veeam.Backup.Service.exe which used 32 GB RAM...

Markus
mkretzer
Expert
 
Posts: 275
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby mkretzer » Sat Feb 04, 2017 7:49 am

More info: at the time of the last crash there were only a few copy jobs and one big tape job running...

It happened 2 times in 24 hours.
mkretzer
Expert
 
Posts: 275
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby mkretzer » Sat Feb 04, 2017 2:32 pm

After talking to Veeam support we uninstalled the MS update. System behaves normal again but REFS will be much more unstable.

Gladly our move back to NTFS was already finished...
mkretzer
Expert
 
Posts: 275
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby Gostev » Sat Feb 04, 2017 8:53 pm

As I have already warned in the corresponding thread, per Microsoft the current KB3216755 package seems to be in the "pre-release preview" state, so I would not recommending deploying it in production environments until it is officially released through Windows Update.
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Veeam services crash after installing KB3216755

Veeam Logoby mkretzer » Sun Feb 05, 2017 1:18 am 1 person likes this post

@Gostev thank you - i just wonder why we are the only ones with this issue. I just uploaded a 40 GB process dump of the veeam backup manager process.
Maybe your team finds something as well :-)
mkretzer
Expert
 
Posts: 275
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby Gostev » Sun Feb 05, 2017 2:46 am 1 person likes this post

Fantastic, thanks for doing this.
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Veeam services crash after installing KB3216755

Veeam Logoby mkretzer » Sun Feb 05, 2017 9:32 am

Whats really interesting is that the Veeam Services which took the 40 GB now after the removal of the update and several backups only takes 94 MB
mkretzer
Expert
 
Posts: 275
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby mih » Mon Feb 06, 2017 7:10 am

Just a small "me too" about veeam.backup.manager processes taking over 60 GB of ram since applying KB2316755.
i have uninstalled it. hopefully the info from mkretzer can help you :)
mih
Novice
 
Posts: 4
Liked: never
Joined: Mon Nov 03, 2014 11:25 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby JvZanten » Mon Feb 06, 2017 9:34 am

We experience the same issue, after installing the kb3216755 hotfix the veeam services run out of memory and crash
JvZanten
Novice
 
Posts: 4
Liked: never
Joined: Wed Apr 01, 2015 12:50 pm
Full Name: Johan van Zanten

Re: Veeam services crash after installing KB3216755

Veeam Logoby SeandG » Mon Feb 06, 2017 10:12 am

Same issue for us too, after installing kb3216755 the Veeam services/server run out of memory.

We get several Resource-Exhaustion-Detector events:
Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: Veeam.Backup.Service.exe (8916) consumed 107299299328 bytes, Veeam.Backup.EnterpriseService.exe (9236) consumed 2399756288 bytes, and Veeam.Backup.Manager.exe (6908) consumed 1611759616 bytes.

We have a case open with Veeam and the memory dumps are being analysed.

Without kb3216755 we get deduplication data corruption.
SeandG
Influencer
 
Posts: 14
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby SeandG » Mon Feb 06, 2017 10:28 am

I don't know why Veeam think KB3216755 is pre-release, the MS article https://support.microsoft.com/en-gb/hel ... -kb3216755 doesn't mention pre-release or preview. Can someone clarify?

Thanks
SeandG
Influencer
 
Posts: 14
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby mkretzer » Mon Feb 06, 2017 10:36 am 1 person likes this post

SeandG wrote:I don't know why Veeam think KB3216755 is pre-release, the MS article https://support.microsoft.com/en-gb/hel ... -kb3216755 doesn't mention pre-release or preview. Can someone clarify?

The info "This release is only available on the Microsoft Update Catalog website" is kind of a hint. Also it is not generally avaiable in microsoft update....
mkretzer
Expert
 
Posts: 275
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby rendest » Mon Feb 06, 2017 11:52 am 2 people like this post

SeandG wrote:I don't know why Veeam think KB3216755 is pre-release, the MS article https://support.microsoft.com/en-gb/hel ... -kb3216755 doesn't mention pre-release or preview. Can someone clarify?

math.random()
rendest
Influencer
 
Posts: 18
Liked: 5 times
Joined: Wed Feb 01, 2017 8:36 pm
Full Name: Stef

Re: Veeam services crash after installing KB3216755

Veeam Logoby Gostev » Mon Feb 06, 2017 5:44 pm

@SeandG click the link in my post for the explanation.
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Veeam services crash after installing KB3216755

Veeam Logoby Gostev » Mon Feb 06, 2017 7:06 pm

OK, so the crash dumps were not immediately helpful because the leak is happening outside of the managed code. For now, it does not look like the issue is in our code - but it is really hard to diagnose since leaking is unmanaged memory. However, the leaked memory content seem to contain lots of SQL Server stuff (column names, stored procedure parameters and such), which prompts to ask all of you the following question - do you use local or remote SQL Server?

I know most of you in this thread are affected by the issue, but I also saw one confirmation on the other thread that someone else did not have any problems after installing the update. So, if we can confirm that all of you affected have local SQL Server installed on your backup server, and those who are not have remote SQL Server - this will be a big step forward in the investigation.

Thanks!
Gostev
Veeam Software
 
Posts: 21239
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], ilya.konopak, nmdange and 26 guests