Veeam services crash after installing KB3216755, KB4010672

Availability for the Always-On Enterprise

Re: Veeam services crash after installing KB3216755

Veeam Logoby SeandG » Tue Feb 07, 2017 2:32 pm

@Gostev - Thanks for your message. I am really confused with this one. You are saying that this patch is not for Server 2016, but one of the links you posted says it is:
https://blogs.technet.microsoft.com/filecab/2017/01/30/windows-server-2016-data-deduplication-users-please-install-kb3216755/

Will Gries from MS on the comments says:
I’m working with the team responsible for the Microsoft Update Catalog to get the answer to this question. Note however that it is possible to download and install the “Windows 10” version of the package on Windows Server 2016.

But then you also say that someone else from MS says the update is not meant for server. What a mess. We seem to have one MS team saying something and another something else!

It looks like we have three options available to us:
Go back to Windows 2012 R2 by downgrading our central B&R server with all the risks that entails. Leave the other servers as 2016, working as storage repositories with KB3216755
Install another Veeam B&R server and move all backup jobs to it and let that server manage all the backup jobs. Keep the other servers as 2016 servers working as storage repositories with KB3216755
Since KB3216755 is not meant for server, go back to Windows 2012 R2 by downgrading all our B&R servers with all the risks that entails.

All our copy job storage repositories use Windows deduplication, so we can't run Windows 2016 without KB3216755 due to the corruption issues, but if the fix is not for servers and it is causing issues with Veeam and SQL comms, I see the third option as being the only viable one. Downgrade everything and hope for the best.

Quite ironic that we upgraded to 2016 due to the added benefits around deduplication!
SeandG
Influencer
 
Posts: 13
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Veeam services crash after installing KB3216755

Veeam Logoby dcoenen » Tue Feb 07, 2017 2:45 pm 1 person likes this post

First post from me... :D
What about this information?
https://support.microsoft.com/en-us/hel ... te-history
Helps the KB4010672 for the issue because this is for 2016.
https://support.microsoft.com/en-us/help/4010672
dcoenen
Lurker
 
Posts: 1
Liked: 1 time
Joined: Tue Feb 07, 2017 2:42 pm

Re: Veeam services crash after installing KB3216755

Veeam Logoby ds2 » Tue Feb 07, 2017 8:22 pm 1 person likes this post

KB4010672 is definitely the Server version of KB3216755. We've installed it and it comes to the same issue. The bad thing is that we can't uninstall it, because it always runs into a error by uninstalling! So we are very happy to have a backup of our Veeam-Server. :D

So i can say don't install KB4010672!

Also I hope we don't see this update next tuesday as a regular update for server 2016 at the patchday.
ds2
Enthusiast
 
Posts: 69
Liked: 15 times
Joined: Thu Jul 16, 2015 6:31 am
Full Name: Rene Keller

Re: Veeam services crash after installing KB3216755

Veeam Logoby Gostev » Tue Feb 07, 2017 11:10 pm

Thank you both for the information on Windows Server specific update package KB4010672, I've added it to the topic's name.

Latest update from Microsoft:

This is currently a known and confirmed issue that SQL Server team is working on, and they are discussing a rollback of offending changes. Next Tuesday's update should not have this leak issue, as the affected changes are being removed from this release.

I will update the thread once I hear back more, but the current call to action seems to be to wait for the new build of KB3216755 (and perhaps KB4010672 then too), which presumably should be ready before next week's Patch Tuesday at the latest.
Gostev
VP, Product Management
 
Posts: 20960
Liked: 2239 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Veeam services crash after installing KB3216755, KB40106

Veeam Logoby SeandG » Wed Feb 08, 2017 9:25 am

This is our experience with Windows 2016 and deduplication so far.
We upgraded Windows 2012R2 to 2016 to get improved deduplication functionality of Windows 2016 (how ironic!)
We started getting regular corruption on our backups due to Windows 2016 deduplication issues.

Following the official Server Storage at Microsoft blog and their advice, we installed KB3216755 on all our B&R servers to deal with the deduplication corruption we were experiencing.
We ran into the issues listed on this topic, the server that is managing all the jobs started running out of memory and Veeam services were crashing.
We increased the physical RAM and page file sizes and that didn't help. We went from 64GB to 112GB physical RAM.

Following this thread and some of the questions and answers, we decided to move the local Veeam SQL databases B&R and EM to an external SQL server.
We disabled the local SQL services on the B&R server
Rebooted and for the first time since we installed KB3216755 our backups ran correctly without any RAM issues.

The Veeam.Backup.Manager.exe process that is consuming the most RAM at the moment is only using 125MB as before it would be using Gigs of RAM. We have seen this process using more than 6GB.
Veeam.Backup.Service.exe is using 90MB of RAM, we have seen this process using 80GB.

It is early days, but we are hoping this resolves our issues until MS release patches of patches. Due to data corruption we can't afford to run without KB3216755 even if there is doubt and confusion around it.

The SQL server being used runs on Windows 2012 R2 and is running SQL 2012.

We are keeping our fingers crossed!
SeandG
Influencer
 
Posts: 13
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Veeam services crash after installing KB3216755, KB40106

Veeam Logoby kubimike » Fri Feb 10, 2017 1:51 pm

I to run Windows 2016 w/ Veeam 9.5 and my server crashes every 12-15 hours. I've tried re-installing the OS I don't have the patches installed mentioned in this thread. However, I am using the dedup in windows 2016. Since Im a new Veeam customer how are you guys seeing data corruption? I've done an instant VM recovery and that worked. I've restored single e-mails for exchange and that worked as well. What am I missing ? To read about corruption is alarming since this is the only backup method I have at the moment. My SQL instance is local 2012 Express.
kubimike
Enthusiast
 
Posts: 93
Liked: 14 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: Veeam services crash after installing KB3216755, KB40106

Veeam Logoby SeandG » Fri Feb 10, 2017 2:24 pm

@kubimike

We started getting messages like this on a few of our backup and copy jobs:
Error: All instances of the storage metadata are corrupted

These seem to have stopped since installing KB3216755, we haven't installed KB4010672.

With KB3216755 we started having the service crash due to memory exhaustion reported on this thread. Moving the Veeam Databases seems to have fixed that.

In theory the corruption only happens to very large files - I believe 2TB or larger
SeandG
Influencer
 
Posts: 13
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Veeam services crash after installing KB3216755, KB40106

Veeam Logoby kubimike » Fri Feb 10, 2017 2:35 pm

@SeandG
Thanks, I have not seen that error message yet. I do have files in my repository that are larger then 2TB. This all seems a bit unnerving since I moved us from a stable backup platform that I had previously! Also have a ticket open with Microsoft about the crashes as well ticket #117020615276541

Code: Select all
Source        Description      HotFixID      InstalledBy          InstalledOn
------        -----------      --------      -----------          -----------
Security Update           KB3192137     NT AUTHORITY\SYSTEM  9/12/2016 12:00:00 AM
Security Update           KB3211320     NT AUTHORITY\SYSTEM  2/9/2017 12:00:00 AM
Security Update  KB3213986     NT AUTHORITY\SYSTEM  2/9/2017 12:00:00 AM
kubimike
Enthusiast
 
Posts: 93
Liked: 14 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: Veeam services crash after installing KB3216755, KB40106

Veeam Logoby Mike Resseler » Tue Feb 14, 2017 5:24 pm

Bad news about the hotfix!

See here: https://blogs.technet.microsoft.com/msr ... e-release/

Microsoft has postponed the update cycle
Mike Resseler
Veeam Software
 
Posts: 2357
Liked: 287 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Veeam services crash after installing KB3216755, KB40106

Veeam Logoby mkretzer » Tue Feb 14, 2017 6:12 pm 2 people like this post

LOL... I wonder if the issue they found is our memory leak bug...
mkretzer
Expert
 
Posts: 213
Liked: 49 times
Joined: Thu Dec 17, 2015 7:17 am

[MERGED] Climbing emory usage of "Veeam.Backup.Service"

Veeam Logoby kevin.mcdonnell » Thu Mar 09, 2017 9:39 am 1 person likes this post

Just thought to make you aware that there is an issue with a MS patch that causes memory leaks in Windows 2016 on which I recently installed VBR.

This is the "Known Issues" listed with KB3216755:

https://support.microsoft.com/en-us/hel ... -kb3216755

However, this also occurs with KB4010672 although MS have not detailed it in the notes:

https://support.microsoft.com/en-us/hel ... -kb4010672

I experienced the problem with both Veeam Backup and MS SQL server, thus giving a double-hit on the server in question.

Memory gain was about 500MB/hour for both applications, which can be seen in Resource Monitor but if you'd like a more automated approach this short section of PowerShell will list the usage for you. In this example it will show the value in bytes at 1 minute intervals over an hour - adjust as required.

PS C:\temp\logs> for ($i = 0; $i -lt 60; $i = $i + 1) {
>> Get-Process Veeam.Backup.Service | Select-Object PM
>> Start-Sleep 60
>> }

My advice, as was that of Veeam support who traced the problem, is to remove both MS hot-fixes from your servers.

Case ID: 02090506

Hope this helps.

Kev
kevin.mcdonnell
Novice
 
Posts: 5
Liked: 1 time
Joined: Tue Sep 23, 2014 12:58 pm
Full Name: Kevin McDonnell

Re: [MERGED] Climbing emory usage of "Veeam.Backup.Service"

Veeam Logoby GarrettL » Mon Mar 13, 2017 3:52 pm

Kevin,

This is tremendously helpful and thank you! We have been pulling our hair out for weeks with our 2016 CC server and many 2016 B&R servers and all of which are crashing due to memory bloating. Support hasn't been very helpful with this and hopefully this gets fixed soon. I will go through and remove these updates and deny them moving forward as well.
GarrettL
Service Provider
 
Posts: 20
Liked: 3 times
Joined: Sun Jun 19, 2016 6:28 am
Full Name: Garrett Lynch

Re: [MERGED] Climbing emory usage of "Veeam.Backup.Service"

Veeam Logoby GarrettL » Mon Mar 13, 2017 7:02 pm

May I also advise anyone who's running a 2016 Server in Azure (as we are), removing KB4010672 completely destroyed our CC server... :cry:
GarrettL
Service Provider
 
Posts: 20
Liked: 3 times
Joined: Sun Jun 19, 2016 6:28 am
Full Name: Garrett Lynch

Re: Veeam services crash after installing KB3216755, KB40106

Veeam Logoby Mike Resseler » Tue Mar 14, 2017 6:38 am 1 person likes this post

Garrett,

What do you mean by destroying your CC server? As in, completely unresponsive? Do you have a support call with MSFT for this?
Mike Resseler
Veeam Software
 
Posts: 2357
Liked: 287 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Veeam services crash after installing KB3216755, KB40106

Veeam Logoby GarrettL » Tue Mar 14, 2017 11:47 am

Mike Resseler wrote:Garrett,

What do you mean by destroying your CC server? As in, completely unresponsive? Do you have a support call with MSFT for this?


Hi Mike,

For our Azure instance, it caused a BSOD and failure to boot and I was unable to rollback as well so I had to revert back to a disk snapshot to restore. I took a look at the new deployment image and it is 1/2 updates that are included with the image so I guess I'm stuck with it for the time being. I may call it in to see if there's a fix pending that I can implement.
GarrettL
Service Provider
 
Posts: 20
Liked: 3 times
Joined: Sun Jun 19, 2016 6:28 am
Full Name: Garrett Lynch

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot], Majestic-12 [Bot], MSNbot Media, Yahoo [Bot] and 10 guests