Veeam services crash after installing KB3216755, KB4010672

[SeandG]

@Gostev - Thanks for your message. I am really confused with this one. You are saying that this patch is not for Server 2016, but one of the links you posted says it is:
https://blogs.technet.microsoft.com/fil ... kb3216755/

Will Gries from MS on the comments says:
I’m working with the team responsible for the Microsoft Update Catalog to get the answer to this question. Note however that it is possible to download and install the “Windows 10” version of the package on Windows Server 2016.

But then you also say that someone else from MS says the update is not meant for server. What a mess. We seem to have one MS team saying something and another something else!

It looks like we have three options available to us:
Go back to Windows 2012 R2 by downgrading our central B&R server with all the risks that entails. Leave the other servers as 2016, working as storage repositories with KB3216755
Install another Veeam B&R server and move all backup jobs to it and let that server manage all the backup jobs. Keep the other servers as 2016 servers working as storage repositories with KB3216755
Since KB3216755 is not meant for server, go back to Windows 2012 R2 by downgrading all our B&R servers with all the risks that entails.

All our copy job storage repositories use Windows deduplication, so we can't run Windows 2016 without KB3216755 due to the corruption issues, but if the fix is not for servers and it is causing issues with Veeam and SQL comms, I see the third option as being the only viable one. Downgrade everything and hope for the best.

Quite ironic that we upgraded to 2016 due to the added benefits around deduplication!

[dcoenen]

First post from me...
What about this information?
https://support.microsoft.com/en-us/hel ... te-history
Helps the KB4010672 for the issue because this is for 2016.
https://support.microsoft.com/en-us/help/4010672

[ds2]

KB4010672 is definitely the Server version of KB3216755. We've installed it and it comes to the same issue. The bad thing is that we can't uninstall it, because it always runs into a error by uninstalling! So we are very happy to have a backup of our Veeam-Server.

So i can say don't install KB4010672!

Also I hope we don't see this update next tuesday as a regular update for server 2016 at the patchday.

[Gostev]

Thank you both for the information on Windows Server specific update package KB4010672, I've added it to the topic's name.

Latest update from Microsoft:

This is currently a known and confirmed issue that SQL Server team is working on, and they are discussing a rollback of offending changes. Next Tuesday's update should not have this leak issue, as the affected changes are being removed from this release.

I will update the thread once I hear back more, but the current call to action seems to be to wait for the new build of KB3216755 (and perhaps KB4010672 then too), which presumably should be ready before next week's Patch Tuesday at the latest.

[SeandG]

This is our experience with Windows 2016 and deduplication so far.
We upgraded Windows 2012R2 to 2016 to get improved deduplication functionality of Windows 2016 (how ironic!)
We started getting regular corruption on our backups due to Windows 2016 deduplication issues.

Following the official Server Storage at Microsoft blog and their advice, we installed KB3216755 on all our B&R servers to deal with the deduplication corruption we were experiencing.
We ran into the issues listed on this topic, the server that is managing all the jobs started running out of memory and Veeam services were crashing.
We increased the physical RAM and page file sizes and that didn't help. We went from 64GB to 112GB physical RAM.

Following this thread and some of the questions and answers, we decided to move the local Veeam SQL databases B&R and EM to an external SQL server.
We disabled the local SQL services on the B&R server
Rebooted and for the first time since we installed KB3216755 our backups ran correctly without any RAM issues.

The Veeam.Backup.Manager.exe process that is consuming the most RAM at the moment is only using 125MB as before it would be using Gigs of RAM. We have seen this process using more than 6GB.
Veeam.Backup.Service.exe is using 90MB of RAM, we have seen this process using 80GB.

It is early days, but we are hoping this resolves our issues until MS release patches of patches. Due to data corruption we can't afford to run without KB3216755 even if there is doubt and confusion around it.

The SQL server being used runs on Windows 2012 R2 and is running SQL 2012.

We are keeping our fingers crossed!

[kubimike]

I to run Windows 2016 w/ Veeam 9.5 and my server crashes every 12-15 hours. I've tried re-installing the OS I don't have the patches installed mentioned in this thread. However, I am using the dedup in windows 2016. Since Im a new Veeam customer how are you guys seeing data corruption? I've done an instant VM recovery and that worked. I've restored single e-mails for exchange and that worked as well. What am I missing ? To read about corruption is alarming since this is the only backup method I have at the moment. My SQL instance is local 2012 Express.

[SeandG]

@kubimike

We started getting messages like this on a few of our backup and copy jobs:
Error: All instances of the storage metadata are corrupted

These seem to have stopped since installing KB3216755, we haven't installed KB4010672.

With KB3216755 we started having the service crash due to memory exhaustion reported on this thread. Moving the Veeam Databases seems to have fixed that.

In theory the corruption only happens to very large files - I believe 2TB or larger

[kubimike]

@SeandG
Thanks, I have not seen that error message yet. I do have files in my repository that are larger then 2TB. This all seems a bit unnerving since I moved us from a stable backup platform that I had previously! Also have a ticket open with Microsoft about the crashes as well ticket #117020615276541

Code: Select all

Source        Description      HotFixID      InstalledBy          InstalledOn
------        -----------      --------      -----------          -----------
Security Update           KB3192137     NT AUTHORITY\SYSTEM  9/12/2016 12:00:00 AM
Security Update           KB3211320     NT AUTHORITY\SYSTEM  2/9/2017 12:00:00 AM
Security Update  KB3213986     NT AUTHORITY\SYSTEM  2/9/2017 12:00:00 AM

[Mike Resseler]

Bad news about the hotfix!

See here: https://blogs.technet.microsoft.com/msr ... e-release/

Microsoft has postponed the update cycle

[mkretzer]

LOL... I wonder if the issue they found is our memory leak bug...

[kevin.mcdonnell]

Just thought to make you aware that there is an issue with a MS patch that causes memory leaks in Windows 2016 on which I recently installed VBR.

This is the "Known Issues" listed with KB3216755:

https://support.microsoft.com/en-us/hel ... -kb3216755

However, this also occurs with KB4010672 although MS have not detailed it in the notes:

https://support.microsoft.com/en-us/hel ... -kb4010672

I experienced the problem with both Veeam Backup and MS SQL server, thus giving a double-hit on the server in question.

Memory gain was about 500MB/hour for both applications, which can be seen in Resource Monitor but if you'd like a more automated approach this short section of PowerShell will list the usage for you. In this example it will show the value in bytes at 1 minute intervals over an hour - adjust as required.

PS C:\temp\logs> for ($i = 0; $i -lt 60; $i = $i + 1) {
>> Get-Process Veeam.Backup.Service | Select-Object PM
>> Start-Sleep 60
>> }

My advice, as was that of Veeam support who traced the problem, is to remove both MS hot-fixes from your servers.

Case ID: 02090506

Hope this helps.

Kev

[GarrettL]

Kevin,

This is tremendously helpful and thank you! We have been pulling our hair out for weeks with our 2016 CC server and many 2016 B&R servers and all of which are crashing due to memory bloating. Support hasn't been very helpful with this and hopefully this gets fixed soon. I will go through and remove these updates and deny them moving forward as well.

[GarrettL]

May I also advise anyone who's running a 2016 Server in Azure (as we are), removing KB4010672 completely destroyed our CC server...

[Mike Resseler]

Garrett,

What do you mean by destroying your CC server? As in, completely unresponsive? Do you have a support call with MSFT for this?

[GarrettL]

Garrett,

What do you mean by destroying your CC server? As in, completely unresponsive? Do you have a support call with MSFT for this?

Hi Mike,

For our Azure instance, it caused a BSOD and failure to boot and I was unable to rollback as well so I had to revert back to a disk snapshot to restore. I took a look at the new deployment image and it is 1/2 updates that are included with the image so I guess I'm stuck with it for the time being. I may call it in to see if there's a fix pending that I can implement.

[Mike Resseler]

Garrett,

Can't force you but I think it would be helpful to call it in. The more people that report it, hopefully the quicker there is a resolution

My 2 cents

[GarrettL]

Mike,

I'm working with Microsoft engineering on the issue and they're hoping to get an official response from Veeam regarding the leak, however, due to the recent cumulative updates released today, I opted to first get them installed on some of our 2016 servers and see how they fare.

KB4013429
KB4013418

I would be curious to hear how others are doing with these updates as well.

[Gostev]

Original blog post updated with the new KB > https://blogs.technet.microsoft.com/fil ... kb3216755/

[GarrettL]

All,

I'm very pleased to say that so far, early indications after installing the two new KBs have appeared to have resolved the memory leak. I'm hovering at 8GB /32 GB which by now would already have hit 20+ and chugging along quite nicely. I'll update should things go south.

[Mike Resseler]

Thanks Garrett! Appreciate coming back with the feedback. @All, let us know your results with the new patch also!

[GarrettL]

Mike,

Just to follow up, memory usage remains normal and everything looks great. This is the longest we've ever been able to run our Veeam server on 2016 to date (2 days) and we're still at 9.5/32 GB and a whole slew of concurrent jobs going.

[mih]

Seems to work fine here, no abnormal memory use anymore. Hopefully we will soon get a fix to the "Bad checksum - Recall bitmap" thing also.