Windows 2016 registry bloat - server will not boot

[ned]

I have a Dell 730xd server running Windows 2016 standard with 13TB of local storage that functions as one of my Veeam 9.5.2 backup servers. The server will not boot up. I get into repair/recovery mode. I opened a Microsoft support case (Premier Support - Sev A), provided memory dumps and the config folder content, and after 3 days they've informed me it's because the system hive is 1.5GB and they cannot figure out how to compress or reduce the size. Has anyone seen this with Windows 2016? I understand that it was a problem with Windows 2012 and earlier, but support states they have not seen it with 2016. I have Veeam Windows 2.0 agent backup, but was hoping to resolve the issue. (Veeam SR 02317963) (Microsoft SR 117091816357204) Thanks Ned

[Dima P.]

Hello and welcome to the community Ned,

Please open a support case via Control Panel > Support > Technical Support and share the case ID in this thread. Thanks.

[ned]

The Veeam SR id is in the original post.

[ned]

A Microsoft tool devnodeclean resolved this issue. There were over 700K entries of MPIO paths in the registry. Devnodeclean removed the bloat. I had Veeam storage integration enabled.

https://support.microsoft.com/en-us/hel ... er-be-used
https://www.microsoft.com/en-us/downloa ... x?id=42286

Windows 2016 server will not boot due to 1.5GB registry/system hive - Case # 02317963

[Dima P.]

Hello Ned,

Thank you for providing details regarding this issue. I’ll check your case details on Monday when I have access to the ticketing system. Any idea why this MPIO paths have flooded your registry?

[ned]

I assume it was Veeam storage integration (Netapp) related.

New KB for this issue: https://www.veeam.com/kb2365

[Dima P.]

Hi Ned,

Thanks for sharing the KB article with the community, it's highly appreciated!

[ned]

Here is an update from our Microsoft TAM on the issue. I received this today.


Subject: Follow Up on 117091816357204- Windows Server Case involving Veeam

I just wanted to provide the results of an internal case that was opened between Veeam and Microsoft to discover the source of the server errors from your case.

It was determined that this was not a bug and that Veeam was responsible for buildup of orphaned LUNs and it is their responsibility to clean it.

Per the engineer:

“I would just let them know that our Storage and PnP teams reviewed this and concluded that it was the responsibility of the component that was connecting the storage to Windows to clean it up. This is just proper resource management, especially when the component connecting the storage devices results in tens of thousands of connected devices.”

Below is the KB Article and paragraph for citation purposes:

https://support.microsoft.com/en-us/hel ... er-be-used

It is the responsibility of the software that establishes the connection between the storage device and Windows to properly clean up the information for the device. This process is necessary because Windows does not know when a storage device is removed temporarily or permanently. But the software that establishes the connection typically does know this. For example, if backup software is mounting logical unit numbers (LUNs) for backup purposes and then unmounting the LUNs, it would be the responsibility of the backup software to clean up the LUN information from Windows, because the same storage device will no longer be used again by Windows.

I am not sure if this helps or provides closure in any way but I wanted to make you aware of the results of the internal case to get to the bottom of this.

[ned]

Note that this is still an issue with 9.5.4. I run the devnodeclean.exe in scheduler, but is it possible to get cleanup functionality added to a future Veeam B&R release? Thanks

[Gostev]

Actually, this should not be an issue with 9.5 Update 4, since it has registry cleaner built in... see the What's New document, under Storage Integration for more info. Thanks!