Feature Request: selectable mount server for Windows FLR

[mfbu]

Case # 07559885 - Case #07512898

TL;DR

this is a feature request for

a) selectable mount server for Windows FLR
a1) SECURITY: don't ever use VBR server as mount server

Long version

a) mount servers are configured per Repository server (I am convinced this
grouping doesn't make much sense). So in a scaleout repo, we might have a
different mount server configured per scaleout extent. You might run into a
situation where different restore points of the same VM get to reside on
different repo servers (scaleout extents) in the same scaleout repo, and a
restore operation would then use a different one depending on which restore
point from which scaleout extent is to be used.

Worse, if you have an imported restore point (.vbk) that lies on one of your
repo servers, that per-repository setting for mount server is being ignored.
You have no way to manually override the mount server. Instead, the .vbk then
gets mounted to the VBR server itself (compare with Linux FLR: you HAVE TO
manually select a mount server a.k.a helper host a.k.a helper appliance EVERY
TIME). Currently, we have NTFS dedup software components installed only on the
mount server, not on the VBR server. Therefore, we currently cannot access
imported restore points using NTFS dedup unless we also install NTFS dedup
software components on the VBR server itself. I would prefer not to.

According to Veeam Tech Support, this is because an imported backup does not
get assigned a value in the "Repository" field (and you can't supply that field
afterwards). They don't think this to be a bug, they claim it is "expected
behaviour".
I beg to differ

-----------------------------------------------------------------------
a1) on the VBR server, you deposit the most precious credentials of all your
network: vSphere admin credentials, storage admin credentials, Windows
credentials to get "Local Administrator" rights for FLR, Linux credentials to
get "root" for FLR. I probably have missed some categories. These credentials
get to be used by the VBR server automagically, even across reboots, without
anyone providing a passphrase or something similar for this collection of
credentials. So while they might be stored on the VBR server's disk in some
obfuscated way, they are not effectively encrypted. For Windows FLR, under
circumstances described above (and possibly in other cases, too), the VBR
server itself gets to do the filesystem mounting for the selected restore
point. From a security standpoint, the virtual disk images of a restore point
must be viewed as attacker-controlled material. Now mounting a filesystem
(no matter the noexec or readonly options) is a kernel-level activity (I assume
Veeam did not re-implement this in user space). Any (Windows-based)
vulnerability there might then yield kernel-level access to the attacker,
allowing him to grab all those (obfuscated, not effectively encrypted)
credentials and to use them for fun and profit. Now you might be inclined to
do some finger-pointing towards some bad security of the original VM that got
hijacked in the first place or towards Microsoft for some stupid kernel-level
bug. But essentially this is Veeam ignoring Defense in Depth design principles.

Veeam Tech Support's assessment instead claims that everything is fine
security-wise the way it currently is.
I beg to differ

[HannesK]

Hello,

a) How did you import the backups? With the import button resulting in not having a repository assigned or by rescanning the Scale-Out Backup Repository (SOBR)?



the mount host is selected "as configured" as long as the import knows the repository (marked in green on the screenshot). That only works with rescan

a1) I'm not sure I can follow. If somebody can steal the stored credentials (aka copy the database file), then they are useless on a different machine as they are encrypted with the machine key. Stealing the machine key sounds a bit far-fetched to me as Microsoft invested into that topic with Credential Guard.

Best regards
Hannes

[mfbu]

Thanks for your reply, Hannes.

a) I used the import button. Since I don't have a .vbm file, support told me a rescan would not be able to find anything.

a1) Veeam VBR accesses them at runtime without additional user input, so they are accessible even from user space without user intervention; I fail to see how a kernel-level exploit should then not be able to access them (compile a clear-text version and exfiltrate that). While I can't come up with a working exploit, the mere idea of NOT using a dedicated, separate mount server sounds scary. OTOH, I am not a windows guy and I might miss some of the magic hidden in MS documentation about Credential Guard and VBS (rant: with MS, I often think this hiding is done on purpose). Does Veeam at least check that Credential Guard is used on the VBR server before mounting an image? Also I don't see Credential Guard mentioned anywhere in the Veeam User Guide (vSphere). Do you have a pointer to Veeam documentation stating that Credential Guard actually gets used?

Regards
Matthias

[HannesK]

Hello,

a) to get correct mount server selection working, the VBM file is needed. Otherwise the backup files are not recognized as part of a repository (the column would be empty then in the console). Whether the backup files are "just by chance" located in a folder that is also a Veeam repository is unknown to the software without VBM file.

a1) Yes, we have a filter driver that mount the backups transparently to show the backups. You can use a dedicated mount server as long as the backups are imported with VBM file. Credential guard is transparent for applications. Veeam (like any other software that would use dpapi) uses what Windows offers. There is nothing to be documented on our side because it's just "Data Protection API (DPAPI) mechanisms", which is documented in the link before.

Best regards
Hannes

[mfbu]

Hi and thanks again Hannes,

a) yes, that is just what support told me, and therefore this feature request: to be able to use a mount server (instead of the VBR server). The current release does not allow it, but It is still unclear to me why it should not be possible to implement that.

a1) thanks for the dpapi link; it describes dpapi from the XP era (2010), so some contents might be outdated. I only found one current post containing dpapi and Credential Guard: https://learn.microsoft.com/en-us/windo ... w-it-works (2024), but it does not dive very deep. Quoting the old dpapi link:

Applications either pass plaintext data to DPAPI and receive an opaque
protected data BLOB back, or pass the protected data BLOB to DPAPI and
receive the plaintext data back. Figure 1, below, shows these two
operations.

Credential Guard may protect against offsite decoding of encrypted credential BLOBs, since the secrets required for decoding do not reside on disk anymore but in other parts of the virtual hardware you can't access from inside the VM: vTPM / UEFI machine key / ... .
I don't see how Credential Guard keeps an intruder on the same host from using the same API with the same BLOBs to decode the secrets to cleartext.

From what I could gather from some further links, you can't simply assume Credential Guard to be active. Disclaimer: I am not a windows guy.
It seems that if your server has some history, Credential Guard might not have been activated on upgrades. It requires VBS (virtualization based security), which does not work if e.g. Secure Boot is not enabled. I found a Powershell snippet in https://learn.microsoft.com/en-us/windo ... abs=intune to check if Credential Guard is active:

(Get-CimInstance -ClassName Win32_DeviceGuard -Namespace root\Microsoft\Windows\DeviceGuard).SecurityServicesRunning

Company policy does not allow me to post the result.

Regards
Matthias