v.11 Lacks Backwards Compatibility Related to VIX Guest Processing

[vmikhelson]

Hi,

I have run into a peculiar situation when I upgraded one of my customers to v.11 with May 2021 patch. The customer runs Windows workgroup with no Windows Domain in place. All computers have an "admin" account which is a member of the Local "Administrators" group. This account is used for Veeam guest processing.

After the v.11 was installed all Windows Guest Processing started failing. Analysis showed v.10 was capable to failover to VIX whereas v.11 fails to do that even though the Guest Credentials Test showed the VIX based processing succeeded. Support case #04852543 did not provide any feasible resolution. I ended up rolling back to v.10.

I hope these notes will help somebody.

-Vladimir

[foggy]

Hi Vladimir, thanks for sharing. I'm checking what changes in v11 could result in the observed behavior. According to the case notes, v11 does fail over to VIX but fails due to the fact that a not built-in account is being utilized. Disabled UAC or a built-in account has always been a requirement for processing over VIX so v10 should've behaved similarly unless there's something else that we're missing.

[vmikhelson]

Alexander,

I definitely do not use the built-in Administrator as it would be insecure to have it not disabled in the first place.

No changes were made to the administrative account used or to the UAC on the Windows side, the only change was upgrading to v.11 where VIX stopped working, and then downgrading to v.10 where everything works again as expected. I call it "lack of backwards compatibility." UAC was and should stay enabled as a critical security measure in Windows.

I hope you will find what you are missing soon.

Thank you,
Vladimir

[Gostev]

"Lack of backward compatibility" requires that compatibility existed originally however, this was never supported and we had the following bullet in the Release Notes document for more than a decade now:

Networkless interaction with Microsoft Windows guests having UAC enabled (Vista or later) requires
that Local Administrator (MACHINE\Administrator) or Domain Administrator (DOMAIN\Administrator)
account is provided on Guest Processing step

This does not make this situation any less interesting though, as if this works indeed with V10 as you say it does, then you may have found a critical security issue in Windows that allows bypassing UAC protection somehow! Assuming of course the machine in question is a vanilla Windows install without a custom local security policy.

If you can attach the logs from V10 to the case, then we can confirm if VIX is indeed used successfully w/out a built-in Administrator account, then go from there.

[vmikhelson]

Mr. Gostev,

Please see my replies in-line.

Thank you,
Vladimir

-->> "Lack of backward compatibility" requires that compatibility existed originally

In my books, if it works in one version and stops working in the next one with no other changes in the settings or in the environment, backwards compatibility test has failed.

-->> however, this was never supported and we had the following bullet in the Release Notes document for more than a decade now:
Networkless interaction with Microsoft Windows guests having UAC enabled (Vista or later) requires
that Local Administrator (MACHINE\Administrator) or Domain Administrator (DOMAIN\Administrator)
account is provided on Guest Processing step

My only comment, VIX was happy in v.10 and BTW in v.11 in the Guest Processing Test. Please see the case notes.

-->> This does not make this situation any less interesting though, as if this works indeed with V10 as you say it does, then you may have found a critical security issue in Windows that allows bypassing UAC protection somehow! Assuming of course the machine in question is a vanilla Windows install without a custom local security policy.

The subject machines are Windows 10 20H2 or 21H1 and Windows Server 2019. Please let me know which specific local security policies you would like me to verify as I could have tweaked something for reasons not related to Veeam Backup.

-->> If you can attach the logs from V10 to the case, then we can confirm if VIX is indeed used successfully w/out a built-in Administrator account, then go from there.

Can you please reopen the case? As soon as it is reopened I will attach the logs.

[Gostev]

backwards compatibility test has failed

The test did not fail. Rather, there was no test to start with, because unsupported configurations are never tested in principle. The testing scope is strictly defined by system requirements and documented limitations. In other words, we don't test things we don't support

Can you please reopen the case? As soon as it is reopened I will attach the logs.

You can just open a new support case, and refer the existing one in it. Just let us know the support case ID once the logs are uploaded, and you won't need to worry about this. Unless they will want some specific logs from the VM in question directly, then the support engineer will follow up with you.

[vmikhelson]

See case #04870931 for the v.10 logs.

[foggy]

Thanks for opening a new case to help us investigate. So far our engineers are not able to reproduce the different behavior for v10 and v11 internally so they will request some additional details from you.

[nn@zitcom.dk]

I'm seeing similar issues with AAIP on v11 using VIX. We are having a customer which are using VIX only and have always used a local account for Veeam (non-Builtin) with UAC enabled. After upgrading from v10 to v11 this stopped working for some ever reason. This is indeed not a scaleable solution for SPs that do not have ip connectivity with the OS and must use VIX. This is ofc. by design and not a veeam issue though.

Veeam support also told that this was never supported also.

Please provide your findings here.

[Gostev]

At this point our QC has confirmed that with clean Windows installs, VIX processing does not work with non-Builtin administrator accounts either in V10 or V11.

This means that just as I predicted, V10 installs where it did work likely had some baseline Windows security settings lowered to allow for this. Next, we will need to work with some of you guys to determine what are these settings, so that we could build the same lab internally and compare V10 and V11 in that lab.

[foggy]

Hi @nn@zitcom.dk, could you please also share your case ID - we're interested in taking a look at some UAC parameters on the affected systems, if possible. Thank you!

[couchman75]

Hi
We're experiencing a very similar issue on a Workgroup Windows 2012R2 Server after upgrading from v9.5 to v11 back in May, however the built-in administrator account doesn't work either. Everything was fine on 9.5 with a regular local admin account. The issue is currently with support (case ID 04808648) who requested we build a fresh server to test with, on this server VIX succeeds during the test but fails when actually running the backup. There is something in the Windows Server hardening templates we're running that breaks VIX with v11 as it worked OK before applying these.
Russ...

[foggy]

Hi Russ, thanks for sharing, although your particular case looks a bit differently taking into account the fact that the actual built-in administrator account doesn't work for you. Let's see how the support investigation goes.

[couchman75]

Hi Foggy, the conclusion at the moment is that we need to disable UAC to make it work regardless of what account we use and the linked article is outdated pre v11

[For networkless guest processing over VMware VIX/vSphere Web Services] Check that UAC is disabled on VM guest OS and the specified account has local Administrator permissions in addition to the permissions listed in the table. For more information on how to disable UAC, see this Veeam KB article.

This is from a fresh document for v11 https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[Gostev]

This is indeed an alternative to using the built-in Administrator account, but the latter alone should still work fine and is the recommended approach that allows to keep UAC enabled. I'll ask the documentation corrected, not sure why and when it was changed.

[nn@zitcom.dk]

Hi @nn@zitcom.dk, could you please also share your case ID - we're interested in taking a look at some UAC parameters on the affected systems, if possible. Thank you!

#04797070

[couchman75]

These are the settings that break VIX for our environment, if I set either of these to disabled VIX will work for the built-in administrator.

Security Settings/Local Policies/Security Options: "Admin Approval Mode for the Built-in Administrator account: Enabled"
MS Security Guide GPO Template setting: "Apply UAC restrictions to local accounts on network logons: Enabled" (sets the LocalAccountTokenFilterPolicy registry key).

[weeam]

Hello,

Same conclusion for me (like vmikhelson, thank you for your time). Since we have upgraded to v11. Guest processing stop working for some VMs.

Case #04936905

Regards,
Eric

[weeam]

We were able to fix this issue on most VMs with this registry entry (with a local administrator account and UAC enable):

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System]
"LocalAccountTokenFilterPolicy"=dword:00000001

Ref: https://docs.microsoft.com/en-us/troubl ... estriction

Thank you couchman75 for sharing your knowledge.

[couchman75]

I'm surprised we have to weaken the hardening of our servers to make V11 work the same as 9.5.

Gostev, has there been a fundamental change within v11 security to require this security weakening?

[Gostev]

There were no changes in VIX processing engine in V11. System requirements are still the same as they were since we added support for application-aware processing over VIX: you must specify Domain Administrator or built-in Administrator account. Only using any other accounts requires "security weakening".

If you have issues with VIX when using Domain Administrator or built-in Administrator account, open a support case as we're not seeing any in our labs.

[couchman75]

As above, case ID 04808648

If both of these settings below are enabled then VIX doesn't work on V11 for ANY account. Fresh Windows Server 2012R2 Server with CIS templates applied.

Security Settings/Local Policies/Security Options: "Admin Approval Mode for the Built-in Administrator account: Enabled"
MS Security Guide GPO Template setting: "Apply UAC restrictions to local accounts on network logons: Enabled" (sets the LocalAccountTokenFilterPolicy registry key).

Thanks
Russ..

[Gostev]

I am not prepared to comment on how these non-default settings impact VIX processing. Our QC is validating our products against clean Windows installs only, as testing billions of possible non-default settings combinations is simply impossible.

[soncscy]

I got curious and just looked it up, and it makes absolute sense why these Policies break VIX:

https://docs.microsoft.com/en-us/window ... or-account

Basically, if this is enabled, the admin account no longer gets the Administrator behavior for privileged requests and must have a user interaction for UAC like any other user -- obviously VIX (though really, it is Web Services I suppose) cannot do this! The setting is meant for a human to be aware of things, not for an automated service as the service cannot respond to the prompt.

[Zew]

There's one thing I'm super confused about, and I'm pretty sure the Veeam Devs and QC are as well. As mentioned many times by Gustev "There's been no changes to the engine for that feature".

If someone already had a hardened Windows system, and did not disable UAC, the only option is the built-in administrator account.

Since the mentioned settings have been around since apparently 2017, it would seem that something has changed on the Veeam side?

[Gostev]

Maybe, just not in the engine itself.

[foggy]

The reason for the behavior change in v11 is an additional check that verifies account membership in Administrators group and also checks a couple of flags that are enabled for built-in account only. We've removed verification of these flags in v11a to bring back the v10 behavior.

[couchman75]

Thanks Foggy, that makes a bit more sense. When is v11a due?

[foggy]

Cannot share any particular ETA at the moment but it's not too far away.

[weeam]

version 11a next month