using gMSA through Veeam Security Domain fails

[DaStivi]

Hello there,
i'm currently testing the usage of group managed service accounts for guest procesing.
so far first tests when dooing ah backup with gMSA with VMs that are in the same Domain as the VBR (proxy) servers is working as expected.

But trying to backup ah server from ah different Active Directoy Domain does not work!

First "issue" you run into how you add the Security Desciptors from the backup-domain to the other domains... but i think i got this sorted out, especially interesting is the Domain Admin group, it should work the way that i've created an gMSA in the productive domain, added there an domain-local group from the productive domain and this domain-local group you can add the computer accounts of the proxys from the backup domain!

in veeam i'll just added then the productive\gmsa account... but using this that way does not work... i even tried to add only "gmsa" as account, but there's a syntax check inside veeam that only allows domain\gmsa or gmsa@domain... ok

the job right now has following error:

Failed to prepare guest for hot backup. Details: Failed to connect to guest agent. Errors:
'Cannot connect to the admin share. Host: [server.prod.local]. Account: [prod\gmsa_veeam].
Failed to perform safe logon
Failed to create a process token for account prod//gmsa_veeam$
Win32 error:The user name or password is incorrect.
Code: 1326
Cannot connect to the admin share. Host: [192.168.1.1]. Account: [prod\gmsa_veeam].
Failed to perform safe logon
Failed to create a process token for account prod//gmsa_veeam$
Win32 error:The user name or password is incorrect.
Code: 1326


the double slashes looks ah little bit strange to me... its also almost the same in the log of the proxy
the error code is slightly different with 1355 and there is an additional line inside the logs:

Win32 error:The specified domain either does not exist or could not be contacted.

(anonymized usernames/domains and ips here)
Case-Nr. 05889607

[HannesK]

Hello,
the syntax check exists because I wasted some hours of troubleshooting during beta testing. The format is required and not "just for the sake of it"

accounts of the proxys from the backup domain!

the Guest interaction proxy must be in the production domain. Or do you have a trust relationship?

both the backup proxy and the target machine should have access to the domain controller to obtain the gMSA password.

I'm currently working with tech writers to replace the "should" with a "must"

Best regards,
Hannes

[DaStivi]

Hi Hannes,
Proxys are in trust relationship, as the BP recommends.

But even there it's somewhat tricky to use gmsa, but as described theoretically it should be possible that way .

I did saw some security domain implementations with Veeam where there is also used users from these restricted domain, honestly not something I would do personally , as my paranoid security me didn't want to even have (User) Accounts from the security domain popping up in network traffic on production domains... But that's ah different discussion anyway...

[HannesK]

Hello,
hmm, with domain trust, it should work (that was not mentioned initially as far as I see).

During gMSA testing I figured out a couple of test steps. Do they all work for you?

1) use the "Test-ADServiceAccount" cmdlet on all involved machines (VM, guest interaction proxy, VBR server)
- The command can be installed with Add-WindowsFeature RSAT-AD-PowerShell
- this command must return "true". otherwise AAIP will fail

Code: Select all

Test-ADServiceAccount svc_gmsa04

2) Check whether computer can select gMSA account in services.msc
- Try to select a user and restart a service
- To switch back, you need the console https://www.wojcieh.net/change-windows- ... l-account/

Code: Select all

sc config VeeamDeploySvc obj=lab\administrator password=PASSWORD!

and restart service and change back to Local System
- On a machine without permissions, you get an error like this one

Windows could not start the Veeam Installer Service service on Local Computer. Error 1069: The service did not start due to a logon failure.

3) check which machines can retrieve passwords
- https://serverfault.com/questions/58297 ... e-accounts

Code: Select all

Get-ADServiceAccount -Identity svc_gmsa04 -Properties PrincipalsAllowedToRetrieveManagedPassword 

Best regards,
Hannes

[DaStivi]

Sorry thought it was clear with trust, else you wouldn't even be able to add proxy from the Management-Backup domain to the domains where the to be backup'd VMs are in!

I gonna test your described steps....

I also thought maybe through the domain trust it's necessary to "install" the gmsa, normally, as far as I understood this, it shouldnt be needed with gmsa .. this was only ah thing with the older normal "MSA"...

[DaStivi]

hmm, with domain trust, it should work (that was not mentioned initially as far as I see).

i guess the question now is what directions of trust is needed... right not its an one-directional trust only... prod domain trusts the mgmt domain (as described in bp: https://bp.veeam.com/vbr/Security/Secur ... e-of-trust)

as described ah couple of times now, but i'll write down now the complete list of groups involved (using the domainnames from the bestpractices example):

Assumption is, we would like to make Backups of the Domain Controllers from production.local!

production.local:

gMSA-Account -> MemberOf Domain Admins (espacally interestting as this is ah GLOBAL group and no objects from trusted domains can be inserted directly!!)
in gMSA_Account though we can add a domain-local-group from the production.local and in this group there you can add all the computeraccounts from veeam vbr/proxys inside (again with an trusted group, shouldn't matter i guess "mngt\VeeamProxyServers") = PrincipalsAllowedToRetrieveManagedPassword

now in theory these computeraccounts should be allowed to retrieve the password of the production\gMSA-Account!

i'll found ah following post that descripes this too:
https://stackoverflow.com/questions/583 ... ed-domains

I have used gMSA accounts across a domain trust. The gMSA principal needs to be a group in the same domain, but as long as the group is type Domain Local, you can add computers from the other domain as members to that group, and they are then able to retrieve the password successfully.

i did some tests now...

During gMSA testing I figured out a couple of test steps. Do they all work for you?

1) use the "Test-ADServiceAccount" cmdlet on all involved machines (VM, guest interaction proxy, VBR server)
- The command can be installed with Add-WindowsFeature RSAT-AD-PowerShell
- this command must return "true". otherwise AAIP will fail

Code: Select all

Test-ADServiceAccount svc_gmsa04

first test didn't succeed, when trying to "test-adserviceAccount production\gmsa-account" from VBR Server in mngt.local domain i'll get an error:
Cannot find an object with identity: 'production\gMSA-Account' under: 'DC=mngt,DC=local'.

2) Check whether computer can select gMSA account in services.msc

in mngt.local side this is simply not possible to use the gMSA Accounts from production.local domain, as there is no trust and hence you can't select the production domain...

3) check which machines can retrieve passwords

Code: Select all

Get-ADServiceAccount -Identity svc_gmsa04 -Properties PrincipalsAllowedToRetrieveManagedPassword 

this one now gets interessting again!
with just "Get-ADServiceAccount -Identity svc_gmsa04 -Properties PrincipalsAllowedToRetrieveManagedPassword" its the same output as with the test-serviceaccount, the command tries to find the account in the mngt.local domain... -> cant find it...

but there is an additional parameter -server, and with that set to the production.local domain the gMSA Account is retrieved correctly!

Code: Select all

Get-ADServiceAccount -Identity gmsa-account -Server production.local -Properties PrincipalsAllowedToRetrieveManagedPassword


DistinguishedName                          : CN=gmsa-Account,CN=Managed Service Accounts,DC=production,DC=local
Enabled                                    : True
Name                                       : gmsa-Account
ObjectClass                                : msDS-GroupManagedServiceAccount
ObjectGUID                                 : d3446df3413-0961-43dd9-806c-4a55ffsa2bc73d
PrincipalsAllowedToRetrieveManagedPassword : {CN=gTrustedServers,OU=TEST,DC=production,DC=local}
SamAccountName                             : gmsa-Account$
SID                                        : S-1-5-21-16634533820-776332457623-456333347-5348
UserPrincipalName                          :

Notice here again, the "gTrustedServers" is ah (domain-local) group within the production, but this group hosts the group with all the computeraccounts from the trusted mngt.local domain...


there is no "-Server" parameter with the test-adserviceaccount available...

[HannesK]

Hello,
QA confirmed what I wrote in my initial answer: the guest interaction proxy has to be in the production domain.

I will ask tech writers to clarify the text in the user guide.

Best regards,
Hannes

[DaStivi]

Ok thx for clarification... I also come after some other tests to this conclusion... We've also tried some different configurations where the gMSA from the management domain should have been used...

I've also found something related to this ... When using gMSA for SQL transaction Log backups ... The SQL server account also need s to be allowed to retrieve password. Didn't read this in the docs, might also just missed it ..

One other thing regarding gMSA, i was ah little sad when I found out that for adding Proxys etc no gMSA can be used... Only normal account can be selected...

[rriley1030]

One other thing regarding gMSA, i was ah little sad when I found out that for adding Proxys etc no gMSA can be used... Only normal account can be selected...

Could you expand on this please? I'm not sure what you mean when you say "adding Proxys etc no gMSA can be used". Are you referring to choosing specific proxy servers on the Guest Processing tab vs. using "Automatic Selection"? Thank you.

BTW, I'm currently working through this on case #05985777.

Thank you, Renee.

[HannesK]

Hello,
my understanding of Stephan's statement is "when you add the proxy machine as a managed Windows server, then username / password is required".

Best regards,
Hannes

[DaStivi]

Hi, yes as Hannes stated, i meant that in theory it should also be possible to use the the gMSA's as "connection" users für the Proxy Component management/installation!

[hias122]

Hi Are there any news regarding this problem, we do have exactly the same issue

[HannesK]

Hello,
and welcome to the forums.

Which one? We discussed a few things in this topic. Adding a Windows server requires username / password. That's the current design.

Best regards,
Hannes

[hias122]

Hi Hannes,
sorry for the late response. We finally managed to figure out our problem. a group policiy blocked the access, which has been deleted now. Everything is working fine now. We had the same error as described in the first message.

[exarch]

I've also found something related to this ... When using gMSA for SQL transaction Log backups ... The SQL server account also need s to be allowed to retrieve password. Didn't read this in the docs, might also just missed it ..

I've been trying, and failing, to get transaction logs to backup using gMSA for weeks now, and this is the first time I've encountered this little bit of rather crucial info in my searches.
I'm not sure how or why this would be required, but it's not a minor detail.

[DaStivi]

the sql transaction log agent "runs" on the sql server and connects to the "veeam repository" hence the sql needs to be allowed to use the gMSA...

[exarch]

Oddly enough, the Log Pruning service doesn't seem to be storing anything on the Log Shipping server (assuming I've set it up correctly), it's just using it as a proxy. Everything is still being stored on the primary backup server/repository.
I suppose this is the way it's supposed to work? Although it makes me wonder what the advantage of Log Shipping servers is, as the logs were previously stored on the main repository without the need for any proxies.

Unfortunately, at this point, I'm no longer sure which issues are the result of recently upgrading from v11 to v12 (and some settings that have been added and aren't properly set), and which are due to the change from a single domain admin user account to gMSA.

[drehstrom]

Stumbled upon your thread having the same problem.
Found out that veeam doesn't like it when the accounts are part of the Protected User group. Any attempt to put the gMSA backing up our DCs into this group ends up with exactly the same error. Seems veeam doesn't like kerberos at all and keeps using NTLM!? Which if true would be pretty bad... Maybe someone can confirm this?

Regards
Stephan

[DaStivi]

Hi, gmsa are not working in protected users group because of Microsoft..... Gmsa's rely on Kerberos constraint Delegation, what would be blocked if in the protected users group... The gmsa also not necessarily needs to be in there... To administer these accounts only tier 0 admins should have access... And in the hosts where it's allowed to retrieve the password only limited admins should have access...

It might also be ah could idea to split them up into multiple accounts for different tasks.

Best regards and nice holidays
Steve

[techepet]

Hi Hannes,
sorry for the late response. We finally managed to figure out our problem. a group policiy blocked the access, which has been deleted now. Everything is working fine now. We had the same error as described in the first message.

In our case we had to ensure we added the gmsa to the allow to log on as a service GPO as we apply CIS Level 1 policies to our environment.