Feature request: getting rid of admin credentials for application-aware processing?

[cosmik]

This was also asked at the AMA session during the Veeam 100 summit: I’d like to have application aware processing on Windows boxes by pre-installing some special Veeam software, but without having to specify an administrator-level account during job configuration.

@HannesKback then replied that on V11 AAP did that trick just fine, but further testing showed that it was not possible. At least not without providing an account with administrative access corresponding to the VM.

@Mildur at the time provided me with an alternative: instead of backing up via the ESXi API, install a standalone agent on the VM boxes, that would allow backing up without having to configure an admin account to the box. Have not tested it yet, should work just fine though!

In any case, it’s day one of the VMCE course (thanks Veeam ) and during discussion of the AAP-related options AlexH was pondering about exactly the same issue.

In any case, I’d like to re-iterate my request to Veeam for providing a way to have account-less AAP, by pre-installing some sort of Veeam agent beforehand.

[HannesK]

Hello,

@HannesKback then replied that on V11 AAP did that trick just fine, but further testing showed that it was not possible. At least not without providing an account with administrative access corresponding to the VM.

it wasn’t me who replied that But I did not want to start a discussion, whether the persistent guest agent could be a solution or not, because I was not 100% sure.

I postponed testing it a couple of times… but your post made me test it now and I can confirm what you say and I believed in Prague: a local admin account is required for authentication.

A local admin account requirement makes sense, because otherwise any “random user” with a VBR server could just take over the machine.

I agree, that it’s a valid request. Just thinking out loud: would you be comfortable to deploy certificates manually on the machines instead of providing username / password centrally?

Best regards,
Hannes

[cosmik]

it wasn’t me who replied that But I did not want to start a discussion, whether the persistent guest agent could be a solution or not, because I was not 100% sure.

My bad, blame my poor memory.

A local admin account requirement makes sense, because otherwise any “random user” with a VBR server could just take over the machine.

I agree, that it’s a valid request. Just thinking out loud: would you be comfortable to deploy certificates manually on the machines instead of providing username / password centrally?

Not a dev here. I was thinking that this could be done in a manner similar to the Veeam agent. Or the way endpoint security products implement their communication agent (the component that provides the glue between the endpoint software and the server), ie using certificates to authenticate the actual server.

[HannesK]

The Veeam agent also requires local admin for authentication if it gets deployed / managed centrally.

The alternative could be "single use credentials" that are used place the certificate on the machine. But then it would fail, as soon as a new VM is added to a job.

[cosmik]

The Veeam agent also requires local admin for authentication if it gets deployed / managed centrally.

True, but I guess one could configure it manually from within the system-to-be-protected? And an account is not needed afterwards at all.

The alternative could be "single use credentials" that are used place the certificate on the machine. But then it would fail, as soon as a new VM is added to a job.

Thankfully, I'm not the one that has to answer the how can we get that done part

[Regnor]

When using the persistent guest agent the first time, I also had the same idea of using it without user credentials; acctually initially I thought it would work that way.
Like you wrote Hannes, Veeam could connect to it via a certificate or machine-key, which is generated from VBR side.
The persistent guest agent is manually deployed, so you could also initially register it with VBR, use a VBR-generated setup or add the certificate/machine key with a parameter.

From the security aspect, what damage could be done with the persistent guest agent if it's somehow taken over?
As long as it only offers backup capabilities like create VSS snapshots, prepare AD/SQL, etc, and no write/execute actions, then it should be safe in my opinion.
(assuming that there is no security vulnerability)

[HannesK]

The persistent guest agent is manually deployed, so you could also initially register it with VBR

yep, same as the pre-installed agents, with the XML file, that includes a certificate for initial connection.

From the security aspect, what damage could be done with the persistent guest agent if it's somehow taken over?

everything... it allows to run "pre-scripts" and "post-scripts" as administrator.

As far as I see, we would have to implement a "block scripts" option first.

It's the same with all types of software, that allow to run scripts (monitoring agents, backup agents, probably even antivirus agents).

Security vulnerabilities are a second topic, yes... I remember many times, that antivirus etc. agents allowed to get administrator access... that's not so great for a software, that should protect

[Regnor]

Yeah the pre/post-scripts would just open every possibility. I didn't think about everything
So either there would be a very secure way of authentication between VBR and the persistent guest agent needed, it would have to offer a limited feature set or only run Scripts available on the VM itself.
When using a piece of software, you should by default expect such vulnerabilities. And the more permissions a software uses, the more could happen.

[Cragdoo]

would something like the secure SSL connection used in VCC be an idea, open a custom port between the guest VM and the VBR server for processing ...actually just re-read the comments about certificates ...ignore me

[ITP-Stan]

Even if you block scripts, you would still need to make sure only an 'authenticated' VBR server is allowed to back-up.
Otherwise someone could use a rogue VBR server to back-up the machine, and then restore files to another location and read those files (takeown) which they did not have access too.

[HannesK]

you would still need to make sure only an 'authenticated' VBR server is allowed to back-up.

yes, that's why I mentioned deploying certificates

I mean, for VM backup it's not really a big thing (because one has to have hypervisor credentials), but for agents it would be the same challenge.

[jfreeman]

We are a VBR subscriber, and we would like to submit the following feature request for security purposes: no longer require credentials to perform backups and other Veeam-agent duties.

Ideally, this is what should occur during the installation of the Veeam agent from the VBR server:

1. You decide to add a new backed-up virtual machine to your Veeam inventory.
2. In the VBR console, you are prompted for information about the VM. Among the requested information would be administrative credentials to the newly backed-up VM.
3. You type your administrative credentials (e.g., could be a domain administrator or just a local administrator account).
4. VBR automatically logs into the backed-up virtual machine to install the Veeam agent, etc.
5. One of the things VBR would do is setup certificate-based communication between the Veeam agent and the VBR server.
6. Once the certificate-based communication was setup, the VBR console would permanently delete your administrative credentials, as they would no longer be necessary, since all communication between VBR and the Veeam agent would be certificate-based from that point forward.

The current method in Veeam of using credentials is dangerous. I understand that gMSAs are coming in V12, but that's still dangerous. If an attacker discovered the credentials -- even if a gMSA credential -- the entire environment would be compromised.

Just to confirm, Microsoft Defender for Identity and BloodHound both show Veeam as paths for attack in our environment, and it's all because of the way VBR communicates with agents.

Some competing backup products already work without permanent credentials, so I would like to see Veeam move in that direction, too. Long-term storage of any credentials -- even gMSAs -- should no longer be necessary.

[HannesK]

Hello,
I merged your request to a similar request.

For agents: you can already work with certificate based authentication and without stored passwords with agents today. The protection group with pre-installed agents is doing that.
For vm-based backup: are you using application aware image processing actively? If not, then you can back up VMs without any credentials. A recent discussion on that topic is available here

Question 1: is your request about virtual machines or physical machines? Because 1. mentioned virtual machines and later it sounds like agent-based backup (VMs are normally backed up without agent)?
Question 2: how could one scale your approach with many VMs? Would you want to type in credentials for each new machine or what do you think about the manual certificate deployment suggestion from above?
Question 3: as far as I see, your approach is missing the attack vector via pre-script / post-script. What's your take on that? Would deploying a reg key on each machine that allows / blocks scripts be a valid approach for you?

Best regards,
Hannes

[cosmik]

Some competing backup products already work without permanent credentials, so I would like to see Veeam move in that direction, too. Long-term storage of any credentials -- even gMSAs -- should no longer be necessary.

Just adding here that I couldn't agree more with jfreeman. His proposal matches mine: utilize certificate-based authentication between agent and the specific VBR server. Again, not a developer here. Seeing AV solutions implement a similar approach perhaps it would be implemented here as well.

[jfreeman]

Question 1: is your request about virtual machines or physical machines? Because 1. mentioned virtual machines and later it sounds like agent-based backup (VMs are normally backed up without agent)?

It's about virtual machines in my case, but for this discussion, you could pretend that they're physical because they're virtual machines where we have regulatory and/or security reasons we can't connect into the hypervisor. So yes, they're virtual machines, but it's still agent-based.

Question 2: how could one scale your approach with many VMs? Would you want to type in credentials for each new machine or what do you think about the manual certificate deployment suggestion from above?

In our case, we would honestly be okay with one-time credential typing (just long enough to get the setup completed), although I do understand that may not work in some environments based on staffing and environment complexity.

Question 3: as far as I see, your approach is missing the attack vector via pre-script / post-script. What's your take on that? Would deploying a reg key on each machine that allows / blocks scripts be a valid approach for you?

I don't follow on this question. The attack vector is that a compromise of the VBR server is an automatic compromise of every single connected agent. An additional attack vector is that if an attacker somehow gets limited privileges on one of the backed-up virtual machines, if an attack is successful against the local SAM to figure out the local admin password that Veeam uses to perform backup jobs, that also means an automatic compromise of every single connected agent.


Regarding using "computers with pre-installed agents": isn't the limitation there that you can no longer use the VBR server to control backup jobs? If so, isn't that kind of worthless for most enterprises? I seem to remember running into this when we first setup VBR and the agents; we thought we needed deploy the agents first to the backed-up instances and then connect them to the VBR server, and that indeed worked, but we totally lost control over our backups, which is unacceptable. Why does this limitation exist?

[HannesK]

Hello,
Foreword: I think we all agree, that certificate based authentication can increase security. I'm just trying to figure out, how you would like to handle things at scale (thousands of machines and more) and what you think about different options from a usability perspective.

I don't follow on this question.

And Veeam does not do "security by obscurity" . If an attacker takes over the backup server while still having the option to run scripts with admin permissions, then having certificate based authentication is not really helping against anything.

The attack vector is that a compromise of the VBR server is an automatic compromise of every single connected agent.

That's why one should protect the backup server. There are many Windows hardening guides by different organizations (DISA STIG, CIS Benchmarks, whatever) and one can also follow the Veeam best practices guide

An additional attack vector is that if an attacker somehow gets limited privileges on one of the backed-up virtual machines, if an attack is successful against the local SAM to figure out the local admin password that Veeam uses to perform backup jobs, that also means an automatic compromise of every single connected agent.

as far as I remember, that's a low risk with NTLMv2 and proper passwords. But agree in general, that's an agent-specific attack vector if same credentials are used everywhere.


About pre-installed agents: yes, these agents behave different (scheduler is running on the agent side instead of the backup server, no manual "start / stop" of jobs). Whether it's really a "limitation" or just "different" is something each customer needs to decide on its own. I'm just saying, that there is an option today available, that does not require storing passwords on the backup server. For AIX & Solaris, we only have "pre-installed" agents and customers definitely use it.

but we totally lost control over our backups, which is unacceptable

could you maybe describe what that means? what exactly was lost?

Best regards,
Hannes