encrypted credentials and NAS

[robg]

Hello

I'm investigating other ways to hide backups from ransomware without an air gap. Let's say I have a NAS repository running a regular SMB share protected by a user and pass, and credentials to access it are encrypted inside veeam. Does this mean that the attacker can't reach the data unless they access Veeam? Is there a way to protect access to Veeam itself by way of a master password?
thanks

[Mildur]

Hi Rob

If the data is on a NAS, they don‘t need to reach veeam.
They can delete all data on the NAS directly.

A master password is not really secure. If you want to protect your backup server, you need to leverage MFA and make sure, that the backup server is not connected to the production active directory. You will find some solutions in the best practice guide:
Infrastructure Hardening

[robg]

Wait a minute. How would they go about deleting the NAS directly? This doesn't make any sense. You're probably assuming that the NAS is tied to AD and its contents are easily reachable if you're an admin. That's a disaster waiting to happen.

Let me be clearer. Access to the NAS is by a unique user and password that is only inside of Veeam. Is it secure in there with the credential encryption feature turned on?

I would also disagree with you about the master password. You're assuming that everything on the network is right there on a silver platter for the taking. Not every environment is that poorly designed.

[Mildur]

Is it secure in there with the credential encryption feature turned on

No credential is secure inside of veeam. You can decrypt them with the backup servers machine key if you gain admin access to the backup server.

[robg]

Now we're getting somewhere.. I would have to look into this more, thanks for the info. However, it doesn't make sense to me that Veeam's encryption would be so easily defeated, otherwise what's the point in putting it in there, only to stop the casual bad actors and not the pros? Maybe someone from the company can chime in on this

[Mildur]

Which encryption your are talking?

The one from the configuration backup? This one can only be decrypted with your password or the enterprise manager.

Beside of that, veeam writes the credentials from the backup repos and jobs to the sql database in an encrypted form. The encryption is done by the Data Protection API from the operating system. And this API can be used to get the passwords decrypted.

Veeam - Encryption Standards

[robg]

Yes, that too- the configuration backup (I would assume contains the passwords, I've never restored a configuration). It makes sense to encrypt that.. beyond that, there is no protection?

I'm sure that there is some sophisticated ransomware out there that can automatically manipulate backup software like Veeam, I was looking for more ways to protect it. My backup server is already isolated from AD.

[Mildur]

You could disable RDP on the Backup Server. There is no need to directly logon to the VBR Server.
For daily operations, it is enough to access it with the veeam console from a dedicated machine.
You only need RDP Access in the case of maintenance activity.

But then again, if it's a VM, a hacker can get access over the vm console, if he has access to the Hypervisor Management. Resetting a windows password is not that difficult if you have control about the boot process of a vm.

[robg]

Yeah, I'm aware of the various methods of securing a backup server, it's already secure enough, but I was asking about the mechanisms that exist within Veeam itself, if any. If there is encryption, it shouldn't be circumventable in any way, maybe I'm missing something.

[veremin]

I'm sure that there is some sophisticated ransomware out there that can automatically manipulate backup software like Veeam, I was looking for more ways to protect it.

Isn't this just a continuation of the discussion that you've previously started? If you want to rest assured that no one can do something with your backups, why not to leverage Hardened Repository that makes backups immutable? Thanks!

[robg]

No, it isn't. In this thread, I am asking about the level of protection that Veeam has for its saved credentials.

[veremin]

Understood, was thinking that underneath it's still the question of how you guarantee the maximum level of protection for backups without taking them offline (Hardened Repository can you help you with that).

As to encrypted credentials, password recovery and other related things - those have been discussed several times on these forums, including this topic. So feel free to search the community and ask clarifying questions, if any are still left.

Thanks!

[robg]

I read the post you linked, and the result is disappointing.

Gostev said this:

" 1. There's no protection against an account with root privileges in principle, as any "limits" you put they can always revert. "

Disagree as far as encryption is concerned. If a root account is compromised, and within it there are encrypted files with a different password, being root won't recover it.

A hardened repository requires a linux server, and that isn't an ideal thing to setup and maintain for everyone. It's disappointing that Veeam doesn't have this function internally with encryption..

[Gostev]

Since the encryption password history is stored in the configuration database to enable things like self-service restores from said backup files, root will be able to access those encryption passwords and perform recovery.

Backup file encryption was never meant to be a protection against root. These guys can always steal production data by simply creating some new backups without any encryption. Its sole purpose is to ensure that if backup files are copied off backup repositories by a malicious actor, they won't be recoverable by anyone who does not know the password. The only way to restore without supplying the password is through the backup server, where this activity is monitored and audited.

[robg]

Yeah, but I'm asking about backup credential encryption, not backup file encryption . It would be nice if Veeam could keep the passwords it uses to access resources safe via encryption, that doesn't seem to be the case.. I don't think this would affect self-service restore or the program's operation, only that the passwords themselves can never be found..

[Mildur]

If veeam would encrypt the passwords for accessing the nas, in a way, that nobody can read it, how would veeam use it to access the nas?

It would need to decrypt the passwords for accessing the nas over network when the backup job starts. So the decryption key must be accessible for the veeam services to decrypt the encrypted credentials when needed.

The decryption key needs to be stored somewhere on the backup server, or veeam cannot use it to access your nas share. Storing it in the database would be bad too. An sql admin could export the keys then and use it to decrypt.

As soon a decryption key is stored on the backup server, an privileged user (local administrator) can get access to that key and use it outside of veeam.

There is no solution to encrypt something without the possibility to decrypt it when it needs to be accessed.

The only way to get rid of encrypted/decrypted credentials in veeam is the Linux hardened Repo with single use credentials. That‘s veeam‘s solution for this specific scenario. And it‘s a good one. Simple and effective. And it doesn‘t cost a fortune to implement.

[robg]

> If veeam would encrypt the passwords for accessing the nas, in a way, that nobody can read it, how would veeam use it to access the nas?

Mildur, by hashing it, have you ever looked at the configuration for a cisco device. The passwords aren't stored in plain text, but there is no trouble connecting persistent VPNs. It would appear that Veeam is storing passwords in plain text.

[Gostev]

LOL, no. All credentials are stored encrypted with the machine-specific encryption key. Thanks to this, making a copy of Veeam configuration database files will not do a malicious person any good, as credentials stored in the database will not be decryptable on any other machine except on the backup server itself.

And as Mildur already explained above, they MUST be decryptable on the backup server, meaning root can always do the same too.

[Mildur]

Veeam needs to read the password from the veeam sql database, and then open a smb connection to the nas with that password. Veeam cannot use a encrypted string to access the nas. The nas would say: „wrong password, I don‘t let you in “. Veeam needs the plain text password, which you have configured, to start the smb connection.

A VPN Connection from a Client to a server is something else.

[Gostev]

But it's really the same? A client still needs to use some secret only known to this client to establish the connection with a server, otherwise anyone else would also be able to connect to a server while impersonating this client? And this secret needs to be in the plain-text form whenever it is used by the code that establishes the connection.

[robg]

> Veeam cannot use a encrypted string to access the nas. The nas would say: „wrong password, I don‘t let you in

Mildur, this isn't what I was saying. There is lots of software and hardware on the market that doesn't need to store any plaintext passwords. I'm not saying that I expected Veeam to transmit a hashed password.

If Veeam stores the password encrypted, then that answers my question! If Gostev is saying that any attacker can reverse this just because they're root, that only works if they know what type of hash it is, right? Great discussions here as always

[Gostev]

No, root does not need to know anything. Root can just call the same function that the application uses to decrypt stored credentials when it needs them. Everything this function requires will be stored on the system, as without this data an application won't be able to perform a decryption when it needs credentials (for example, to connect to an SMB share during a scheduled backup job).

[robg]

I see. At this point though we're getting into the "how likely is any of this" - in terms of the sophistication of either a live human attack or an automated process. My opinion is that it's unlikely for them to go that far as to call functions to decrypt these credentials, but I never say never. That's why the measures to secure the backup server itself are important.

The second part of my question is, would a master password thwart a casual attacker? I think it would, unless they spent a significant amount of time reverse engineering Veeam to get past it.

[Gostev]

It is not just likely - it's almost for certain! Don't bet on complexity because it is not sophisticated at all, in fact it is so easy that my grandma could do this!

You just use PowerShell as root to load the Veeam library with the decryption function and then call this function:

Code: Select all

Add-Type -Path "C:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Common.dll"
$encoded = 'encrypted credentials string from configuration database'
[Veeam.Backup.Common.ProtectedStorage]::GetLocalString($encoded)

"Master passwords" concept doesn't work in principle. They are only for people who actually follow the rules. But bad guys never do... so why would they bother providing that extra master password when they can just cut to the chase and call the decryption function directly, just like the product will ultimately do once the correct master password is supplied? Even requiring 10 people each with their own master password to come and supply it does not increase security by a bit.

Microsoft does not even bother and just allows all users with Local Administrator privileges to see all saved credentials in the Windows Credentials Manager (while they are too stored encrypted). Same with Google Chrome, while as you know Google has some of the world's top security engineers. Because really, there's no meaningful protection against root is possible in principle.

So you should turn your focus on making sure attackers cannot obtain root privileges on your backup server. Because once they do, nothing will stop them from getting your stored credentials. Putting your backup server in the dedicated AD forest, setting up MFA solution like Duo and disabling all remote interfaces which your chosen MFA product does not cover is a great start.

[robg]

Once again, I know that it's important to protect the backup server, three times now, and I've done that, this is a fact-finding thread, not a "I'm seeking advice thread" but thank you.

I'd disagree on the master passwords, they do work, but only if what they are protecting is 100% un-breachable, like with strong encryption. You (Veeam) have your reasons for being able to pull credentials out that easily from the command line, but that function could be password protected. The bad guys can't break unbreakable rules. And of course it goes without saying that the master password has to be properly controlled. That's much easier to do in small environments with less moving parts.

What about keyloggers capturing passwords? Sure, only if I'm incompetent..

Safari in OS X requires the user to enter their password in order to reveal credentials, which I'm sure is the same in the other examples you mentioned.

[Gostev]

I'd disagree on the master passwords, they do work, but only if what they are protecting is 100% un-breachable, like with strong encryption. You (Veeam) have your reasons for being able to pull credentials out that easily from the command line, but that function could be password protected. The bad guys can't break unbreakable rules.

Sorry to say, but these statements show that you have no idea what you're talking about... so it's best we stop wasting each other's time and finish this conversation.

[robg]

How do you figure that? How are they going to get that password out of the system? Brute force it? What if it's sufficiently complex? Are we speaking in the area now of "Well a billion years to crack something isn't a zero percent chance" ?

[robg]

Ok, I'd like to make sure I understand what was said, apparently I have no idea what I'm talking about, maybe somebody could help enlighten me?

1. Veeam needs to be able to decrypt NAS credentials from its database and send them to the device. Ok, fair..
2. A master password can't prevent an attacker from seeing credentials because if he has root, he can access everything, and all bets are off. < This is the part that doesn't make sense to me.

If you're root, and there's an encrypted container on the system, being root doesn't help you, you still need the password to get into it. Why can't some of Veeam's functions be encapsulated in that way?

I am not saying I disagree with any design decisions, but it seems to me that implementing something like this could go a long way to protect backups in simple environments, a linux installation for immutability can't possibly be the only way.

[Gostev]

For Veeam itself to be able to access data in this "encrypted container" when jobs run, the "master password" would have to be stored somewhere on the system, meaning root can always retrieve it too... simple as that.

[robg]

Got it.. What if everything that Veeam does lives in this container? All of its internal functions are there. I suppose then it couldn't start up on boot without someone entering the password.