Comprehensive data protection for all workloads
Gostev
Chief Product Officer
Posts: 31809
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: encrypted credentials and NAS

Post by Gostev »

Sounds like Windows BitLocker :) I cannot boot my laptop without supplying pre-boot authentication to enable OS to access data on encrypted disks.

But once the OS does get access, nothing else matters. Ultimately it will be the OS running all the business logic of that "container" just like it does any other app, so OS root user will always be able to access all data.

Besides, making "everything that Veeam does live in this container" is impossible even in theory, because credentials have to be passed onto external systems, meaning they have to leave the container in an unencrypted form.
mcz
Veeam Legend
Posts: 945
Liked: 221 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: encrypted credentials and NAS

Post by mcz »

...and even if it was not stored clear-text, say if you used a hash or any other form, that wouldn't help at all. That's why pass-the-hash attacks are possible. No matter what you're trying to do, veeam (or it's OS) always holds a secret that can be stolen when you're root or if you have access to the hardware. Like the story in the one article Anton has shared some weeks ago when someone grabbed the bitlocker-key by "sniffing" directly on the hardware...
robg
Expert
Posts: 176
Liked: 18 times
Joined: Aug 15, 2014 11:21 am
Full Name: Rob
Contact:

Re: encrypted credentials and NAS

Post by robg »

Well, assuming the software operates inside of this encrypted container (and assuming it can start itself without the master password). How would the attacker be able to get the hash? The only thing he could do is sniff the network connection when it's authenticating to the NAS, and I don't see that happening when it's SSL. I'm not aware of the story you mentioned, but I assume that the bitlocker key was stolen by a keylogger.
Gostev
Chief Product Officer
Posts: 31809
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: encrypted credentials and NAS

Post by Gostev » 2 people like this post

We're going in circles.

The software cannot "operate inside of the encrypted container", this is not how computers work!!! The software runs on the operating system! While a program can have its data stored in an encrypted container, this data has to be unlocked before the program can even start running. And once it does, ultimately the program is executed by the OS using unencrypted data from the container, as if there were no encrypted container at all.

And because the OS has access to all of the application's data, so does the root user. For example, if backup server has a function that, as a part of a backup job, establishes a connection to some external system by passing credentials to it, then while your backups server is running (which is 24/7) root can call this function directly too and obtain credentials this way.

Just accept the fact that root has the exact same privileges and the same level of data access as the OS itself. While the OS can only work with unencrypted data when running any program.
robg
Expert
Posts: 176
Liked: 18 times
Joined: Aug 15, 2014 11:21 am
Full Name: Rob
Contact:

Re: encrypted credentials and NAS

Post by robg »

Gostev, I realize all of that, and this is more of a conceptual/theoretical discussion at this point. It would take a fundamental re-design of a lot of things to accomplish.

The backup software itself would probably have to be some sort of appliance VM, closed source. We can agree that there is no getting into that if its console has no known exploits.

Regardng the backup function's backup job example, it doesn't have to offer that ability, it can say "I can only call backup functions internally."

Yes, "just secure the backup server itself" is the real world answer, but this idea of an "internal immutability" is interesting to me.
Gostev
Chief Product Officer
Posts: 31809
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: encrypted credentials and NAS

Post by Gostev » 2 people like this post

robg wrote: Nov 22, 2021 3:23 pmThe backup software itself would probably have to be some sort of appliance VM, closed source. We can agree that there is no getting into that if its console has no known exploits.
A regular Windows VM meets all these requirements already ;)
robg wrote: Nov 22, 2021 3:23 pmRegardng the backup function's backup job example, it doesn't have to offer that ability, it can say "I can only call backup functions internally."
There's no possibility to restrict certain function to be called "internally" only. The operating system (and so the root user) cannot be restricted from calling any function of any executable or library present on the system even in theory.

For example, to execute any function from a library, you can run:

Code: Select all

rundll32 some.dll,function argument
You can try the following command yourself:

Code: Select all

rundll32.exe keymgr.dll,KRShowKeyMgr
robg
Expert
Posts: 176
Liked: 18 times
Joined: Aug 15, 2014 11:21 am
Full Name: Rob
Contact:

Re: encrypted credentials and NAS

Post by robg »

Right, but the system would need to be unlocked and open to execute those functions. What I'm talking about is a system designed from the ground up to be resilient from intruders, but of course the argument could be made "just follow best practices and existing methods." Nothing would ever improve if no one ever questioned that :)
Gostev
Chief Product Officer
Posts: 31809
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: encrypted credentials and NAS

Post by Gostev »

This is exactly why I made a reference to Windows Bitlocker earlier in this thread. It comes very close to what you're trying to explain, providing an excellent protection of your data as long as the system remains powered of... so if your laptop gets stolen, data on disks will not be extractable.

However, the system must be unlocked the moment you start it, because neither operating system nor hardware can work with meaningless blobs of encrypted data. And the moment it is unlocked, you can just forget it was ever locked, because all encryption becomes transparent for the OS (as if encryption did not exist at all).

Basically, your suggested approach is great for interactive systems which only work for a split second and immediately shut down. In fact, you use this approach daily when you pay with credit cards and have to provide a PIN code, which "unlocks" the system just for a brief moment to allow the microchip to calculate a response from a challenge provided by the payment system's server.

But servers are just not like that. Once you power them on and unlock, they keep running for weeks and the fact that they were "locked" before they were started makes absolutely no difference.
robg
Expert
Posts: 176
Liked: 18 times
Joined: Aug 15, 2014 11:21 am
Full Name: Rob
Contact:

Re: encrypted credentials and NAS

Post by robg »

I'm sure Bitlocker is just like os x filevault, you have to enter the password for it to boot, and then the services and processes start up. But if the system goes to sleep, it wakes up and is still running with everything in the background, you need the password again to get past the lock screen. This is the part that I'm focused on.

The solution would be two passwords. If "encrypted at rest" is a concern then the first password would be retrieved from a mechanism that is unreachable from the backup server (an external source), and the second is the "master password" that the backup admin uses to interact with the program.

This is of course all theoretical and conceptual. No one would be thinking about this just to secure a backup server, but in the larger context of stopping ransomware in these peculiar times, it's a worthy thing to consider
Gostev
Chief Product Officer
Posts: 31809
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: encrypted credentials and NAS

Post by Gostev » 3 people like this post

The problem with your proposals is not that they are theoretical or conceptual, but rather that they are making no sense in principle. For some reason, you just don't listen to the voice of reason and keep returning to the same points we've already been over again and again. Now, you're back to a magical "master password" without knowing which root is supposed to give up and walk away... OK.

In that case I can only repeat that as soon as the backup application is "unlocked" and starts running, the OS (and so the root user) can access all encrypted data used by the application transparently. So at this point, requiring the second "master password" to allow a user to "interact with the program" does not add any additional security against guys with root privileges. Because they can just interact with all data structures (including stored credentials) directly, and could not care less that the UI is locked by some second password.

Not to mention, they can as well simply overwrite the "master password" with their desired value, because the application will always have a function to set and update one (as required to be able to update the password periodically). And nothing prevents root from calling that function directly, completely bypassing all extra authorizations you put in place.

Not to mention, that they can always just directly trigger the UI function that the application calls in cases when the correct "master password" is provided by the user, thus launching the UI as if the correct password was supplied.

Not to mention, that they can just hook the function that checks the password and make it always return "true" regardless of what password was supplied.
robg
Expert
Posts: 176
Liked: 18 times
Joined: Aug 15, 2014 11:21 am
Full Name: Rob
Contact:

Re: encrypted credentials and NAS

Post by robg »

Yeah, I am talking about theory and concept and an unprecedented design.

You keep talking about root having access to everything, and I am saying that this system is in a container that is completely inaccessible. Can you get into a VM without credentials? Yes, but only by breaking into it with known methods. What if there aren't any known methods? Then the only thing you can do is erase it. The data is still safe on the NAS, because the credentials were never compromised.

Once again, imagine a second VM inside the OS that contains the backup software. Rooting the OS gets you nowhere, because nothing can go in or out of the backup system except the backup data streams and credentials to external storage via SSL.

As I said before, the system can only start up through another means of authentication that's outside of the system's reach, which makes it bulletproof. From a conceptual point of view, what's wrong with this design?
Gostev
Chief Product Officer
Posts: 31809
Liked: 7300 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: encrypted credentials and NAS

Post by Gostev » 5 people like this post

Everyone who knows anything about virtualization knows what is wrong with this design: there are about 1 million ways of breaking into the VM once you have root privileges to the hypervisor host. Since hypervisor can read and modify VM's memory and disk content, nothing prevents the hypervisor OS root user from doing the same. And this is all that is needed to take over the guest OS.

And now that you provided the perfect demonstration of your actual level of knowledge on the subject of security, I'm really done with this conversation for good. Sorry, but I'm not willing to waste any more of my time on this...
robg
Expert
Posts: 176
Liked: 18 times
Joined: Aug 15, 2014 11:21 am
Full Name: Rob
Contact:

Re: encrypted credentials and NAS

Post by robg »

Yeah, they can only break into the VM with "known methods" - I did say "what if there aren't any known methods?" If there is nothing sensible to get from the memory and disk, as in, the system is closed and proprietary, that wouldn't do them any good. We're talking about some serious efforts at reverse engineering now when we both know that most of the "ingenuity" of a typical ransomware hacker is to exploit known weaknesses on the backs of other people's work, not discover new ones. From that standpoint, the concept is sound.
robg
Expert
Posts: 176
Liked: 18 times
Joined: Aug 15, 2014 11:21 am
Full Name: Rob
Contact:

Re: encrypted credentials and NAS

Post by robg »

And thanks for your time Gostev, I don't want you to feel like you've wasted it. If a repository can be made immutable, then other important things probably can too, that's an idea that I am interested in, and I don't think it's impossible.
mcz
Veeam Legend
Posts: 945
Liked: 221 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: encrypted credentials and NAS

Post by mcz »

robg wrote: Nov 22, 2021 8:29 pm The data is still safe on the NAS, because the credentials were never compromised.
I bet there's a Linux running on that NAS.
robg
Expert
Posts: 176
Liked: 18 times
Joined: Aug 15, 2014 11:21 am
Full Name: Rob
Contact:

Re: encrypted credentials and NAS

Post by robg »

I bet that's true, and I bet there may be known exploits for it, even with the latest updates. But at the start of this thread, I was only asking about protecting its username and password. We could go endlessly on hypothetical what-if's. Even air-gapped backups aren't perfectly safe from physical theft to the most dedicated criminal!
mcz
Veeam Legend
Posts: 945
Liked: 221 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: encrypted credentials and NAS

Post by mcz »

We could, be we won't - at least not me. I'm out as well. Everything said here.
robg
Expert
Posts: 176
Liked: 18 times
Joined: Aug 15, 2014 11:21 am
Full Name: Rob
Contact:

Re: encrypted credentials and NAS

Post by robg »

Neither will I. Rome wasn't built in a day, there is room for improvement in terms of design resiliency for these kinds of attacks. Either the software will get better, or the threat will diminish as it gets harder for bad actors to participate due to pressure from groups like Interpol.
Post Reply

Who is online

Users browsing this forum: Amazon [Bot], CoLa, Google [Bot], ITP-Stan and 308 guests