Security error when doing file level restore

[withanh]

I can't seem to find anything on this, but when I do a file level restore on a Windows VM, I get a security error when trying to grab the file. I get the same error whether I use the Veeam GUI or Windows Explorer

The error message is "You don't currently have permission to access this folder. Click Continue to permanently get access to this folder." When I click Continue, I get "You have been denied permission to access this folder. To gain access to this folder you will need to use the security tab." The security tab on the folder then says that "You do not have permission to view or edit this object's permission settings". When I go to Advanced and add my Veeam user as the owner I can get into the folder, but have to do this for each folder level down to the file, then when I do it to the file it says "You need permission to perform this action. You require permission from <domain>\Veeam to make changes to this file."

What am I missing here? I'm sure it's something obvious. Kind of like "You can't see the forest because the trees are in the way"

[Gostev]

Hi Darhl, what Veeam version is that for? I looked up some docs and this should not be happening starting v5 when using Backup Browser window to get the files, because we are now adding SeBackupPrivelege to the restore backup browser process, thus providing the ability to restore guest files that your account does not have NTFS permissions to access.

[withanh]

It's 5.0.1. To get around I made my Veeam service account a Domain Admin, but would like to remove those rights for obvious reasons.

Thanks!

[Gostev]

OK. Please open a support case and send the logs of restore attempts with different service accounts in place. Thanks.

[tsightler]

You might want to add the "Backup Operator" role to the Veeam service account rather than Domain Admin. That's what most backup products use and that should allow you to copy file regardless of permissions.

[Gostev]

Tom, this is the privilege we are adding automatically now, on the fly within the FLR process. And I am guessing this addition is what fails for some reason if user is not a Domain Admin. Or may be not. I'd like our devs to investigate this.

[tsightler]

Understood, but he doesn't want his Veeam service account to be a Domain Admin (good idea in my opinion) so he can just explicitly add "Backup Operator" to users which need to perform the restores. I hate the lame "automatically add roles" stuff anyway since it's effectively an unannounced privilege escalation which would be a major violation of our corporate security policy.

[Gostev]

I hate the lame "automatically add roles" stuff anyway since it's effectively an unannounced privilege escalation which would be a major violation of our corporate security policy.

If we'd implememt this in a lame way, than yes this would not be pretty. In our case though, we are not adding roles to a user account. Instead, we grant a single privilege locally within the backup browser UI process only (by modifying process token), so you cannot really do much with these elevated privileges other than restore files from backup. User does not have, and cannot leverage this privilege in any other OS process.

In fact, the workaround you are suggesting (adding permament role to the service account) is less secure, because this gives the service account permament and system-wide Backup Operator privilege (not temporary and for specific process only).

[tsightler]

I think you and I are going to have to agree to disagree on this one. If you do not see how having an application silently provide a privilege that has not been specifically granted is a major security issue then it would seem you haven't spent much time in an enterprise security setting and the auditing and documentation that must be performed for even the slightest changes. Now if it only works when the user is already an admin I guess it's not the end of the world, but it would still send our auditors and security guys into a tizzy if they discovered this behavior.

If it does somehow work when the user is not an admin, well, that's even worse. That's a silent privilege escalation that should be reported to CERT since, if your application can somehow do it, then I could easily write an exploit application to do the same thing.

As far as the user not being able to leverage the privilege, I think I can leverage quite easily to wipe out some serious data, and certainly can leverage to read files that I shouldn't have access to and since that's the reason the security exist in the first place your product has already failed by allowing access to data to an unauthorized person.

[Gostev]

Yes, let's disagree on this one in order to perform file level restore with Veeam, you have to be Local Administrator on the box, and have to be granted Restore Operator role within Veeam. Since the user is explicitly granted this role, I do not see how it is the issue that he/she can restore the files it does not have permissions to *from backup file*. In the end, isn't this what Restore Operator is all about? Otherwise, he/she would not be able to do its job?

And no, you cannot leverage this privilege to read some other data you do not have access to, you can only get whatever is stored in backup files. Note that this is not the case with the alternative workaround you have suggested (granting the Backup Operator role within OS), as this give you access to every file on the system immediately.

BTW, the elevation is not "silent" - the feature is documented in the product documentation.

[tsightler]

And no, you cannot leverage this privilege to read some other data you do not have access to, you can only get whatever is stored in backup files. Note that this is not the case with the alternative workaround you have suggested (granting the Backup Operator role within OS), as this give you access to every file on the system immediately.

Of course I can, that's pretty much the whole point of this thread, Veeam is providing a way to access files in the backup that the users doesn't otherwise have permissions to (that's what's not working in the OP's case). The user can exploit that to read data from the backups that they shouldn't have access to.

And yes, a restore operator should be able to restore files he doesn't otherwise have access to, but he should not be able to actually access the contents of those files. With our other backups products the service account for the backup application was granted the "Backup Operator" role, and the application controlled access to who could perform restores on what systems. They could restore files that they didn't have access to but they couldn't actually read the files since the restore was handled by the backup application itself, the backup operator simply scheduled the restore.

I also find it hilarious that your argument against using the "Backup Operators" group is that you add it only to the "process", but to do that the account has to be an admin. Requiring the account to be an admin seems far less secure than being a backup operator. If you notice, my suggestion was to add this permission in lieu of making the account a Domain Admin so my suggestion would provide far less privileges than what the OP actually did.

[Gostev]

Of course I can, that's pretty much the whole point of this thread, Veeam is providing a way to access files in the backup that the users doesn't otherwise have permissions to (that's what's not working in the OP's case). The user can exploit that to read data from the backups that they shouldn't have access to.

Because the user doing FLR must be local administrator, he can perform this exploit manually anyway. Just needs to know how to grant extra privileges to self. If you agree to this, then what is wrong with Veeam doing this automatically within its own process, not even system-wide as the user could potentially do?

Without this feature, users were unable to restore some files when they really needed to, and had to open a support case... did not really help in disaster situation when fast recovery is needed. Yes, you were one of few who did figure out the Backup Operator workaround, but that was more of an exception.

And yes, a restore operator should be able to restore files he doesn't otherwise have access to, but he should not be able to actually access the contents of those files.

Well, unfortunately Windows security makes it impossible for an account to read the content of the file without access rights this content and our current FLR architecture requires content copying.

With our other backups products the service account for the backup application was granted the "Backup Operator" role, and the application controlled access to who could perform restores on what systems. They could restore files that they didn't have access to but they couldn't actually read the files since the restore was handled by the backup application itself, the backup operator simply scheduled the restore.

You know this is not how our product works. Today, restores need to be done by users who are members of Local Administrators group on backup server, interactively, and by fetching the files directly from backup image that is mounted locally as a file system, to the user-specified location. Since you have questioned implementation of a very specific feature, in a very specific product (and version), why refer other products? They work differently, so they are irrelevant to this discussion. With this feature, we are addressing a very specific challenge users were facing with restores using our product with its current FLR architecture.

I also find it hilarious that your argument against using the "Backup Operators" group is that you add it only to the "process", but to do that the account has to be an admin. Requiring the account to be an admin seems far less secure than being a backup operator. If you notice, my suggestion was to add this permission in lieu of making the account a Domain Admin so my suggestion would provide far less privileges than what the OP actually did.

Well, the fact that using Domain Admin resolved the issue for Darhl does not yet mean this is a system requirement for our product. Just local admin should be sufficient. This is exactly why I requested 2 set of log files.

Now, because local admin (which we do require) is not a backup operator by default, it still cannot get files he/she does not have access to (other than from backup file). He can indeed grant himself Backup Operator at any moment, but this operation would be captured through Event Log monitoring system (which is always in place since we are talking about "an enterprise security setting"), so he would have explaining to do. With your workaround however, you grant this privilege and thus let the account full uncontrolled access to all files on the system... "officially" and forever. Which is why I thought this workaround is not as good (and is less secure) than what our software provides. See, this is not so hilarious in the end. Unless I am missing something funny in what I am saying

By the way, this feature was documented since early v5 aplhas (check out the Alpha3 release notes in focus group). I guess it just did not look too alarming until now, when this whole argument started

[tsightler]

Because the user doing FLR must be local administrator, he can perform this exploit manually anyway. Just needs to know how to grant extra privileges to self. If you agree to this, then what makes you upset with Veeam doing this automatically within its own process, not even system-wide as the user could potentially do?

Right, that's exactly the problem, the user performing the restore has to be a local admin. I'd rather the user always be "Backup Operator" than have your application require the person to be a "Local Admin" and dynamically give him "Backup Operator" level permissoins. To me it is totally obvious that requiring the user to be a "Local Admin" is far more privileged than being a "Backup Operator". That's pretty much the principal of "Least Privilege", don't require the user to have more privileges that required to get the job done. Unless I'm misunderstanding you (possible) it seems like you're arguing that requiring a user to always be a "Local Admin" is more secure than always being a "Backup Operator". I can't wrap my head around that argument.

Without this feature, users were unable to restore some files when they really needed to, and had to open a support case... that did not really help in disaster situation. Yes, you were one of few who did figure out the Backup Operator workaround, but that was more of an exception.

It's not a workaround, it been a standard part of Windows backup since the earliest days and would be considered startard operation 101.

What is the point of saying this in the context of this discussion? You know this is not how our product works. Today, restores need to be done by users who are members of Local Administrators group on backup server, interactively, and by fetching the files directly from backup image that is mounted locally as a file system, to the user-specified location. Since you have questioned implementation of one very specific feature, in a very specific product (and version), let's stick to this instead of referring to other products? They work differently, so they are really irrelevant to this discussion. With this feature, we are addressing a very specific challenge users were facing with restores using our product with its current FLR architecture.

Because that was pretty much the basis of my comment on the lameness of the solution. It wasn't so much just the "lame" privilege escalation, it's the entire lame file level restore, perhaps I didn't make that clear. IMO, Veeam should work like a well behaved enterprise application in this regard. It would be relatively trivial to implement a backend process that runs under the Veeam service account that performs the actual copy. This account could be granted the Backup Operator role - done automatically by the installer with a prompt saying the account was added to that role, or explicitly by the administrator - just like every other backup product I've ever used. The Veeam backup browser would simply talk to this backend service that has the appropriate role, and only users with specific Veeam privleges could perform restores. This backend process could also correctly handle permissions for Windows and other OS's.

Well, the fact that using Domain Admin resolved the issue for him does not yet mean this is a system requirement for our product. Just local admin should be sufficient. This is exactly why I requested 2 set of log files.

Understood, but a Local Admin is still far more privileged that a local Backup Operator. That seems obvious or he wouldn't be able to grant himself the lesser permission. How can you argue otherwise?

By the way, this feature was clearly documented since early v5 aplhas, check out the Alpha3 release notes in the Backup Focus Group forum. Guess it did not look too alarming to you until now, when this whole argument started

I never said it was "alarming" I said it was "lame" and that I "hated" it. I was generally referring to the entire FLR process, not just the silent privilege escalation (and it's silent because it just does it without any interactive notification, it doesn't matter if it's documented in some release notes somewhere, every other backup I've ever installed either pops up a message saying the service account is being added the Backup Operator role, or requires that the admin add that role and warns if it hasn't been done). The FLR weakness is pretty much the most glaring weakness of the product in our company and one of the reasons we continue to look for the ideal backup (Veeam's as close as we've been able to get to in a decade, but it still has lots of pain points). I'm quite aware of how it works and we have internal processes that satisfy our auditing requirements for this issue, but I'm allowed to hate it.

[Gostev]

Uh, I would never argue that the current FLR architecture in the whole is bad for an enteprise security setting. I was only arguing against calling the specific feature's implementation (runtime privilege elevation) lame, because I think it is actually quite cool way to deal with the issue we have been facing due to the current FLR design. Current architecture is there in the product for historical reasons, and these minor improvements make it more usable.

Anyway, good news is, in v6 FLR was quite improved to address the issues you are talking about above.

[withanh]

in order to perform file level restore with Veeam, you have to be Local Administrator on the box, and have to be granted Restore Operator role within Veeam.

My service account was a Local Admin, but had not been granted Restore Operator role in Veeam. I'll do that, and if that doesn't fix it, then I'll open a case with support.

h

[Gostev]

Restore Operator must be granted to actual user account performing the file restore, not to service account. But I am sure you are doing everything using Veeam Backup administrator's account anyway, which includes Restore Operator.

[withanh]

I log in to the VBR server using the Veeam service account. I use that account to do pretty much everything, including the restore I attempted. It did not have explicit "Restore Operator" rights, are you saying I should not need to explicitly apply those rights because it is the service account?

I'm doing a large file copy right now (1.3T and can't log out/log in to reset the security. When it is done later today I will check to see what happens with restoring a file.

h

[tsightler]

Uh, I would never argue that the current FLR architecture in the whole is bad for an enteprise security setting. I was only arguing against calling the specific feature's implementation (runtime privilege elevation) lame, because I think it is actually quite cool way to deal with the issue we have been facing due to the current FLR design. Current architecture is there in the product for historical reasons, and these minor improvements make it more usable.

Yes, looking back at the original comment I can see how I was not very precise in my criticism. I was attempting to indicate that I thought the whole reason for having to dynamically add a privilege was lame, but my comment reads as if I'm criticizing the method, which I do hate, but understood, and then we got all off on a tangent (sorry OP). I guess I should have been more clear.

[withanh]

It was a good read, I don't mind the hijack of the thread

[Gostev]

I log in to the VBR server using the Veeam service account. I use that account to do pretty much everything, including the restore I attempted. It did not have explicit "Restore Operator" rights, are you saying I should not need to explicitly apply those rights because it is the service account?

No, but if you used this account to install the product, then it should have been automatically granted full backup administrator role in Veeam anyway, which includes all privileges.

Again, seeing restore logs with service account being just a member of Local Administrators group, versus a member of Domain Admin should help. I already have an idea though, may be UAC is causing this. So try disabling it, and retrying.

[withanh]

It was originally installed using domain\administrator account. Was later switched to domain\veeam service account. But I think at least one update (5.0.0 to 5.0.1 was done using this account).

h

[Gostev]

OK, then please double check user roles under Options just in case.

[withanh]

Was already there:
Administrators = Veeam Backup Viewer
Administrators = Veeam Backup Administrator

I then added:
domain\Veeam = Veeam Restore Operator

Other item of note, the domain\Veeam user already was (and still is) a member of the local Administrator group.

h