Comprehensive data protection for all workloads
Post Reply
tom11011
Expert
Posts: 192
Liked: 9 times
Joined: Dec 01, 2010 8:40 pm
Full Name: Tom
Contact:

Managing security in Veeam, read-only backup targets

Post by tom11011 »

I happened to read Gostev's weekly forum digest email last night. In the digest he says that an entire company was destroyed and put out of business because a hacker destroyed an entire infrastructure along with all the backups.

You can read all about it here http://www.codespaces.com
In case the message is removed I'm going to post it at the end.

So this got me thinking about my own security and areas where I might be lacking and subject to the same kind of disaster. I wanted to start a meaningful discussion on what steps should be taken to mitigate this risk.

To give you a little background on our setup, we have a Production site (site A) and a DR/Development site (site B). The sites are geographically distant.

Site A backs up to site A.
Site B backs up to site B.
Site A replicates all day long to site B.
And as replicas are not really backups, we do a copy job for the two most important database servers from site A to site B daily.

Our vmware environment from a management perspective is integrated with Active Directory. I can now see that this might be a mistake. I am going to propose that the AD integration be removed from SSO. The worry is so many other pieces are tied to AD such as vpn access and enterprise wireless. Even a careless admin could leave a session open while he steps away from his desk.

That would give me a little piece of mind, but what about Veeam? Should the Veeam application not have the ability to have a password prompt prior to launching the management application?

We have successfully moved away from things such as archive and tape. The goal has always been a disk based solution as the ultimate achievement but now I'm having some second thoughts.
marty9876
Enthusiast
Posts: 30
Liked: never
Joined: Sep 22, 2010 2:53 pm
Contact:

[MERGED] : Read only backups targets- format options

Post by marty9876 »

Hi All,

Looking to get some ideas on what is being used for read only storage targets for Veeam backups. After a few reports of companies being wiped out (Codespaces.com etc.) makes me wonder how much of a wall can and should be built around backups. Something akin to the 3-2-1 backup plan where a copy lives on a different format.

Seems anything on disk (be it SAN/NAS/Cloud etc.) ultimately has it's permissions set and as such these permissions could be changed. Outside of a actual tape on a shelf or WORM tape I'm at a loss for ideas. USB drives or removable disk media (Dell PowerVault RD1000) seems to be an option but is clunky at best.

Is separation of systems via different permissions setting good enough for you or will tape never die? :)
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Managing security in Veeam

Post by dellock6 » 1 person likes this post

Hi all,
I'm happy that Anton's digest hit the point, security is often overlooked in data protection designs...
The read-only repository is not the best solution, because than you create problem to the retention settings and the ability to remove older restore points when it kicks in. We are working on a paper or a blog post addressing these concerns, but some ideas:
- least privileges: reduce the access to Veeam console as much as possible, use dedicated service accounts to run services instead of existing users, use Enterprise Manager with limited accounts for daily operations. You do not need always an enterprise admin
- put some backups offline. Not offsite, OFFLINE. This is easy with tapes and RD1000 (write to them, and remove them), but it can be done also with disk repositories. Have a secondary repository storage, NOT managed by Veeam so it does not appear in the console and it cannot be accessed when opening the console. With another tool, every while copy VBKs into this storage, so these copies DO NOT appear and are not reachable from Veeam console...

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by Gostev »

And here is the proof this white paper is much needed, Luca :D
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by dellock6 »

I got the message :)
Working on it, expect the paper in time for V8.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
tfelice
Influencer
Posts: 10
Liked: never
Joined: May 23, 2012 12:28 pm
Full Name: Tony Felice
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by tfelice »

The very simple scenario that worries me is that someone at the production site manages to get access to the Windows domain administrator credentials, logs in to the Veeam B&R server, and deletes backups. Since right now there is no additional challenge in the B&R UI, once you have access to the UI, you can delete backups and backup copies from any or all backup targets, be they local or remote. Thus, you can be just as SOL as the Code Spaces folks, having multiply-redundant backups, all able to be deleted by one malicious actor having gotten access to one machine/interface. Is there a resolution to this vulnerability in the present B&R software that I am not aware of?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by Vitaliy S. »

I think the solution should be taking backups offline, as Luca has advised above.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by Gostev »

Tony, you have just explained the reason why large amount of companies still use tapes stored in a secure offsite location like Iron Mountain ;)
tfelice wrote:Since right now there is no additional challenge in the B&R UI, once you have access to the UI, you can delete backups and backup copies from any or all backup targets, be they local or remote.
Actually, no amount of challenges in the B&R UI would help in your scenario, because having access to the Windows domain administrator credentials, you can simply logon directly to your backup repository boxes, and physically delete all backup files there. Thus, adding challenges to B&R UI is similar to reinforcing your main house door when you have no lock on your back door... no pun intended.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by dellock6 »

This is not a way to defend Veeam, but Tony you are describing a problem that is somewhere else.
How do you protect your domain administrator accounts? Once you protect them, EVERY service based on their authentication (Veeam and other software) are automatically protected. One of the challenges (no pun intended) in security is identifying the real source of a problem.

I'm not going into too many details before the said paper, but some ideas:
- run EVERY service with a dedicated account, that has a super-complex password and the minimum set of privileges required. Never use domain admins if the softare can run with lower privileges. Never use the same account into two or more services.
- create few, really few domain/enterprise admin, have a decent password, change it often, audit their logins. hopefully, each admin should have his own account, so you can map who used it.
- for same reason, rename "administrator" in the domain, set an incredibly strong password, and save it on paper, not in any electronic document. Save the paper in a safe place, write a procedure for password retrieval (who's authorize to retreive it, under which conditions). Change it again after every use.

And as Anton said, check everythng. Is ITsec this is called Defense in Depth, which means add layers of protections to any component. Your overall security is good as the weakest link you have. If you plan in that way for AD, also plan physical security, access control, auditing, otherwise AD protection alone is useless.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
tom11011
Expert
Posts: 192
Liked: 9 times
Joined: Dec 01, 2010 8:40 pm
Full Name: Tom
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by tom11011 »

Gostev wrote:Actually, no amount of challenges in the B&R UI would help in your scenario, because having access to the Windows domain administrator credentials, you can simply logon directly to your backup repository boxes, and physically delete all backup files there. Thus, adding challenges to B&R UI is similar to reinforcing your main house door when you have no lock on your back door... no pun intended.
While this may be true, I think we would all agree that security is a multi layer effort. Veeam should reconsider and at least offer the option to have a login challenge. Why should Veeam be different than VMware in this case?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by Gostev »

Hi Tom, what VMware login challenge are you talking about?
tom11011
Expert
Posts: 192
Liked: 9 times
Joined: Dec 01, 2010 8:40 pm
Full Name: Tom
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by tom11011 »

Gostev wrote:Hi Tom, what VMware login challenge are you talking about?
When I launch the vmware client, a login dialog box is presented.
DavorS
Veeam ProPartner
Posts: 16
Liked: 8 times
Joined: Jul 09, 2014 11:42 am
Full Name: Davor Stanec
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by DavorS » 1 person likes this post

tom11011 wrote: When I launch the vmware client, a login dialog box is presented.
But when you log onto your repository server, a login dialog box is also presented :)

But i would like to see some sort of granular permissions on Veeam B&R - especially in Veeam Explorer for MS Exchange - Anyone with rights to log on to Veeam B&R server can open up the console and read ALL the emails. This is a big problem for some of my clients as they don't want their whole admin team to be able to read emails, but they do want them to have all the other options. I would greatly appreciate a feature like that.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by Gostev »

Davor, what other options would they like them to have? For example, if they still want them to be able to do file level recoveries, then there is nothing preventing admins from restore MDB and... still reading all emails with other tools similar to Veeam Explorer for Exchange.

However, if you want to really limit your admin, then we already have some limited roles available that you can use today. For example, Backup Operator who can only control backup jobs, but nothing else.
tfelice
Influencer
Posts: 10
Liked: never
Joined: May 23, 2012 12:28 pm
Full Name: Tony Felice
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by tfelice »

Actually, no amount of challenges in the B&R UI would help in your scenario, because having access to the Windows domain administrator credentials, you can simply logon directly to your backup repository boxes, and physically delete all backup files there.
I don't believe this is correct. I'm thinking of a customer site where backup copies go to an off-site repositories in a data center. The off-site servers are not members of the Windows domain. Nobody at the customer site has any kind of access to the off-site servers except through the Veeam UI. A second authentication factor, such as a challenge for a secret not maintained in Active Directory, would certainly add a layer of protection here. Just getting to the console of the Veeam server would not allow a bad guy to delete the remote backups.

I understand the advice posted here about Windows/network security best practices. I observe such practices on my own corporate network and advise my clients to do the same. But as a consultant, I deal with many organizations, and the reality is that customers don't always take their consultants' advice. I think it would be wrong to just say, "too bad for them, they should have had better policies" if something goes wrong that a more thoughtful software design could have avoided. I suggest that Veeam consider that your customers are not all end users. Some of us are consultants, and the last thing we ever want to hear is "you told me that this backup system was safe and reliable, and now I'm in trouble because I accepted your recommendation." Sure, one could argue that the customer is simply wrong and should know better. But again any decent consultant will want to do everything possible to avoid ever having to make that argument.

My suggestion is simply this. Add a second challenge when a user attempts to delete a backup, and make that challenge independent of Active Directory. Most users would rarely if ever have to jump through that hoop, but it would provide protection against one of the most catastrophic things a malicious party could do. You could make this feature an option, and even default it to off. That solves the specific vulnerability I'm proposing and breaks nothing.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by dellock6 »

Quick idea Tony: what about a "backup retention protection" enabled in the backup job? in my mind, this setting will still allow to delete a restore point once the configured retention is reached, while it will ask the protection password to delete any restore point "before" their expiration date. The password will be configured in the job, using the "credential manager" we already have in place.
Obviously, still an admin can login into the server, so additional protection will need to be in place.

PS: you can already limit (and is a great best practice) access to the admin console, and enable users to only login via Enterprise Manager, assigning for example only the "restore operator" role to most of the users of Veeam.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
tfelice
Influencer
Posts: 10
Liked: never
Joined: May 23, 2012 12:28 pm
Full Name: Tony Felice
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by tfelice »

The password will be configured in the job, using the "credential manager" we already have in place
Luca, if what you mean is that when you create a job you set a "retention protection password" in the credential manager, and when anybody (even Administrator) wants to delete a non-expired restore point they are interactively challenged for that password, then yes, I think that would be very helpful. That would seem to prevent an intruder who happened to get Admin's credentials from deleting backups. And I personally like having repositories that are not accessible via the Windows domain credentials.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by veremin »

I think the mentioned approaches are unlikely to help, since having access to a backup console would mean that a malicious party can simply lower retention to 1, optionally exclude all disk from processing, and execute jobs. Once they're finished, a company will be left with no backups. Thanks.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by dellock6 »

Not if also the protected retention is protected by a second-level authentication. Something like sudo authorizations in linux and unix... :)
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by veremin »

Protect every single option in GUI sounds like a good idea, indeed. :)
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by dellock6 »

It's usually called an RBAC system (Role-based access control) :D

1. define roles
2. assign capabilities to roles
3. assign users and groups to roles
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by veremin »

If my memory serves me well, we were discussing issue with domain administrator credentials being compromised. Domain admin is still likely to be able to login into backup console, lower retention and leave you without backups. And “backup retention protection” would not be that helpful in that case. :D That was my point.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by dellock6 »

Account violation has nothing to do with an RBAC system, but more with the complexity of a password. And there's two-factor authentication to protect this.
If someone steals my remote control for my home alarm system, the fault is not in my alarm system.
And for best practices, none should NEVER login into a management console everytime with the highest available privileges, rather login first of all with a personal account instead of a general one ("who logged in with the administrator user???"), and with limited privileges, and then thanks to sudo-like commands be able to run additional commands, if authorized.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by veremin »

I do understand what you're saying, but I believe there is a sort of confusion between us, as we tend to discuss different things.

You're describing security model best practices, meanwhile, I'm trying to provide a reply to the Tony's post. Trying to say that a second authentication factor required to delete backups wouldn't be that helpful, since there still might be some loopholes , such as lowering retention, that malicious party can use, and protect every GUI option doesn't appear to be a good idea.

So, the reliable approach will be more about using Windows/network security best practices, like the ones described by you, rather than adding some stuff to the GUI.

Thanks.
tfelice
Influencer
Posts: 10
Liked: never
Joined: May 23, 2012 12:28 pm
Full Name: Tony Felice
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by tfelice »

Clearly adding protection based on retention schedule only helps if a bad actor can't just defeat it by changing the retention schedule. Lowering the number of retained backups is of course tantamount to deleting backups in an of itself. So role-based access-control should be designed carefully, thinking of use cases (and misuse/abuse cases) so that it really does effectively govern who can do what--including who can do what intentionally, accidentally, unwittingly, or maliciously.

In this sort of discussion my perspective is always heavily informed by my experience as a consultant. The issue I raise is not just a technical one. If I were dealing exclusively with competent systems administrators, I'd probably confine the discussion largely to best practices, security models, attack vectors, etc. But as a consultant, I help organizations with a wide range of IT expertise and an equally wide range in their willingness and ability to understand, let alone to follow, best practices. To put it another way, if I were Veeam's competition, I'd implement a solution to the problem I've posed and then brag about doing a better job of protecting my customers from themselves and their lack of deep technical knowledge. Of course, my marketing guys would not present it quite that way. They'd call it rock-solid protection that you don't need a PhD to implement.
meeyou
Influencer
Posts: 22
Liked: never
Joined: Oct 16, 2012 5:47 pm
Full Name: John White
Contact:

Re: Managing security in Veeam, read-only backup targets

Post by meeyou »

Gostev wrote:Tony, you have just explained the reason why large amount of companies still use tapes stored in a secure offsite location like Iron Mountain ;)
Actually, no amount of challenges in the B&R UI would help in your scenario, because having access to the Windows domain administrator credentials, you can simply logon directly to your backup repository boxes, and physically delete all backup files there. Thus, adding challenges to B&R UI is similar to reinforcing your main house door when you have no lock on your back door... no pun intended.
This is assuming that domain administrator accounts have access to the backup repositories...
Post Reply

Who is online

Users browsing this forum: No registered users and 186 guests