Suggestions to improve Veeam security

[dcolpitts]

Recently, I've been dealing with the after affects and cleanup of an attack on a non-client site (I was hired after the attack to help with the remediation, recovery, and prevention of future attacks). The attackers broken via an exposed RDP server, managed to obtain domain admin access, changed all the domain admin account passwords, secure wiped all the disk based backups, and then proceeded to encrypt the remaining data across the WAN. In all fairness here, the site I was hired to help after the fact was NOT a Veeam customer either, they did not follow 3-2-1 rules, or a bunch of other things that are basic security no-brainers that we all take for granted. This still got me thinking more and more about security in depth and things that we collectively could do to make it more difficult for rogue admins and other attackers to destroy our data and our companies. When I look at Veeam, I see a few things I would like to see implemented to bolster our backup security, and hopefully some if not all of these changes can be implemented.

We know most ransomware and data destruction attacks happen on a weekend, when no one is around at smaller organizations that do not have dedicated NOCs or staff to monitor systems 7/24. To this effect, I'd like to see something similar to HPE's Primera Virtual Lock be implement in Veeam. Primera's Virtual Lock feature enforces the retention period of any volume or copy of a volume. This prevents the volume from being deleted, intentionally or intentionally, before the retention period elapses. Locked volumes cannot be deleted, not even by a storage administrator with the highest level of privileges. Lock volumes can not be overwritten. It's basically tamperproof. If implemented in Veeam, it would mean we could theoretically set a 96+ hrs virtual lock on our replicas, storage snapshots, and backups (I'm specifically thinking of D2D systems such as HPE StoreOnce Catalyst stores here which is only accessible via API and not via the filesystem), and even if an attacker got in on Friday night, Thursday night's backups would persist for at least 96 hours and the attacker would not be able to delete them via the Veeam interface (they would have to go through the storage or backup system's UI or API to do so, which would be much more difficult in the case of a HPE StoreOnce Catalyst repo for example). If Virtual Lock is enabled, then it should be impossible for the number of restore points on disk to be modified as a result of that that either. If an admin enables Virtual Lock - maybe it needs to be a one way street and the only way to disable it is to call support for a time based password (much like HPE does for the 3Par de-installation password or the StoreOnce support password), and even then there needs be a 72hr/96hr callback window from Veeam support to the licensing admin only or something (attackers are getting pretty crafty at spoofing users on the phone - Apple currently does this for accounts with lost credentials too). That of course would need to be clearly speed out in the UI that the admin has to go through to enable it to begin with though.

I would like to see automatic email notifications (think auditing notification) when certain settings are changed. For example any changes to any SMTP settings (maybe make them be confirmed via pin sent to email too). So for example, maybe the attackers change the SMTP server to a non-existing server so the IT organization no longer gets email alerts, or deletion of backups, replicas, storage snapshots, or the number of restore points for a given job (which Virtual Lock should also block being changed) - an email notification needs to be sent to original SMTP server (if it was being changed). I'd also like to see the ability added natively to Veeam to send an email alert when the Veeam services start and stop (I do miss this setting after the migration from Backup Exec). Veeam should also send email notifications for any restores (even if the user starts a guest file or application item restore and mounts the restore, but doesn't actually restore anything - a notification still needs sent that it happened)! And I'd like to see the ability for Veeam to send an email alert when job settings are changed, or the system configuration is restored (send it right before the config restore please!).

I'd like to see the option to decouple Veeam's administrative login from Active Directory and let me create my own, non-AD enabled users and assign them roles. If an attacker can get Domain Admin access (which is generally easy if they can get a foot hold on a domain joined machine), then they can pretty much grant themselves Veeam administrator access as well. And while it *may* be possible to replace the users / credentials table in SQL with the attacker's own copy with their own non-AD enabled credentials (maybe we need unique public key / private key signing solutions on the credentials table too - I don't know - I'm just talking out loud here), I'm going to know the Veeam services were restarted (for the new credentials to take effect) via email (see previous suggestion) - and this may not stop the attack right then and there, but it if happens on Friday night I'm going to know about it Saturday morning when I get out of bed rather than finding out on Monday morning as staff roll in after the long weekend to find no data left!

Maybe I'm just being too paranoid - but at the end of the day, we rely on Veeam as one of our last defenses against an attacker that is determined to destroy our data for profit or revenge. I believe these suggestions will help better harden Veeam and the IT organizations that rely on Veeam to fend off these attacks, and I look forward to any feedback others may have or wish to contribute to this thread. I certainly hope Gostev will embrace this thread with open arms and make careful consideration of the suggestions here.

Thanks!

dcc

[HannesK]

Hello,
there are many different topics in your question which makes it quite difficult to answer.

First I always like to refer to https://www.veeambp.com/infrastructure_hardening as it covers many topics.

For VirtualLock: I just read the user guide and I see no problem that you use it today for your Veeam repository. As an alternative we support S3 object lock out of the box (Amazon and other vendors support it).

I see no way how replicas could be protected. They are "just" VMs and if an attacker has access to your VCenter, then he can delete the VMs.

For the email settings: I suggest to use the Veeam ONE change tracking report. All the notifications you ask for would make the UI pretty complicated and an attacker could just block the connection to the email server (Windows firewall...)

AD: I'm not sure what you mean. AD users are not required for Veeam in any case except for application aware backup of domain controllers (forum search has some hints on that).

I always opt for "keep it simple" and applying the Microsoft hardening guides. Proper hardening implemented, chances for non-insider attacks are reduced to a minimum.

Best regards,
Hannes

[dcolpitts]

Interesting... I hadn't see that website before - I will take some and go through it.

Virtual Lock - only HPE Primera currently has this feature. I've been asking if there are plans to bring it to the previous generation 3Par, Nimble, MSA, and StoreOnce, but have yet to get an answer from anyone at HPE. While I agree with your comment on Replicas, however if VC is not integrated with AD, then that adds an extra challenge to the attacker (to gain access to VC and / or the ESXi console). But that is a moot point if the attacker can access Veeam and delete the replicas from there - thus the reason I suggest a virtual lock on the backups from a Veeam perspective. At the end of the day, if the attacker can access the underlying file system, it's game over anyways - but for something like 3Par, Nimble, or StoreOnce, it's not possible to access the filesystem - just the UI, which can be secured. Now if we are talking a NAS - that's a different story... The bulk of the storage systems out there will never get a feature like Virtual Lock, but if Veeam could add it....

Email settings - VeeamOne is insufficient for what I am asking for, nor does it reactively alert on restores and such. I'm sure if Veritas was smart enough to figure out how to handle notifications within their application, Veeam can easily handle it. It just becomes an additional tab in settings where you click a series of check boxes for what you want to alert on. And while you are correct the attacker could block it, they also may not block it... The SMTP notifications are still important and have a role outside of alerting of a attacker in daily ops.

AD - let me rephrase, and you can correct me if I am wrong... By default, users in the local Administrators group are Veeam Backup Administrators. So Veeam relies on Windows users accounts (which are typically AD) for access to the console. I'm asking for a way to de-couple this and use only Veeam managed credentials independent of the OS. Then I can create separate credentials (in Veeam) to access the Veeam console. This way if an attacker gains access to the VBR Windows server or AD, they still don't have access to the VBR console (which protects my replicas above). Nor would they have access to VC (assuming it's not AD integrated), nor would they have access to the storage systems (again assuming it's not AD integrated). In each case separate, strong credentials would protect the respective consoles.

dcc

[HannesK]

Hello,
welcome. I also recommend the user guide security section: https://helpcenter.veeam.com/docs/backu ... ml?ver=100

referring to the user guide: if you lost the backup server, then the data is lost without air-gapped / offline / immutable backups. With every backup software I can dump all passwords I need from the backup software if I'm a local administrator (in worst case directly from RAM). It's not Veeam specific. That also covers "Veeam native" authentication. There is no way to protect against a local admin.

Today's malware does a call-home as soon as it exploited a backup server. Then a human is taking over.

I also recommend using forum search. Many ideas have already been discussed on this topic.

Best regards,
Hannes