Just because you're paranoid ...

[c0utta]

You know the rest of this homily I'm sure...

A number of my peers have been crypto-lockered to the point where their business has ground to a halt for weeks until they recovered. Their recovery was usually some fluke where they found backups they didn't know they had. Scary stuff indeed, so I thought I'd like to start a discussion about how far "should" we all go to protect ourselves without being completely paranoid. My company has undertaken a number of security reviews and we've identified gaps that we've filled. I've also come up with some further protection mechanisms, but I don't want to be complacent about this changing threat.

Our environment is ESX based with about 75 guests with a total Veeam backup size of about 15TB - not large by some standards. We maintain an onsite DR, together with a offsite backup repository.

In onsite DR we have a physical Veeam server (on the domain), and a backup repository that is Windows-based (not on the domain). We also have a tape drive in the same location, together with a Linux NAS that has both ssh and network interface disabled (more on this later).

Offsite backup repository is Windows-based (not on the domain) and receives backup copy jobs.

Best practice seems to be having repositories off the domain, with the credentials stored within Veeam. Is this still best practice and can these credentials be secured even further? Have there been any instances where these credentials have been compromised?

Every week we do a full backup of environment, write it to the onsite DR repository and back it up to tape (ejected of course). This backup is shared read-only to a single user, and we also copy other important documentation into the backup location, together with vmdk's of a DC and an emergency Veeam server.

Every weekend the Linux NAS brings up the network interface, mounts the share, copies the contents of the full backup and disables the network interface. This means if things go pear-shaped we can physically get to our Linux NAS (we call it the "go bag") and have a copy of our data that is (at worst) a week old. Once we have an ESX host, we can load the DC and Veeam and start restoration.

We think we're following Veeam best practice, but are there any holes we can plug further? Is the Linux NAS concept complete overkill?

Any contribution greatly received.

[IanJ]

I have had it twice (last time was a few years ago) and for us Veeam wasn't part of the solution, however it you are taking replica's of your file server then it could be.

First thing is catching it quickly. We caught it both times (as in saw it happening) within an hour. We push all traffic through an internal firewall so we could quickly determine which machine was infected and encrypting everything and then block it from talking to the network.

If you are using a SAN or some sort of NAS that can take storage level snapshots then this is the easiest way to recover. We took 4 hourly snapshots with a retention of 3 days.

Recovery was then just mounting the unaffected snapshot on to the same file server and then running a script that found all the encrypted files (ours were helpfully name .encrypted) and copy the unaffected version back.

In terms of best practice, its more whats right for you. So that one is hard to answer. If your Veeam servers and datastores are segregated from the rest of the network and you are using credentials that are specifically and only for Veeam and you aren't doing silly things on your servers infrastructure (like downloading and running unknown files) then there shouldn't be any reason they can't be on the domain.

In the cases we have had ransomware it basically attacks as the current user that got infected, so your users shouldn't have access to your Veeam repositories to encrypted them, and the Veeam credentials should never be running anything but Veeam services so any ransomware will never run in the Veeam credentials user context (so wont have access to start encrypting your Veeam backups). This is a fairly large discussion on ICT security and policy (for example do you let your ICT log into machines as their privileged accounts, do you have role based access on all your staff with least privileged concept being used etc).

However, from how you describe your setup, without actually seeing it, I would say your backups are pretty safe!

[nitramd]

Is it paranoia or is it preparedness?

I suggest, and vote for, the latter.