Comprehensive data protection for all workloads
frankj
Service Provider
Posts: 29
Liked: 2 times
Joined: May 27, 2016 4:53 pm
Full Name: FRANK Jacques
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by frankj » Mar 13, 2017 11:02 am

zfs wrote:Hello, I would suggest using ZFS or similar as underlying storage for your Veeam backups and have snapshotting/replication enabled. This saved me about a month back. I would even recommend that you have this kind of storage capabilities for your VM storage, because in that case you can probably do snapshotting more frequent than with Veeam since the snapshot is underneath the hypervisor layer and is done without any snapshotting overhead in the hypervisor.
do you have a setup for this you can explain ?

we have 2 fnas'es running for backups, but my plan was to take backup01 and have that rsyinc/ed snap'ed to backup02 on location 02..

Novox
Enthusiast
Posts: 40
Liked: 14 times
Joined: Jul 12, 2016 12:51 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Novox » Mar 13, 2017 1:11 pm

@netmask, @smartsys et al, I was advised by Veeam that if the backup repository is on a CIFS/SMB share, and the credentials are embedded in B&R (or the job), that during the job, the share is connected, the files are modified, and the share is disconnected. Even if the Veeam B&R server is infected, the CIFS/SMB backups shouldn't be. We have QNap NAS devices as repositories, and each have a single, local, user account for backups (known only to B&R and the NASs). They are not connected to AD.

Now, I believe a prior post indicated the OPs infection was perhaps listening for keystrokes for a while
by the looks of it, it seems like this particular nasty sits on your systems for a while, harvesting usernames/pws with key loggers and using a variety of other injected binaries - @lando_uk
and may have logged the keystrokes into Veeam for the repository account's username/password. That makes sense to me (an infected seperate process on the Veeam B&R server logging in to the share and wiping out backup files). Otherwise, as I've been advised, CIFS/SMB repositories (in this specific scenario) should be protected.

Can anyone confirm, or else explain where I'm wrong? (This post is not meant to indicate that offline backups aren't necessary.)

cbc-tgschultz
Enthusiast
Posts: 50
Liked: 10 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by cbc-tgschultz » Mar 13, 2017 2:06 pm

You should Always password protect the backup share with an account that only you and veeam B&R knows. I even do it this way at home.
Yeah, that's nice and all, but if Veeam is running on a Windows server in your domain and the malware can get to it with a properly credentialed account, it can read the veeam configuration and hose you anyway. Or even if it isn't on your domain, it could key log that information. In fact, even if your NAS has snapshot features, any kind of key logger that sits on an admin's system long enough will get credentials for that too and could wipe out your snapshots.

So: multiple online backups of any kind, even geographically dispersed, aren't going to help much.

Unfortunately, the best solution to this particular threat that I can see is rotating offline backups at a frequent interval, which is a pretty expensive, if not impossible, solution for those of us who have several terabytes of data per backup. Even that probably won't be safe for long, as attackers get more sophisticated it is entirely possible that they could subtly corrupt all backups over time until they are confident you have no good ones, or any good ones are too old to be very useful so you'll likely pay the ransom anyway. Of course, if you're testing backups frequently like you're supposed to, you might catch it in time, maybe. If they're corrupting the backups, they could potentially just corrupt certain parts of the backup instead, making it seem as though the target was restored properly when in reality the database data inside it is useless, for instance; though I admit that is pretty out there as far as probabilities go.

As much as I hate to say it, there are a lot of situations in this area where (probability * cost of problem) < (cost to fix problem), and I don't see that ever getting better.

It might be more worthwhile to try and eliminate the problem from the other end, that is, prevent ransomware in the first place. Hard as that is, there are some relatively easy steps that can help:

If you have Exchange or some similar product, you can create rules that match common phrases or dangerous words and attachments and force those emails to be read and approved for delivery by an administrator*. Oddly, Microsoft did not see fit to include a way in Exchange to filter for Office documents that contain macros, so unfortunately you have to assume ALL pre-2007 document formats contain macros. Worse, Office will happily open an *.RTF file that is, in fact, a Word file with a macro. If at all possible, you should probably just disable macro execution entirely in Office and apply that policy globally. I know, right? Good luck selling that to management.

(*From experience I can assure you that this gets tedious in a hurry, and worse, the fact that it is so tedious inclines whoever is approving messages to get lazy about it, defeating the purpose. It will never be perfect, but with enough tuning it can be worth the effort.)

Another strategy is to implement FSRM (File Server Resource Manager) on your Windows file servers to take action automatically in certain circumstances. For instance, my own implementation monitors for known file extensions of ransomware variants and runs a script when one is detected. This script emails the administrators and puts a deny entry for the offending user on all shares. Another similar policy monitors canary files placed strategically throughout the share drives for modification and performs the same lockout (note: this uses a scheduled task, not FSRM).

Of course, in an ideal world, we could Whitelist executables by hash using Software Restriction Policies. If you work in an environment where this is something you can actually get away with, then please tell me because I want a job that easy.

It seems to me that the future of IT is going to be decided soon because of Ransomware. Are we going to start taking security seriously as an industry, or are we going to continue blundering forward in a world where convenience almost always wins out and just start paying for Ransomware insurance the same way you pay for K&R insurance in some countries?

chillware
Influencer
Posts: 23
Liked: never
Joined: Apr 07, 2011 4:38 pm
Full Name: David McCoy
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by chillware » Mar 13, 2017 6:24 pm

cbc-tgschultz wrote:Another strategy is to implement FSRM (File Server Resource Manager) on your Windows file servers to take action automatically in certain circumstances. For instance, my own implementation monitors for known file extensions of ransomware variants and runs a script when one is detected. This script emails the administrators and puts a deny entry for the offending user on all shares. Another similar policy monitors canary files placed strategically throughout the share drives for modification and performs the same lockout (note: this uses a scheduled task, not FSRM).
FYI - For others, here is an awesome script that will do the hard work for you to get FSRM setup and denying infected users... https://fsrm.experiant.ca/

aporter
Novice
Posts: 8
Liked: 1 time
Joined: May 18, 2012 2:44 am
Full Name: Andrew Porter
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by aporter » Mar 14, 2017 6:09 am

One approach I haven't seen mentioned on this thread yet is to configure a pull using something like rsync, or perhaps another instance of Veeam, from your backup repository to another server which is on a different physical and logical network. This server could even be configured to power on to do the scheduled sync, and power off afterwards. This way nothing that is online would have access to the last resort archive server, but it would have access to pull from the primary backup repository (one way, read only).

I still think periodic rotating offline copies are best, but they will always require some manual rotation of drives/tapes, while the above workflow could get pretty close to automating more frequent offline copies.

The other scary factor to think about is offline backup retention periods. With ransomware that waits a long time before taking action, it is conceivable that you could rotate out all of your offline copies with infected backups over time, which could make recovery a lot more complicated.

Mike Resseler
Product Manager
Posts: 5662
Liked: 593 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Mike Resseler » Mar 14, 2017 6:16 am

Not a bad idea, but requires some automation / time / testing and so on.

I understand your scary factor, however, imagine that (in worst case) you realize that your backups have been infected because the ransomware was sitting there already for a long time. If you get infected, you could spinup a virtual lab and see / investigate where the infection is. That could probably help you in figuring out where the root cause is and eliminate that server from restoring it completely. While again taking time, at least you will be able to recover your data from 1 day ago where the infection might be on the server already, but the files are not encrypted yet.

mkaec
Expert
Posts: 318
Liked: 72 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by mkaec » Mar 14, 2017 12:42 pm

We used to have tape at all our sites and rely on local users to rotate them. I'm sure you can imagine how many backups were missed. We were so excited to move to disk based backup, which meant removal of the human factor. I never imagined it would have this kind of drawback.

cbc-tgschultz
Enthusiast
Posts: 50
Liked: 10 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by cbc-tgschultz » Mar 14, 2017 5:33 pm

One approach I haven't seen mentioned on this thread yet is to configure a pull using something like rsync, or perhaps another instance of Veeam, from your backup repository to another server which is on a different physical and logical network. This server could even be configured to power on to do the scheduled sync, and power off afterwards. This way nothing that is online would have access to the last resort archive server, but it would have access to pull from the primary backup repository (one way, read only).
Yeah, I had this same idea some time after posting the other day and am glad to see I'm not the only one thinking along these lines. It seems relatively straightforward and will accomplish 98% of the same purpose as tape (or similar) in this scenario with a lot less cost and hassle. Given that the other 2% is particularly unlikely yet significantly more labor intensive to mitigate, I think it's a good option.

A few things to keep in mind about it: Like your regular backups you'll need to check and make sure you can actually restore from them periodically, and ensure that they're actually being updated properly. Additionally, an absolutely critical component of this scheme is making sure that you never log into the system in a way that an attacker with a keylogger can get at it, no RDP, no SSH, none of that. Ideally it is on its own VLAN and can only talk to the file server it needs to talk to and only with an outbound-initiated connection, this has the added bonus that it is unlikely an attacker will ever even know it exists. Veeam B&R Server doesn't have a Linux version does it? Because ensuring that this system isn't infected by a usb device (probably used to move those backups around for testing, right?) would be a nice bonus. Actually, come to think of it, this is a scenario where you probably could use SRP to skin that cat. Finally, it would be prudent not to keep the credentials where you keep all your other credentials at work (we use an encrypted vault, for instance) because a keylogger will give the attacker direct access to that pretty easily. The last piece is pretty well mitigated away by the bit about access and firewalling, but no sense not taking such a simple step I think.

I haven't yet had time to investigate this setup very thoroughly, there could be pitfalls or other considerations I haven't accounted for, but so far I really like it as an alternative to putting tape back in the environment.

nitramd
Expert
Posts: 177
Liked: 35 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by nitramd » Mar 14, 2017 9:08 pm

mkaec wrote:We used to have tape at all our sites and rely on local users to rotate them. I'm sure you can imagine how many backups were missed. We were so excited to move to disk based backup, which meant removal of the human factor. I never imagined it would have this kind of drawback.
Ahh yes, I remember the human factor well. Years ago I recall DBAs getting angry with me after restoring tables for them on several occasions. Short story, the people responsible for changing tapes typically made mistakes when rotating them, regardless of how many times I explained the rotation scheme to them.

I'm starting to wonder if this is a case of "pick your poison".

yasuda
Enthusiast
Posts: 61
Liked: 10 times
Joined: May 15, 2014 3:29 pm
Full Name: Peter Yasuda
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by yasuda » Mar 14, 2017 10:29 pm

Mike Resseler wrote:...Samas is indeed a very painful variant of Ransomware. It used to target the healthcare vertical at first but it seems it is growing and attacking other verticals now also. It is known to search for backup files and basically can find backup files of most of the backup vendors and deletes them...
More bad news for Andrew, I am afraid: From what I read, Samas is introduced by targeted attack against network facing servers, following the usual techniques of getting access to login credentials, then island hopping and escalating access, passing hashes, etc. It's not automated, like phishing. Hospitals have a wealth of valuable data. And schools have student records and maybe even financial aid data. So my conclusion is the ransomware is a bonus; the real value is the data they exfiltrated before installing the ransomware. Really, given that you pwned the network, how could you not take the data? Unless you're running something like Splunk or AlienVault, you won't know data's being exfiltrated, unless, maybe, your ISP has records of bandwidth stats.

larry
Expert
Posts: 387
Liked: 92 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by larry » Mar 15, 2017 7:43 pm

Using an inheritable rights block with only the Veeam service account able to access the files would reduce the risk to the backup files from ransom ware. I think I would leave read rights for admins so they would know where the space is used but safer without. With the new gui there is no need to use the local account for day to day stuff. During Veeam install set the password and then never use it again, would still keep in case something went wrong.
I believe the real solution is offsite and offline and have not tested the above. I do use the blocks to keep admins out of some spots.

wasc
Service Provider
Posts: 21
Liked: never
Joined: Jan 11, 2012 4:22 pm
Full Name: Alex
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by wasc » Mar 16, 2017 3:14 pm

I suppose for me, the other issue is that I don't think you can truly protect a Veeam repository, without maybe Veeam introducing some new functionality.

For example we have a Veeam Repository which is sitting on a server *not* joined to the domain. In addition, the credentials veeam uses to connect to this server are denied rdp access. In theory, this server is partially air-gapped as an attacker would have no way of connecting onto it in order to encrypt or destroy the Veeam backups stored on this repository.

In reality, from the Veeam server itself you can simply Browse to Backup & Replication -> Backups -> Disk, select the repository and order the deletion of all data and the repository server will willingly do it. Maybe we need Veeam updated so you can put a password on the repository which needs to be entered before you can delete it via veeam? Similar to the ability where you can put a password on your anti-virus software to prevent tampering?

evander
Enthusiast
Posts: 67
Liked: 4 times
Joined: Nov 17, 2011 7:55 am
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by evander » Mar 17, 2017 9:28 am

I'm thinking if we can create a pre and/or post script as part of the Veeam backup job that will either boot or shutdown a VM hosting the repository or at a minimum enable/disable that VM's network card prior and after the job will run will give you a "sort-of" air gapped machine.
I'm working on the law of averages here in that assuming the ransomeware attack will occur on all desktop/servers at the same time and the red flag will be raised. The up-time of the Veeam repository will only be up during a certain portion of the night (usually) thereby minimising that attack vector. I fully understand that some ransomeware are intelligent enough to wait and attack on cue but that will need to be mitigated in another way. You can perhaps do what others have suggested by scanning for ANY other file types copied to the repository server other than VBKs etc. and then act accordingly.

What I am thinking is that even if I backup to such a repository once a week I will still be "happy" to lose a weeks worth of data as opposed to losing everything forever!
So this would mean that in an average week the server will be "up" a small percentage of the time. If you make that a month the average gets even smaller. Obviously the up-time will depend on the time for your own individual backups to run but Veeam does a great job in making that a quick as possible already.
If you want you can backup every night to such a repository and to another similar protected repository (different server) once a week or month or whatever.
At this point we need to protect everything and settle for whatever we can recover. Its a sad fact really.

Basically, what I am saying is I want to use technology to create an air-gapped server. I'm not saying it will fix but I do reckon it will help, and that's a step forward.

Thoughts?

Mike Resseler
Product Manager
Posts: 5662
Liked: 593 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Mike Resseler » Mar 17, 2017 10:51 am

I certainly read a few interesting ideas. But probably (as always) it will be a combination of different technologies.

@wasc: I am not sure if your thought on deleting backups from disk is realistic for malware. That would mean a specific malware, running specific Veeam PowerShell cmdlets would need to exist. That would limit the attack vector for such an investment a lot. But you never know of course

@evander: We have the possibility in VEB (soon VAW) to "eject" the removable media after a backup. The question is, do you really want to do this with your production server? In the end, your suggestion to enable / disable a NIC will probably not work automatically since you can't remotely switch something on if there is no network :-).

Anyway, certainly very interesting conversation and really like to read ideas, how people are doing it today (or going to do it). I think this topic is helpful for many!

Keep ideas coming

evander
Enthusiast
Posts: 67
Liked: 4 times
Joined: Nov 17, 2011 7:55 am
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by evander » Mar 17, 2017 11:30 am

Hi Mike - my thinking would not be to turn on/off the NIC at the OS level but rather at the VMware (or HyperV) level for the specific virtual machine.
Its a small check box in VMware vCenter to do that so I'm sure there must be a way to do it via command line and hence via a script.

I would be more than happy to disconnect my Veeam repository from the production network (since that is its only purpose, as a Veeam repository) after every backup job has run. If its not on the network it cant be attacked in theory, unless the attack is via the Vsphere ESXi level which currently, as far as I know, has not been attacked.

evander
Enthusiast
Posts: 67
Liked: 4 times
Joined: Nov 17, 2011 7:55 am
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by evander » Mar 17, 2017 11:52 am

Thinking out loud a bit further - possibly a script that would change NTFS permissions on a file or folder that held VBKs pre and post Veeam backup job.
I'm assuming the ransomware would need at least write/modify NTFS permission on the file to encrypt it right? So if 90% of the time that file/folder has NTFS permissions restricted to "read only" then its another layer of protection. I understand the ransomware might be smart enough to change NTFS permissions itself if its running under an admin account but possibly that can be mitigated by some way I haven't thought of yet.

With the power of Veeam my entire organization can be backed up into a handful of files or everything under one top level folder, If we put our minds to it I'm sure we can find a way to protect that folder so that only Veeam can make changes to it, or at a minimum changes can only be made to it only during a small and specific time period.

tacioandrade
Enthusiast
Posts: 29
Liked: 3 times
Joined: Nov 17, 2016 2:04 am
Full Name: Tácio Andrade
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by tacioandrade » Mar 20, 2017 2:29 am

Great idea really, I had not thought of it then .... In case the problem would be only in the part of overwriting the old backup to remove the new one, however it really is a great idea to start working on.

Sincerely, Tácio Andrade.

TrevorBell
Expert
Posts: 353
Liked: 7 times
Joined: Feb 13, 2009 10:13 am
Full Name: Trevor Bell
Location: Worcester UK
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by TrevorBell » Mar 20, 2017 1:30 pm

Hi All,

Just to quick post that something similar happened to me at a previous company... ill share some facts what happened in that scenario.
Had a call one night a director couldn't get email. Then had call from vendor who supports 24/7 infrastructure saying he can see SQL servers being encrypted.
By the time I logged in via VPN 43 servers were hit and 110 use laptops / desktops.
SAMAS was the cause... Now lets look at how it happened..

RRAS server was still being used and was bruteforced as it was a weak password which allowed hackers in and that hacked AD to make an account called administrators. Hackers were in the system for approx. 6 days before to attack took place hence if you call your server VEEAM or VBK etc its easy to locate it - remember its easy for other to identify too. The RRAS server Ip was published on showdan.io and 4 days after it appeared on there as having an RRAS server it was hit.

I had offsite tape backups and recalled them from Iron Mountain and started restore process. 10TB of data wasn't too bad. But the board of the company took the decision it was easier to pay the £15,000 in bitcoins than wait for all laptops to recovered by IT so it was paid and the encryption keys obtained, yes not ideal but in a large PLC it was small change.

There are many ways to hide the server and use other techniques but security starts with every admin. Files don't just disappear with SAMAS they get deleted and then encryption to stop restores being possible ..

We actually caught the SAMAS strain and passed to Symantec and FBI as it wasn't known the strain back in October last year.
We also re-wrote the DEC2.EXE decrypt exe from 3 passes at decoding - it only needed 1 pass speeding the de-crypt process up ( not that the other 2 passes did anything except waste time )

Indeed SSL VPN was installed, yes user account and the lazy IT folks who used the same passwords for both user and their domain admins got educated.

People need to ( if you don't already ) pay for a pen test and see the amount of vulnerabilities it comes back with.. in this case it was over 6,000 redial actions from VMware exploits, windows update exploit, java exploits even Symantec AV Client exploits... IIS exploits SSL exploits even if you think your infrastructure is upto date its worth calling a pen tester in for a day or two... might be the best thing you do this year :)

Thanks

Trev..

cbc-tgschultz
Enthusiast
Posts: 50
Liked: 10 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by cbc-tgschultz » Mar 20, 2017 2:21 pm 2 people like this post

I disagree with the idea of paying for a Pen Test. Here's why: They will always find stuff. Always. And you'll fix it, and then you'll get another Pen Test a while later and they'll find more stuff. Always. And you'll fix it and pay for another Pen Test...

See, the problem is that the industry hasn't taken security seriously... basically ever. It will continue to not take security seriously for the foreseeable future. You don't have any control over most of the software you use in your business, but even if you did it is unlikely you'd ever find and patch all the vulnerabilities and new ones would be introduced with every new feature or update you do, as they are with third party software, because the industry does not take security seriously.

And that's largely because human beings do not take security seriously, because we're just naturally awful at risk assessment. But I digress.

Point being, it is much, much cheaper to, instead of paying for endless Pen Tests, just assume everything is vulnerable, always, and design your practices around that. Assume that someone will break into any and all public facing services, ok, now where can they go from there? Assume admin credentials will be stolen, what can we do about that? Assuming the worst happens, what's our plan for dealing with it? How could that be compromised? Etc. You can't buy security, because it is a mindset.

Ultimately, this particular threat is never going to go away. This is obvious from statements like "But the board of the company took the decision it was easier to pay the £15,000 in bitcoins than wait for all laptops to recovered by IT", so look forward to the bright future of IT where your budget includes K&R insurance. It's easier than security anyway.

Tonksy
Lurker
Posts: 1
Liked: never
Joined: Mar 23, 2017 7:08 am
Full Name: Mike Tonks
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Tonksy » Mar 23, 2017 7:21 am

Using Veeam with a NAS presenting CIFS, NFS, REFS etc exposes the shares. ExaGrid has Veeams own data mover software within appliances adding addtional protection from ransomware.

Insulation from Ransomware
When ransomware strikes, it is critical to have backups insulated from the malicious encryption/damage since they may be your last line of defense. ExaGrid helps insulate backups in the following ways:

1. Comprehensive access security ExaGrid shares can be accessed only from designated backup/media servers. While those severs may also be subject to rampant ransomware, the fewer servers that have access to your backups, the better.

2. SMB signing can be enabled for ExaGrid shares, requiring Windows account credentials to be authenticated and authorized before access is granted to an ExaGrid share, further reducing the chance of malicious access to backups.

3 Veeam Accelerated Data Mover shares require a separate Veeam password and are accessible only via SSH, which also reduces the chance of malicious access to Veeam backups.

4 All accounts used to manage the ExaGrid software are protected using non-default passwords. This includes thebackup “admin” account, the special ExaGrid customer support account, and root access.

5 ExaGrid software is updated at least quarterly with the latest appropriate CVE fixes, reducing the ways ransomware can gain access to ExaGrid servers. Software may be updated more frequently as dictated by CVE severity.

6 Each ExaGrid server runs a proper firewall and a customized Linux distribution that opens just the ports and runs just the services necessary for receiving backups, web-based GUI, and ExaGrid-to-ExaGrid replication.

4. Communications between ExaGrid servers is secured using Kerberos authorization and authentication, protecting from a“man in the middle” attack from malicious users or software.

Finally
Periodic Assessments Using Beyond Trust’s Retina Network Vulnerability ScannerA complete vulnerability assessment is run periodically against ExaGrid’s software using the Retina Network Vulnerability Scanner.This tool is the security industry’s most respected and validated vulnerability assessment tool. Audit risks identified by Retina are
evaluated by ExaGrid engineering and appropriate resolution is applied.

frankj
Service Provider
Posts: 29
Liked: 2 times
Joined: May 27, 2016 4:53 pm
Full Name: FRANK Jacques
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by frankj » Mar 23, 2017 11:08 am

Woudl this be a safe bet ?

Tape autoloader for our 7 TB prod store on veeam

24 tapes rounded in 8 sets of 3 rotated weekly.

giving us a buffer of 8 weeks of tape backup, with a possibility of 1 week old data to 8 weeks if all where corrupted but the last one.

Would an autoloader be safe for air gap ?

nitramd
Expert
Posts: 177
Liked: 35 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by nitramd » Mar 23, 2017 2:45 pm

With all of the good ideas that the community has proposed, has anyone factored in how RTO would be impacted?

For example, if one decides to rename file extensions of the backup files (pre/post script), how long would it take to recover a file/VM? This might also apply to an air gapped server.

cbc-tgschultz
Enthusiast
Posts: 50
Liked: 10 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by cbc-tgschultz » Mar 23, 2017 2:49 pm

Well of course having to restore from an air-gapped source will add to your RTO, but that's a worst-case scenario where your RTO would otherwise be infinity. If the company you work for is the kind that will pay the ransom rather than wait, then it may not be worth your effort.

nitramd
Expert
Posts: 177
Liked: 35 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by nitramd » Mar 23, 2017 2:55 pm

I don't disagree with what you're saying. However, if you've gone on record saying that your RTO will be 10 minutes, or whatever value, and you have not updated that estimate/guarantee since the advent of ransomware then your you may have put yourself in a sling with management.

yasuda
Enthusiast
Posts: 61
Liked: 10 times
Joined: May 15, 2014 3:29 pm
Full Name: Peter Yasuda
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by yasuda » Mar 23, 2017 3:51 pm

cbc-tgschultz wrote:Well of course having to restore from an air-gapped source will add to your RTO, but that's a worst-case scenario where your RTO would otherwise be infinity. If the company you work for is the kind that will pay the ransom rather than wait, then it may not be worth your effort.
Except paying the ransom is no guarantee you will be able to decrypt your data.

cbc-tgschultz
Enthusiast
Posts: 50
Liked: 10 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by cbc-tgschultz » Mar 23, 2017 4:07 pm

In general it is in the best interests of the ransoming parties to uphold their end of the deal, and I've only ever heard of scattered cases where that hasn't happened. It's a risk, to be sure, because you're dealing with unscrupulous individuals after all, but it isn't much of one.

Personally, I know that my company would only consider it as a last resort, so I just have to worry about how to get us to a point where we can survive the kind of worst-case scenario being described here, not how long it is going to take to recover or even so much what the RPO is. For others, well, we've already had someone comment that their company decided to pay the ransom to recover laptops rather than wait on them to be restored from the backup data they had.

yasuda
Enthusiast
Posts: 61
Liked: 10 times
Joined: May 15, 2014 3:29 pm
Full Name: Peter Yasuda
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by yasuda » Mar 23, 2017 4:18 pm

frankj wrote:Woudl this be a safe bet ?
Tape autoloader for our 7 TB prod store on veeam
24 tapes rounded in 8 sets of 3 rotated weekly.
giving us a buffer of 8 weeks of tape backup, with a possibility of 1 week old data to 8 weeks if all where corrupted but the last one.
Would an autoloader be safe for air gap ?
Do the tapes stay in the autoloader? If so, then in the original poster's scenario, no. In that case, the attacker had remote admin access, and, assuming your autoloader is not managed by an air gapped server, it could be commanded to overwrite tapes with garbage.

If you could air gap the tape management server, and still get data to the tapes, that would be good. An interesting architectural challenge.

I think a cloud backup service like Carbonite or Crashplan would be very good for protecting your file data from ransomware, although not perfect. Barracuda Backup Server is probably the best single solution for ransomware protection plus image backup, because it replicates deduped backups off site, and it is managed through a cloud service so you never need to log onto it from the local network.

I don't think there is a really great solution for the scenario where an intruder has undetected access to your network for an extended period of time, because given that, you can imaging Mr Robot scenarios where nothing is safe. Really, your focus should be on putting systems and processes into place to give you a high probability of detecting the breach.

I also think it would be more productive to discuss methods of protecting your backups, where we assume automated ransomware (no active command and control) or that the breach will be detected before the attacker gains unlimited root access to everything. Physical locks are rated in terms of how long they will delay an attacker, and we should rate backup protection the same way. How long will it take a determined attacker to destroy your backups?

yasuda
Enthusiast
Posts: 61
Liked: 10 times
Joined: May 15, 2014 3:29 pm
Full Name: Peter Yasuda
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by yasuda » Mar 23, 2017 4:34 pm

cbc-tgschultz wrote:In general it is in the best interests of the ransoming parties to uphold their end of the deal, and I've only ever heard of scattered cases where that hasn't happened. It's a risk, to be sure, because you're dealing with unscrupulous individuals after all, but it isn't much of one.
Sure, but if you're not considering low probability events, why are you reading the thread?

I think I missed your original point, however, if it was that you need to do the cost to benefit analysis, and I completely agree with that.

And set priorities. If you're not confident a breach will be detected in less than the average 100+ days, address that first. And are you even testing your backups? If you're not running SureBackup tests, ransomware should be a lower priority. Do you test tapes when they rotate back from off site storage?

nitramd
Expert
Posts: 177
Liked: 35 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by nitramd » Mar 23, 2017 4:37 pm

yasuda wrote: Do the tapes stay in the autoloader?
Yes, at least the ones I've used.
yasuda wrote: I don't think there is a really great solution for the scenario where an intruder has undetected access to your network for an extended period of time...
Not completely true. I've heard from Justice Department officials that there are examples of companies who have been unaware of breaches for years; they did not give the number of companies in this category, they did not name names nor give the exact amount of time that elapsed before discovery.

Although, Yahoo does come to mind.

yasuda
Enthusiast
Posts: 61
Liked: 10 times
Joined: May 15, 2014 3:29 pm
Full Name: Peter Yasuda
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by yasuda » Mar 23, 2017 4:44 pm

Random thought about sending tapes off site: If I'm in your network, maybe I've found email or documents, maybe I've found a copy of your DR plan in your wiki. So I call Iron Mountain and say, "We need all our backup tapes delivered to our DR site ASAP! Here's the address..." And for a reasonable fee, not only will I send you the key to decrypt your data, I will mail your tapes to you.

Plausible?

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 9 guests