Yes, Ransomware can delete your Veeam backups.

Availability for the Always-On Enterprise

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby frankj » Mon Mar 13, 2017 11:02 am

zfs wrote:Hello, I would suggest using ZFS or similar as underlying storage for your Veeam backups and have snapshotting/replication enabled. This saved me about a month back. I would even recommend that you have this kind of storage capabilities for your VM storage, because in that case you can probably do snapshotting more frequent than with Veeam since the snapshot is underneath the hypervisor layer and is done without any snapshotting overhead in the hypervisor.

do you have a setup for this you can explain ?

we have 2 fnas'es running for backups, but my plan was to take backup01 and have that rsyinc/ed snap'ed to backup02 on location 02..
frankj
Service Provider
 
Posts: 20
Liked: 1 time
Joined: Fri May 27, 2016 4:53 pm
Full Name: FRANK Jacques

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Novox » Mon Mar 13, 2017 1:11 pm

@netmask, @smartsys et al, I was advised by Veeam that if the backup repository is on a CIFS/SMB share, and the credentials are embedded in B&R (or the job), that during the job, the share is connected, the files are modified, and the share is disconnected. Even if the Veeam B&R server is infected, the CIFS/SMB backups shouldn't be. We have QNap NAS devices as repositories, and each have a single, local, user account for backups (known only to B&R and the NASs). They are not connected to AD.

Now, I believe a prior post indicated the OPs infection was perhaps listening for keystrokes for a while

by the looks of it, it seems like this particular nasty sits on your systems for a while, harvesting usernames/pws with key loggers and using a variety of other injected binaries - @lando_uk


and may have logged the keystrokes into Veeam for the repository account's username/password. That makes sense to me (an infected seperate process on the Veeam B&R server logging in to the share and wiping out backup files). Otherwise, as I've been advised, CIFS/SMB repositories (in this specific scenario) should be protected.

Can anyone confirm, or else explain where I'm wrong? (This post is not meant to indicate that offline backups aren't necessary.)
Novox
Influencer
 
Posts: 11
Liked: never
Joined: Tue Jul 12, 2016 12:51 pm

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby cbc-tgschultz » Mon Mar 13, 2017 2:06 pm

You should Always password protect the backup share with an account that only you and veeam B&R knows. I even do it this way at home.


Yeah, that's nice and all, but if Veeam is running on a Windows server in your domain and the malware can get to it with a properly credentialed account, it can read the veeam configuration and hose you anyway. Or even if it isn't on your domain, it could key log that information. In fact, even if your NAS has snapshot features, any kind of key logger that sits on an admin's system long enough will get credentials for that too and could wipe out your snapshots.

So: multiple online backups of any kind, even geographically dispersed, aren't going to help much.

Unfortunately, the best solution to this particular threat that I can see is rotating offline backups at a frequent interval, which is a pretty expensive, if not impossible, solution for those of us who have several terabytes of data per backup. Even that probably won't be safe for long, as attackers get more sophisticated it is entirely possible that they could subtly corrupt all backups over time until they are confident you have no good ones, or any good ones are too old to be very useful so you'll likely pay the ransom anyway. Of course, if you're testing backups frequently like you're supposed to, you might catch it in time, maybe. If they're corrupting the backups, they could potentially just corrupt certain parts of the backup instead, making it seem as though the target was restored properly when in reality the database data inside it is useless, for instance; though I admit that is pretty out there as far as probabilities go.

As much as I hate to say it, there are a lot of situations in this area where (probability * cost of problem) < (cost to fix problem), and I don't see that ever getting better.

It might be more worthwhile to try and eliminate the problem from the other end, that is, prevent ransomware in the first place. Hard as that is, there are some relatively easy steps that can help:

If you have Exchange or some similar product, you can create rules that match common phrases or dangerous words and attachments and force those emails to be read and approved for delivery by an administrator*. Oddly, Microsoft did not see fit to include a way in Exchange to filter for Office documents that contain macros, so unfortunately you have to assume ALL pre-2007 document formats contain macros. Worse, Office will happily open an *.RTF file that is, in fact, a Word file with a macro. If at all possible, you should probably just disable macro execution entirely in Office and apply that policy globally. I know, right? Good luck selling that to management.

(*From experience I can assure you that this gets tedious in a hurry, and worse, the fact that it is so tedious inclines whoever is approving messages to get lazy about it, defeating the purpose. It will never be perfect, but with enough tuning it can be worth the effort.)

Another strategy is to implement FSRM (File Server Resource Manager) on your Windows file servers to take action automatically in certain circumstances. For instance, my own implementation monitors for known file extensions of ransomware variants and runs a script when one is detected. This script emails the administrators and puts a deny entry for the offending user on all shares. Another similar policy monitors canary files placed strategically throughout the share drives for modification and performs the same lockout (note: this uses a scheduled task, not FSRM).

Of course, in an ideal world, we could Whitelist executables by hash using Software Restriction Policies. If you work in an environment where this is something you can actually get away with, then please tell me because I want a job that easy.

It seems to me that the future of IT is going to be decided soon because of Ransomware. Are we going to start taking security seriously as an industry, or are we going to continue blundering forward in a world where convenience almost always wins out and just start paying for Ransomware insurance the same way you pay for K&R insurance in some countries?
cbc-tgschultz
Enthusiast
 
Posts: 46
Liked: 9 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby chillware » Mon Mar 13, 2017 6:24 pm

cbc-tgschultz wrote:Another strategy is to implement FSRM (File Server Resource Manager) on your Windows file servers to take action automatically in certain circumstances. For instance, my own implementation monitors for known file extensions of ransomware variants and runs a script when one is detected. This script emails the administrators and puts a deny entry for the offending user on all shares. Another similar policy monitors canary files placed strategically throughout the share drives for modification and performs the same lockout (note: this uses a scheduled task, not FSRM).


FYI - For others, here is an awesome script that will do the hard work for you to get FSRM setup and denying infected users... https://fsrm.experiant.ca/
chillware
Influencer
 
Posts: 23
Liked: never
Joined: Thu Apr 07, 2011 4:38 pm
Full Name: David McCoy

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby aporter » Tue Mar 14, 2017 6:09 am

One approach I haven't seen mentioned on this thread yet is to configure a pull using something like rsync, or perhaps another instance of Veeam, from your backup repository to another server which is on a different physical and logical network. This server could even be configured to power on to do the scheduled sync, and power off afterwards. This way nothing that is online would have access to the last resort archive server, but it would have access to pull from the primary backup repository (one way, read only).

I still think periodic rotating offline copies are best, but they will always require some manual rotation of drives/tapes, while the above workflow could get pretty close to automating more frequent offline copies.

The other scary factor to think about is offline backup retention periods. With ransomware that waits a long time before taking action, it is conceivable that you could rotate out all of your offline copies with infected backups over time, which could make recovery a lot more complicated.
aporter
Novice
 
Posts: 8
Liked: 1 time
Joined: Fri May 18, 2012 2:44 am
Full Name: Andrew Porter

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Mike Resseler » Tue Mar 14, 2017 6:16 am

Not a bad idea, but requires some automation / time / testing and so on.

I understand your scary factor, however, imagine that (in worst case) you realize that your backups have been infected because the ransomware was sitting there already for a long time. If you get infected, you could spinup a virtual lab and see / investigate where the infection is. That could probably help you in figuring out where the root cause is and eliminate that server from restoring it completely. While again taking time, at least you will be able to recover your data from 1 day ago where the infection might be on the server already, but the files are not encrypted yet.
Mike Resseler
Veeam Software
 
Posts: 3281
Liked: 365 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby mkaec » Tue Mar 14, 2017 12:42 pm

We used to have tape at all our sites and rely on local users to rotate them. I'm sure you can imagine how many backups were missed. We were so excited to move to disk based backup, which meant removal of the human factor. I never imagined it would have this kind of drawback.
mkaec
Expert
 
Posts: 185
Liked: 48 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby cbc-tgschultz » Tue Mar 14, 2017 5:33 pm

One approach I haven't seen mentioned on this thread yet is to configure a pull using something like rsync, or perhaps another instance of Veeam, from your backup repository to another server which is on a different physical and logical network. This server could even be configured to power on to do the scheduled sync, and power off afterwards. This way nothing that is online would have access to the last resort archive server, but it would have access to pull from the primary backup repository (one way, read only).


Yeah, I had this same idea some time after posting the other day and am glad to see I'm not the only one thinking along these lines. It seems relatively straightforward and will accomplish 98% of the same purpose as tape (or similar) in this scenario with a lot less cost and hassle. Given that the other 2% is particularly unlikely yet significantly more labor intensive to mitigate, I think it's a good option.

A few things to keep in mind about it: Like your regular backups you'll need to check and make sure you can actually restore from them periodically, and ensure that they're actually being updated properly. Additionally, an absolutely critical component of this scheme is making sure that you never log into the system in a way that an attacker with a keylogger can get at it, no RDP, no SSH, none of that. Ideally it is on its own VLAN and can only talk to the file server it needs to talk to and only with an outbound-initiated connection, this has the added bonus that it is unlikely an attacker will ever even know it exists. Veeam B&R Server doesn't have a Linux version does it? Because ensuring that this system isn't infected by a usb device (probably used to move those backups around for testing, right?) would be a nice bonus. Actually, come to think of it, this is a scenario where you probably could use SRP to skin that cat. Finally, it would be prudent not to keep the credentials where you keep all your other credentials at work (we use an encrypted vault, for instance) because a keylogger will give the attacker direct access to that pretty easily. The last piece is pretty well mitigated away by the bit about access and firewalling, but no sense not taking such a simple step I think.

I haven't yet had time to investigate this setup very thoroughly, there could be pitfalls or other considerations I haven't accounted for, but so far I really like it as an alternative to putting tape back in the environment.
cbc-tgschultz
Enthusiast
 
Posts: 46
Liked: 9 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby nitramd » Tue Mar 14, 2017 9:08 pm

mkaec wrote:We used to have tape at all our sites and rely on local users to rotate them. I'm sure you can imagine how many backups were missed. We were so excited to move to disk based backup, which meant removal of the human factor. I never imagined it would have this kind of drawback.

Ahh yes, I remember the human factor well. Years ago I recall DBAs getting angry with me after restoring tables for them on several occasions. Short story, the people responsible for changing tapes typically made mistakes when rotating them, regardless of how many times I explained the rotation scheme to them.

I'm starting to wonder if this is a case of "pick your poison".
nitramd
Enthusiast
 
Posts: 35
Liked: 7 times
Joined: Thu Feb 16, 2017 8:05 pm

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby yasuda » Tue Mar 14, 2017 10:29 pm

Mike Resseler wrote:...Samas is indeed a very painful variant of Ransomware. It used to target the healthcare vertical at first but it seems it is growing and attacking other verticals now also. It is known to search for backup files and basically can find backup files of most of the backup vendors and deletes them...

More bad news for Andrew, I am afraid: From what I read, Samas is introduced by targeted attack against network facing servers, following the usual techniques of getting access to login credentials, then island hopping and escalating access, passing hashes, etc. It's not automated, like phishing. Hospitals have a wealth of valuable data. And schools have student records and maybe even financial aid data. So my conclusion is the ransomware is a bonus; the real value is the data they exfiltrated before installing the ransomware. Really, given that you pwned the network, how could you not take the data? Unless you're running something like Splunk or AlienVault, you won't know data's being exfiltrated, unless, maybe, your ISP has records of bandwidth stats.
yasuda
Enthusiast
 
Posts: 46
Liked: 10 times
Joined: Thu May 15, 2014 3:29 pm
Full Name: Peter Yasuda

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby larry » Wed Mar 15, 2017 7:43 pm

Using an inheritable rights block with only the Veeam service account able to access the files would reduce the risk to the backup files from ransom ware. I think I would leave read rights for admins so they would know where the space is used but safer without. With the new gui there is no need to use the local account for day to day stuff. During Veeam install set the password and then never use it again, would still keep in case something went wrong.
I believe the real solution is offsite and offline and have not tested the above. I do use the blocks to keep admins out of some spots.
larry
Expert
 
Posts: 374
Liked: 89 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby wasc » Thu Mar 16, 2017 3:14 pm

I suppose for me, the other issue is that I don't think you can truly protect a Veeam repository, without maybe Veeam introducing some new functionality.

For example we have a Veeam Repository which is sitting on a server *not* joined to the domain. In addition, the credentials veeam uses to connect to this server are denied rdp access. In theory, this server is partially air-gapped as an attacker would have no way of connecting onto it in order to encrypt or destroy the Veeam backups stored on this repository.

In reality, from the Veeam server itself you can simply Browse to Backup & Replication -> Backups -> Disk, select the repository and order the deletion of all data and the repository server will willingly do it. Maybe we need Veeam updated so you can put a password on the repository which needs to be entered before you can delete it via veeam? Similar to the ability where you can put a password on your anti-virus software to prevent tampering?
wasc
Service Provider
 
Posts: 19
Liked: never
Joined: Wed Jan 11, 2012 4:22 pm
Full Name: Alex

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby evander » Fri Mar 17, 2017 9:28 am

I'm thinking if we can create a pre and/or post script as part of the Veeam backup job that will either boot or shutdown a VM hosting the repository or at a minimum enable/disable that VM's network card prior and after the job will run will give you a "sort-of" air gapped machine.
I'm working on the law of averages here in that assuming the ransomeware attack will occur on all desktop/servers at the same time and the red flag will be raised. The up-time of the Veeam repository will only be up during a certain portion of the night (usually) thereby minimising that attack vector. I fully understand that some ransomeware are intelligent enough to wait and attack on cue but that will need to be mitigated in another way. You can perhaps do what others have suggested by scanning for ANY other file types copied to the repository server other than VBKs etc. and then act accordingly.

What I am thinking is that even if I backup to such a repository once a week I will still be "happy" to lose a weeks worth of data as opposed to losing everything forever!
So this would mean that in an average week the server will be "up" a small percentage of the time. If you make that a month the average gets even smaller. Obviously the up-time will depend on the time for your own individual backups to run but Veeam does a great job in making that a quick as possible already.
If you want you can backup every night to such a repository and to another similar protected repository (different server) once a week or month or whatever.
At this point we need to protect everything and settle for whatever we can recover. Its a sad fact really.

Basically, what I am saying is I want to use technology to create an air-gapped server. I'm not saying it will fix but I do reckon it will help, and that's a step forward.

Thoughts?
evander
Enthusiast
 
Posts: 62
Liked: 4 times
Joined: Thu Nov 17, 2011 7:55 am

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Mike Resseler » Fri Mar 17, 2017 10:51 am

I certainly read a few interesting ideas. But probably (as always) it will be a combination of different technologies.

@wasc: I am not sure if your thought on deleting backups from disk is realistic for malware. That would mean a specific malware, running specific Veeam PowerShell cmdlets would need to exist. That would limit the attack vector for such an investment a lot. But you never know of course

@evander: We have the possibility in VEB (soon VAW) to "eject" the removable media after a backup. The question is, do you really want to do this with your production server? In the end, your suggestion to enable / disable a NIC will probably not work automatically since you can't remotely switch something on if there is no network :-).

Anyway, certainly very interesting conversation and really like to read ideas, how people are doing it today (or going to do it). I think this topic is helpful for many!

Keep ideas coming
Mike Resseler
Veeam Software
 
Posts: 3281
Liked: 365 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby evander » Fri Mar 17, 2017 11:30 am

Hi Mike - my thinking would not be to turn on/off the NIC at the OS level but rather at the VMware (or HyperV) level for the specific virtual machine.
Its a small check box in VMware vCenter to do that so I'm sure there must be a way to do it via command line and hence via a script.

I would be more than happy to disconnect my Veeam repository from the production network (since that is its only purpose, as a Veeam repository) after every backup job has run. If its not on the network it cant be attacked in theory, unless the attack is via the Vsphere ESXi level which currently, as far as I know, has not been attacked.
evander
Enthusiast
 
Posts: 62
Liked: 4 times
Joined: Thu Nov 17, 2011 7:55 am

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: ferrus, mcz, rfn and 52 guests