Comprehensive data protection for all workloads
cbc-tgschultz
Enthusiast
Posts: 50
Liked: 10 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by cbc-tgschultz » Mar 23, 2017 5:07 pm 1 person likes this post

yasuda wrote:if it was that you need to do the cost to benefit analysis, and I completely agree with that.
Yeah, what I was saying is that people need to do the cost/benefit. Added labor hours for making and testing backups, equipment cost, cloud storage if you're going that route, and lost time due to recovery all need to be factored in. That number is likely to be less than whatever the ransom is for a lot of solutions; the ransomers count on it.

That's why I personally like the "offline" Veeam copy. A physically separate non-domain server that is on its own network that is firewalled off from your other networks such that only an outbound connection from that server to your backup repository is allowed. That server just does Veeam Backup Copy Jobs against the repo and stores them locally. Cost is only the server and disk, configuration time, and testing time. The testing time is the only big cost, and that can be mitigated a bit, if you have to, by stretching the definition of "test" to be "poke through some random backups using Veeam Explorer for Filesystem, Exchange, and SQL". I can't stress enough that that isn't really "testing" backups at all, but it is better than nothing.

Also I totally agree that there are preventative measures that, if you're not doing them, you really ought to implement first. Mail filtering for approval based on keywords and content type, web filtering of some kind, FSRM monitoring for known ransomware signatures, canary files, etc. In addition to the more standard security practices like using separate accounts for administrative tasks and all that jazz.

nitramd
Expert
Posts: 175
Liked: 35 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by nitramd » Mar 23, 2017 5:15 pm

yasuda wrote:Random thought about sending tapes off site: If I'm in your network, maybe I've found email or documents, maybe I've found a copy of your DR plan in your wiki. So I call Iron Mountain and say, "We need all our backup tapes delivered to our DR site ASAP! Here's the address..." And for a reasonable fee, not only will I send you the key to decrypt your data, I will mail your tapes to you.

Plausible?
Certainly plausible. We would hope that the company, Iron Mountain in your example, would have protocols in place to correctly authenticate you. This scenario would make me want to investigate the off-site provider, amongst other things.

nitramd
Expert
Posts: 175
Liked: 35 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by nitramd » Mar 23, 2017 5:23 pm

cbc-tgschultz wrote: Also I totally agree that there are preventative measures that, if you're not doing them, you really ought to implement first. Mail filtering for approval based on keywords and content type, web filtering of some kind, FSRM monitoring for known ransomware signatures, canary files, etc. In addition to the more standard security practices like using separate accounts for administrative tasks and all that jazz.
Veritas. Defense in layers.

larry
Expert
Posts: 387
Liked: 92 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by larry » Mar 24, 2017 12:05 pm 2 people like this post

I do not send tapes off site anymore. Tape is for air gap and Veeam Disk to Disk is for offsite, less risk than tape transport much better RTO. In addition, if I want a tape I have it in hand no waiting. Tape is too long of RTO for anything besides the worst case, cannot happen but did day.

Gostev
SVP, Product Management
Posts: 24217
Liked: 3311 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Gostev » Mar 24, 2017 7:58 pm

Right. Most customers who I talked too put a very long SLA for any restore that has to be served from tape. And yes, most use tape solely for air gap and as the "last resort" copy of their backup, so businesses are usually fine with such SLA.

frankj
Service Provider
Posts: 29
Liked: 2 times
Joined: May 27, 2016 4:53 pm
Full Name: FRANK Jacques
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by frankj » Mar 25, 2017 2:42 am

ok why would an auto loader be bad ?

there should be an username/password to access veeam ( off domain ) like a passworded app..

then...i guess the auto loader has some some or management.. off domain as well.. you make it use tap 1-4 week one, 5-8 week 2 , 9-12 week 3, 13-16 week 4....

veeam never knows any better.. it sees same share/drive k: for say... its empty on week 2, it writes, it's empty on week 3. it writes.. etc etc..

never can touch original data till your sets returns to week 1.. tape 1-4.. then it sees it and can delete it if it does.. then you actually detect a loss of data, and remedy with backup of week -1 ....

no credentials can delete backups since they are on a data set not seen by veeam / network..

no ?

ashman70
Expert
Posts: 203
Liked: 12 times
Joined: Dec 04, 2012 2:18 pm
Full Name: Both
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by ashman70 » Mar 27, 2017 1:17 am

I have a couple of questions:

1. If you use an account that is only used to login to the Veeam backup server, is not a member of the domain admin group and there are no shares on your Veeam backup server. How can any ransomeware delete your backups? Unless of course it gets onto your network and sniffs credentials, but is it then logging onto computers with those sniffed credentials, or only deleting files in shares its accessing using those sniffed credentials?

2. If you have a VMWare host that you are replicating to, are your replicas safe from this kind of ransomware attack? Obviously a VM that's had files encrypted or deleted will replicate with those said files encrypted or deleted but previous replicas should be fine, no?

evander
Enthusiast
Posts: 67
Liked: 4 times
Joined: Nov 17, 2011 7:55 am
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by evander » Mar 27, 2017 6:44 am

I have implemented the following, how effective it proves remains to be seen but I think it can definitely help.

I setup File Server Resource Manager (FSRM) on my Veeam copy job top level folder. I followed a different route to other suggestions on this thread and instead of trying to prevent known ransomware extensions on the drive I blocked ALL files and made an exception just for Veeam files. In this way I don't have to continuously update with new file extensions as they become available.
Many ransomware attacks will encrypt the file and change the extension. Changing the extension in this scenario will fail as FSRM will block it.

I suppose the Veeam files could still be deleted but I would rather try and recover a deleted file than an encrypted file, right?

You could also mitigate this delete file risk somewhat by manually or (via script) changing the default Veeam file extensions to something arbitrary that the hacker will not necessarily know. You could for example rename the extensions on all your monthly, quarterly, yearly .vbk files to something with a .fhm5x extension and add that to your FSRM exclude list. I wont tell you why I chose that file extension other than to say the word Hacker and Mother are in there somewhere :wink:
Its not in the ransomware hackers best interest to delete any files other than known backup files so this, in theory, should help. They wont know the file extension you choose and cant really delete all files without shooting themselves in the foot if they want money from you.

I still think implementing this together with a transient technologically controlled air-gapped server/repository via hypervisor (or other) script to disable the NIC or shutdown the repository VM when not explicitly in use during a backup run is an even better solution. Of course if the bad guys are already on your network playing Doom for weeks or months, you probably have bigger problems anyway. This is more for the random ransomware attack that will come in via the unsuspecting user and kick off a mass encryption job across anything it can touch or reach on the network.

PS: If you interested in how I setup FSRM, this is what I did:

Block all files:
*.*

Exclude the following:
*.vbk
*.vbm* (note the trailing *)
*.vib
*.vrb
heartbeat.bin (This may be unique to my environment only)

I am only doing this on Copy jobs but I suspect it will work fine on normal jobs too.

erbr
Novice
Posts: 7
Liked: never
Joined: Jan 28, 2016 8:20 am
Full Name: Erik Brödje
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by erbr » Mar 27, 2017 7:35 am

Hi.


My Backup drives are only attached to the Veeam Server and is not shared on the network at all. Im guessing the reasoning behind having an SMB is that you have proxies?

Dave1981
Lurker
Posts: 2
Liked: never
Joined: Mar 14, 2014 9:51 am
Full Name: Dave Thomas
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Dave1981 » Mar 27, 2017 7:41 am

I've been recently looking at the replication part of Veeam for another project and believe that it may help in this scenario.

At the moment we have an offsite server on it's own work group (different login credentials) that the Primary site performs a Backup Copy job to every night to a Repository located on that server.

We have then installed an additional copy of VB&R on the offisite server as well as enabling Hyper-V and have set up Replication jobs with the source being the VBK files in that repository.

Whist this wasn't set up with Ransomware in mind my thinking is that if this did hit us and the backup files in the offsite repository were encrypted then the daily Replication jobs would also fail due to Veeam being unable to read them. This would mean that the Replicas stored would be left in the state they were in on the final day before the attack and could easily be restored to the live environment.

hercous
Lurker
Posts: 2
Liked: never
Joined: Nov 30, 2012 8:47 am
Full Name: Petr Herzig
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by hercous » Mar 27, 2017 8:22 am

cbc-tgschultz wrote:That's why I personally like the "offline" Veeam copy. A physically separate non-domain server that is on its own network that is firewalled off from your other networks such that only an outbound connection from that server to your backup repository is allowed. That server just does Veeam Backup Copy Jobs against the repo and stores them locally. Cost is only the server and disk, configuration time, and testing time.
Hi,

thanks for very insteresting posts! But how do you configure that "offline" backup server only for outbound connection? Veeam Backup Copy Jobs are managed by the Veeam backup server or not?

Regards

cbc-tgschultz
Enthusiast
Posts: 50
Liked: 10 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by cbc-tgschultz » Mar 27, 2017 1:47 pm

ashman70 wrote:I have a couple of questions:

1. If you use an account that is only used to login to the Veeam backup server, is not a member of the domain admin group and there are no shares on your Veeam backup server. How can any ransomeware delete your backups? Unless of course it gets onto your network and sniffs credentials, but is it then logging onto computers with those sniffed credentials, or only deleting files in shares its accessing using those sniffed credentials?

2. If you have a VMWare host that you are replicating to, are your replicas safe from this kind of ransomware attack? Obviously a VM that's had files encrypted or deleted will replicate with those said files encrypted or deleted but previous replicas should be fine, no?
1. Theoretically that might work, but it has some issues in practice. Namely, if your B&R server is accessible, human beings being lazy, they'll likely remote into it (keylogger can grab this) from time to time to do restores. Once the attacker has access to the B&R server, they just tell it to delete all the data and you're hosed. Additionally, since B&R has domain credentials, it must be storing them somewhere and the attacker could theoretically pull them out (encrypted or not, they have to be decrypted by B&R for use, so there's potentially a way to get B&R to give them up or give up the decryption key because that has to be somewhere too).

2. Remember this whole discussion started because Ransomware isn't just about low hanging fruit anymore, we have actual documented instances of breaches where the attacker has bided their time to gather credentials and enough data to know where to strike to do the most damage. Veeam B&R will let you simply delete a replica from disk using a right-click, so if it is compromised, game over for replicas too.

evander wrote:I setup File Server Resource Manager (FSRM) on my Veeam copy job top level folder. I followed a different route to other suggestions on this thread and instead of trying to prevent known ransomware extensions on the drive I blocked ALL files and made an exception just for Veeam files. In this way I don't have to continuously update with new file extensions as they become available.
Many ransomware attacks will encrypt the file and change the extension. Changing the extension in this scenario will fail as FSRM will block it.

I suppose the Veeam files could still be deleted but I would rather try and recover a deleted file than an encrypted file, right?
Different ransomware strains do things differently. Some will create a new copy of a file during encryption and then delete the old one when completed, and these might be stopped by blocking file extensions, but others will encrypt in-place and rename after the fact. In case of those, all you're doing is preventing the file from being renamed, making it hard to identify which files were hit; the data is still encrypted.

If you're doing FSRM, make sure you're taking advantage of its ability to send alerts. I personally use it to run scripts as well as alerting me. That way, under the right conditions, it will take automated action to limit the damage.

hercous wrote: Hi,

thanks for very insteresting posts! But how do you configure that "offline" backup server only for outbound connection? Veeam Backup Copy Jobs are managed by the Veeam backup server or not?

Regards

I haven't had time to dig into it, so I might be missing something, but here's how I'm thinking of doing it: The "offline" server would sit on a private VLAN completely separate from all your other network resources, or (as in my case) it could be connected to its own private physical interface on the storage hardware (A NAS in my case). Let's call this private network B, and your normal network A. Using either a hardware firewall, or Windows firewall if you can't swing that, configure rules like so:

Inbound (From A to B): Deny all.
Outbound (From B to A): Allow only to storage hardware.

Firewalls are smart enough to understand that a connection allowed from B to A will have to be able to receive reply packets that match the socket from B to A (and only if B initiated the conversation), but otherwise will deny all attempts to talk to A from B.

On the "offline" server, install a new instance of Veeam B&R. This "Veeam Environment" will not be connected to your regular Veeam infrastructure, meaning anyone who manged to compromise that wouldn't even know it existed, which is a bonus. Add your storage as a new backup repository on the "offline" B&R server. If I understand the workings correctly, which hopefully someone can clarify, you can then import backups from your storage and run Backup Copy jobs on those backups to the local storage in the server (or an "offline" NAS that goes with this new server). As I understand it, you won't require any licenses for this since you only need to license for servers you back up or replicate from.

The sticking point in this plan, I think, is if Backup Copy jobs can work under these conditions, which I haven't yet tested.

yasuda
Enthusiast
Posts: 61
Liked: 10 times
Joined: May 15, 2014 3:29 pm
Full Name: Peter Yasuda
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by yasuda » Mar 27, 2017 4:43 pm 1 person likes this post

frankj wrote:ok why would an auto loader be bad ?

there should be an username/password to access veeam ( off domain ) like a passworded app..

then...i guess the auto loader has some some or management.. off domain as well.. you make it use tap 1-4 week one, 5-8 week 2 , 9-12 week 3, 13-16 week 4....

veeam never knows any better.. it sees same share/drive k: for say... its empty on week 2, it writes, it's empty on week 3. it writes.. etc etc..

never can touch original data till your sets returns to week 1.. tape 1-4.. then it sees it and can delete it if it does.. then you actually detect a loss of data, and remedy with backup of week -1 ....

no credentials can delete backups since they are on a data set not seen by veeam / network..

no ?
It's not bad; it's just not as good as tapes on a shelf because, given enough time, the attacker can gain the credentials to the machine managing the autoloader, and wipe your tapes. Unless your opsec is exceptional, and your tape manager is itself air gapped and you walk over to a console to manage it.

That said, the attacker could have been encrypting your backups all along, but with tapes on a shelf, you have a greater chance of detecting that by needing to do a restore before he can cycle through all the tapes.

tfelice
Influencer
Posts: 10
Liked: never
Joined: May 23, 2012 12:28 pm
Full Name: Tony Felice
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by tfelice » Mar 27, 2017 8:25 pm

aporter wrote:One approach I haven't seen mentioned on this thread yet is to configure a pull using something like rsync, or perhaps another instance of Veeam, from your backup repository to another server which is on a different physical and logical network... This way nothing that is online would have access to the last resort archive server, but it would have access to pull from the primary backup repository (one way, read only).
I've used this approach. I think it has a lot of merit in some situations. As mentioned by cbc-tgschultz, it's critical to make sure that the credentials of the "last resort" machine cannot be obtained by keyloggers or other subterfuge. In my case, the host that is pulling backups is a Linux server. There is no Samba installation, so there is no relationship between the Linux box and the Windows domain. Thus, no compromised domain credentials or host, including the Veeam server, can access the "last resort" box. There are a lot of potential tools to use, but I happen to use use rsync. If the source repository is a Windows machine, you need to install an implementation of rsync on it. I use the native rsync protocol rather than ssh if I am copying across a LAN, so as avoid the unnecessary CPU overhead. You can use rsync's --compare-dest and --link-dest to economically create a directory for each backup iteration. You'd have to script the pull so that you get the retention schedule you wish, allowing yourself enough restore points to go back to a good backup.

For administration purposes I do access the "last resort" host via SSH, but only with an RSA key and passphrase and never with password only. Even key logging the passphrase doesn't get an intruder access without the private key. Since the last-resort backup job is pulled rather than pushed by the Veeam repository, then even if the Veeam server is compromised, there is no way for the bad guys to figure out how to access the last-resort host. There isn't even any local evidence that it exists.

One gotcha is that if you are using reverse incremental backups, Veeam by default renames the .VBK file every night. This would cause rsync (or anything trying to capture incremental file changes) to have to copy the entire .VBK every time, which would be expensive both in time and disk space. For a solution, see Veeam https://www.veeam.com/kb1076 on disabling the renaming of the VBK.

Gerald
Novice
Posts: 6
Liked: never
Joined: Sep 04, 2015 5:56 pm
Full Name: Gerald
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Gerald » Mar 28, 2017 7:02 pm

I have so far only used the free Endpoint Backup (now called Veeam Agent) with SAMBA shares on a NAS as destination.

I deliberately don't grant the Windows User accounts access to the backup share, but the Veeam Client does of course need the authentication details so of course a crypto-trojan could read where Veeam stores this information (DB File? Registry? Config File?) and use it to access the share and delete/encrypt the backups also.

My question: Would I be more secure with a non-free Veeam Server instead of a NAS as backup target? When backing up to a Veeam Server instead of a NAS, who manages retention?
Does the client do this? If yes, then of course the client could also delete everything. If retention is configured and handled server-side only on a Veeam Backup server, then it would be more secure than a NAS, provided the l/p authentication for the server cannot be found on any one of the clients?

Mike Resseler
Product Manager
Posts: 5655
Liked: 591 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Mike Resseler » Mar 29, 2017 5:57 am

Gerald,

The authentication details are stored in the DB and certainly not easily readable. That data in the DB is encrypted and you cannot just decrypt that. But that being said, a very specific written Trojan might succeed in getting that information out and then access the NAS and encrypt your backup files. Far fetched? Maybe, possible, unfortunately.

When you backup to a Veeam server, it is still the client who manages retention. After all, in the client, you can change the retention from 7 days to 1 (for example) and the change will be done on next backup.

But if you read through this entire thread (I know, it is a lot :-)) you will certainly find good mechanisms to defend your system. If you consider backup to a VBR repository, then you can do backup copy jobs of that data to a repository that supports rotating drives.

As you will see in this thread, you can harden your solution as much as possible, making the possibility of losing your backup files much less possible, but as long as it is not air-gapped, there is a risk. At my home (for the family) I simply use rotating USB devices (and the nice little checkbox to eject the device after backup). The only thing I had to do was teach my family (make that the kids, they understood it faster :-)) that each day they need to unplug the USB, and plug in the other one. Yes it is manual work and it is not fault-proof but it could be that simple :-)

infused
Service Provider
Posts: 107
Liked: 4 times
Joined: Apr 20, 2013 9:25 am
Full Name: Hayden Kirk
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by infused » Mar 30, 2017 1:32 am

I use NAS's to store backups for this very reason. I suspected something like this might start happening.

The admin account is disabled, with another one I use. There is a Veeam account which has RW to a folder, plus another one for retention on another NAS.

It's slower than using something like storage spaces/windows, but ya know...
http://www.infused.co.nz - My Blog.

evander
Enthusiast
Posts: 67
Liked: 4 times
Joined: Nov 17, 2011 7:55 am
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by evander » Mar 30, 2017 6:57 am

cbc-tgschultz wrote:
Different ransomware strains do things differently. Some will create a new copy of a file during encryption and then delete the old one when completed, and these might be stopped by blocking file extensions, but others will encrypt in-place and rename after the fact. In case of those, all you're doing is preventing the file from being renamed, making it hard to identify which files were hit; the data is still encrypted.

If you're doing FSRM, make sure you're taking advantage of its ability to send alerts. I personally use it to run scripts as well as alerting me. That way, under the right conditions, it will take automated action to limit the damage.
I totally agree, however I'm still optimistic that even if a particular ransomware strain does in-place encryption that it still has to make use of a tmp file of some sort to complete the process, in which case that file will be blocked if its in the same location. Also if it users the process memory to hold the tmp file it might still fail because the size of vbk files are generally much bigger than the available memory, right?
I confess I haven't done any testing to substantiate this thinking but I'd love to know if I'm right or wrong.
Additionally, if it makes a copy of the file, encrypts the copy (again with a possible tmp file of some sort) and then deletes the original (un-encrypted) file, chances of recovering the deleted files is a lot easier that recovering the encrypted file. The thing to take away is that by trying to protect just your backup file(s) versus trying to protect every single file in your organisation should be a lot easier and where I think its worth focusing on.

Overall I'm loving this thread and the different ideas that are being put forward.

AIM_joshuag
Novice
Posts: 3
Liked: never
Joined: May 07, 2015 6:09 pm
Full Name: Joshua Garrett
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by AIM_joshuag » Mar 30, 2017 5:04 pm

MOBO wrote:how about if the repository is a standalone server and veeam program is the only system that has the credentials , am i still at risk for malware to delete backups ?
This would work only if the infection didn't infect a machine that you use those credentials from. If you log into veeam with the veeam credentials then a keylogger could use those to bring harm. If you really wanted to be sure you could have an offline laptop around that was used to check your backups from time to time.

cbc-tgschultz
Enthusiast
Posts: 50
Liked: 10 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by cbc-tgschultz » Mar 31, 2017 3:32 pm 1 person likes this post

Mike Resseler wrote:The authentication details are stored in the DB and certainly not easily readable. That data in the DB is encrypted and you cannot just decrypt that. But that being said, a very specific written Trojan might succeed in getting that information out and then access the NAS and encrypt your backup files. Far fetched? Maybe, possible, unfortunately.
I'm afraid I must disagree that this is far fetched. As you said, the credentials are stored in the DB, and as much as you may obfuscate it the fact remains that the Veeam application must have the decryption key in order to decrypt the credentials. Therefore there is no cryptographic integrity to the credentials if an attacker has complete control of the host Veeam resides on. People unwind these kinds of schemes all the time, often just for fun and/or some prestige, it would be a mistake to believe that there isn't a tool already floating around the criminal underground that will accomplish this for an attacker.

Is it unlikely that you'll encounter an attacker that A) gets onto your network B) compromises your Veeam server, and C) has access to such a tool or the knowledge to do it himself? Maybe. But do you want to take that chance if you have another option?

Mike Resseler
Product Manager
Posts: 5655
Liked: 591 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Mike Resseler » Apr 03, 2017 8:07 am

Hi Tanner,

As I said, impossible... Unfortunately not :-(. But I do agree certainly that you do not want to take that change and have as much defense systems or layers (and don't forget the air-gap) as possible as it is getting worse and worse.

dellock6
Veeam Software
Posts: 5665
Liked: 1593 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by dellock6 » Apr 03, 2017 10:05 pm 1 person likes this post

Actually you don't need this tool at all. Let's not forget that the goal here is not to steal data, it's to steal money. So I don't even need probably to open a Veeam console, I may target directly Veeam repositories to delete backup copies, so that the encrypted files are the only copies, so to force the victim to pay. If you read that thread on Reddit, the attacker encrypted the files, but then they just formatted veeam repos, they didn't need to encrypt those backups or anything else. Air gapping is the only real defense here.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

y1008946
Enthusiast
Posts: 93
Liked: never
Joined: Sep 23, 2013 3:56 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by y1008946 » Apr 05, 2017 11:11 am

Hi, we have a NetApp 2552 where the data is stored and NetApp snapshots are taken.

We can see the NetApp snapshots in Veeam and can restore from them.

Are the NetApp snapshots protected or are they at risk too?

Thanks

cbc-tgschultz
Enthusiast
Posts: 50
Liked: 10 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by cbc-tgschultz » Apr 05, 2017 6:39 pm

dellock6 wrote:Actually you don't need this tool at all. Let's not forget that the goal here is not to steal data, it's to steal money. So I don't even need probably to open a Veeam console, I may target directly Veeam repositories to delete backup copies, so that the encrypted files are the only copies, so to force the victim to pay. If you read that thread on Reddit, the attacker encrypted the files, but then they just formatted veeam repos, they didn't need to encrypt those backups or anything else. Air gapping is the only real defense here.

It isn't the encryption that we're talking about protecting, but the credentials used to access the storage, which happen to be encrypted in the Veeam Database. Since Veeam needs to decrypt those credentials in order to use them, it must also store the decryption key somewhere, so my point was that a suitably savvy attacker could get at them anyway.

Of course, if the attacker has control of the Veeam server, they can much more simply direct Veeam to just delete all the backups anyway.
y1008946 wrote:Are the NetApp snapshots protected or are they at risk too?
If the attacker can get control of any system that has access to those snapshots and permission to delete them, then they are not safe.

y1008946
Enthusiast
Posts: 93
Liked: never
Joined: Sep 23, 2013 3:56 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by y1008946 » Apr 06, 2017 10:28 am

Ok thanks,

The Netapp snapshots are meant to be read only, but I guess Veeam has a facility to delete them through the console?

It is the Netapp controlling and running the snapshots, the link with veeam is just so that we can see them.

I guess if we remove the storage from the veeam console and the credentials that should help.

Nearly all of the time if we need to restore, we do it from nightly backups stored on the Veeam server.

It would only be if we were desperate would we use storage snapshots, and at that point we could re add it to the console?

Thanks

cbc-tgschultz
Enthusiast
Posts: 50
Liked: 10 times
Joined: May 13, 2016 1:48 pm
Full Name: Tanner Schultz
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by cbc-tgschultz » Apr 10, 2017 1:26 pm

I am unfamiliar with NetApp, let alone your configuration, but here's what you have to consider:

If Veeam can delete it, it should be considered vulnerable. If you can delete it using credentials that it would be possible to obtain via keylogging or by otherwise knowing where to find them, it should be considered vulnerable.

To be as safe as you can be under the conditions we've been discussing, you need at least one backup location requires physical access to destroy. If your NetApp snapshots can only be deleted by logging into NetApp from a physical terminal of some kind, or from a system that cannot be remoted into, then they should be ok.

dellock6
Veeam Software
Posts: 5665
Liked: 1593 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by dellock6 » Apr 10, 2017 7:54 pm

Daniel, what we are saying here (and I totally agree with Tanner) is that any system connected to the network can be targeted, yes you can make snapshots read-only, but if I get access to the netapp console, I can probably just delete them. What can help here are two things, but not accessible directly from the network by the attacker:
- 2FA (two-factor authentication) so that user needs to input a piece of credential that the attacker cannot get. Think like Google authenticator. There are solutions to implement it in a Windows RDP for example. It can be compromised if the attacker also gets access to your smartphone, but you surely lower the chances.
- the only real solution, as we are saying, it's air-gapping your backups. I write my backups to a tape, and I physically remove the tape from the library, and I even lock the mechanical read-only mechanism. The only way to destroy my backups at this point becomes to get physical access to those tapes as Tanner said, but this is really dangerous for the attacker, that usually prefers to hide himself behind the internet.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

rmitura
Lurker
Posts: 1
Liked: never
Joined: Jan 31, 2017 10:47 am
Full Name: R Mitura
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by rmitura » Apr 13, 2017 10:37 am

hello,
In my opinion:
1) if any windows host can write to CIFS shares than it can overwrite files with 0 length file. Altering file systems permissions would not help.
2) don't know if malware is capable of exploiting veeam harware snapshots API but if not netapp created snapshot should be safe.
3) in the old days I worked on openvms which had filing system with visioning (any file modification resulted in creating new one with higher version No). Guessing it would be easy to do crawler script that stripped permission from previous versions to local system user and run it on 10-20min scheduler. Not sure if there are file system versioning are still available but could be an option.
4) if malware only targets veeam files altering file extension instantly after backup job has finished (easy enough to add little script to your backup job) should be enough to protect them (won't work if malware encrypts everything if finds).
5) whitelisting application execution seams like good way forward (most likely it will need to be done on all veeam agent servers assuming no other device can contact your backup storage). It will make your system start, windows and antivirus update nightmare but if you do it right you wouldn't even need antivirus on your server.
6) detached volumes. Backup destination volumes are attached just before backup starts and detached after backup finish. Probably false sense of security (especially if your full backup takes 2 days) but better that than nothing.

If you think of anything else please share your ideas :-)
R. Mitura

rntguy
Enthusiast
Posts: 82
Liked: 1 time
Joined: Jan 29, 2016 8:31 pm
Full Name: Rnt Guy
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by rntguy » May 22, 2017 1:43 pm

A situation we encountered was someone gained access to an IT manager's computer, RDPed into the Veeam server, deleted the repository files, encrypted all the servers using his own credentials from the logon session, etc.

By the time we found out, there was 2 hours left until the Data Domain was scheduled to run it's cleaning job which would have wiped out all his backups that Veeam thought it deleted. Had it been a Windows repository it would have been toast. We turned off scheduled cleaning, restored from his backups and within 24 hours all servers were back to normal. Thankfully the attacker (disgruntled employee or someone from the outside) didn't know about DD's cleaning routine. My guess is he could have gone into the DD if he figured it out and manually kicked off the cleaning job. And this file said they wanted over $10K, not $300 per machine.

shaun_ec
Novice
Posts: 5
Liked: never
Joined: Aug 22, 2013 6:55 am
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by shaun_ec » Aug 05, 2017 6:53 am

Need some advise on securing our local VEEAM repositories.

We have a VEEAM server connected via iscsi to nas. Looking to secure this repository by setting the file level permissions. VEEAM seems to need the SYSTEM user to be granted access to the repository else the backups fail with access denied. Looking to setup a local account and assigning only this account access to the backup repositories.

1) If I change the VEEAM services to start-up using this new local account. Will VEEAM still need the SYSTEM user to be granted access to the repository
2) What is the minimum permissions that is needed to be assigned to this account?

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 43 guests