Yes, Ransomware can delete your Veeam backups.

Availability for the Always-On Enterprise

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby cbc-tgschultz » Thu Mar 23, 2017 5:07 pm 1 person likes this post

yasuda wrote:if it was that you need to do the cost to benefit analysis, and I completely agree with that.


Yeah, what I was saying is that people need to do the cost/benefit. Added labor hours for making and testing backups, equipment cost, cloud storage if you're going that route, and lost time due to recovery all need to be factored in. That number is likely to be less than whatever the ransom is for a lot of solutions; the ransomers count on it.

That's why I personally like the "offline" Veeam copy. A physically separate non-domain server that is on its own network that is firewalled off from your other networks such that only an outbound connection from that server to your backup repository is allowed. That server just does Veeam Backup Copy Jobs against the repo and stores them locally. Cost is only the server and disk, configuration time, and testing time. The testing time is the only big cost, and that can be mitigated a bit, if you have to, by stretching the definition of "test" to be "poke through some random backups using Veeam Explorer for Filesystem, Exchange, and SQL". I can't stress enough that that isn't really "testing" backups at all, but it is better than nothing.

Also I totally agree that there are preventative measures that, if you're not doing them, you really ought to implement first. Mail filtering for approval based on keywords and content type, web filtering of some kind, FSRM monitoring for known ransomware signatures, canary files, etc. In addition to the more standard security practices like using separate accounts for administrative tasks and all that jazz.
cbc-tgschultz
Enthusiast
 
Posts: 45
Liked: 9 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby nitramd » Thu Mar 23, 2017 5:15 pm

yasuda wrote:Random thought about sending tapes off site: If I'm in your network, maybe I've found email or documents, maybe I've found a copy of your DR plan in your wiki. So I call Iron Mountain and say, "We need all our backup tapes delivered to our DR site ASAP! Here's the address..." And for a reasonable fee, not only will I send you the key to decrypt your data, I will mail your tapes to you.

Plausible?


Certainly plausible. We would hope that the company, Iron Mountain in your example, would have protocols in place to correctly authenticate you. This scenario would make me want to investigate the off-site provider, amongst other things.
nitramd
Enthusiast
 
Posts: 26
Liked: 2 times
Joined: Thu Feb 16, 2017 8:05 pm

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby nitramd » Thu Mar 23, 2017 5:23 pm

cbc-tgschultz wrote:Also I totally agree that there are preventative measures that, if you're not doing them, you really ought to implement first. Mail filtering for approval based on keywords and content type, web filtering of some kind, FSRM monitoring for known ransomware signatures, canary files, etc. In addition to the more standard security practices like using separate accounts for administrative tasks and all that jazz.

Veritas. Defense in layers.
nitramd
Enthusiast
 
Posts: 26
Liked: 2 times
Joined: Thu Feb 16, 2017 8:05 pm

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby larry » Fri Mar 24, 2017 12:05 pm 2 people like this post

I do not send tapes off site anymore. Tape is for air gap and Veeam Disk to Disk is for offsite, less risk than tape transport much better RTO. In addition, if I want a tape I have it in hand no waiting. Tape is too long of RTO for anything besides the worst case, cannot happen but did day.
larry
Expert
 
Posts: 371
Liked: 88 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Gostev » Fri Mar 24, 2017 7:58 pm

Right. Most customers who I talked too put a very long SLA for any restore that has to be served from tape. And yes, most use tape solely for air gap and as the "last resort" copy of their backup, so businesses are usually fine with such SLA.
Gostev
Veeam Software
 
Posts: 21355
Liked: 2334 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby frankj » Sat Mar 25, 2017 2:42 am

ok why would an auto loader be bad ?

there should be an username/password to access veeam ( off domain ) like a passworded app..

then...i guess the auto loader has some some or management.. off domain as well.. you make it use tap 1-4 week one, 5-8 week 2 , 9-12 week 3, 13-16 week 4....

veeam never knows any better.. it sees same share/drive k: for say... its empty on week 2, it writes, it's empty on week 3. it writes.. etc etc..

never can touch original data till your sets returns to week 1.. tape 1-4.. then it sees it and can delete it if it does.. then you actually detect a loss of data, and remedy with backup of week -1 ....

no credentials can delete backups since they are on a data set not seen by veeam / network..

no ?
frankj
Service Provider
 
Posts: 20
Liked: 1 time
Joined: Fri May 27, 2016 4:53 pm
Full Name: FRANK Jacques

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby ashman70 » Mon Mar 27, 2017 1:17 am

I have a couple of questions:

1. If you use an account that is only used to login to the Veeam backup server, is not a member of the domain admin group and there are no shares on your Veeam backup server. How can any ransomeware delete your backups? Unless of course it gets onto your network and sniffs credentials, but is it then logging onto computers with those sniffed credentials, or only deleting files in shares its accessing using those sniffed credentials?

2. If you have a VMWare host that you are replicating to, are your replicas safe from this kind of ransomware attack? Obviously a VM that's had files encrypted or deleted will replicate with those said files encrypted or deleted but previous replicas should be fine, no?
ashman70
Expert
 
Posts: 193
Liked: 11 times
Joined: Tue Dec 04, 2012 2:18 pm
Full Name: Both

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby evander » Mon Mar 27, 2017 6:44 am

I have implemented the following, how effective it proves remains to be seen but I think it can definitely help.

I setup File Server Resource Manager (FSRM) on my Veeam copy job top level folder. I followed a different route to other suggestions on this thread and instead of trying to prevent known ransomware extensions on the drive I blocked ALL files and made an exception just for Veeam files. In this way I don't have to continuously update with new file extensions as they become available.
Many ransomware attacks will encrypt the file and change the extension. Changing the extension in this scenario will fail as FSRM will block it.

I suppose the Veeam files could still be deleted but I would rather try and recover a deleted file than an encrypted file, right?

You could also mitigate this delete file risk somewhat by manually or (via script) changing the default Veeam file extensions to something arbitrary that the hacker will not necessarily know. You could for example rename the extensions on all your monthly, quarterly, yearly .vbk files to something with a .fhm5x extension and add that to your FSRM exclude list. I wont tell you why I chose that file extension other than to say the word Hacker and Mother are in there somewhere :wink:
Its not in the ransomware hackers best interest to delete any files other than known backup files so this, in theory, should help. They wont know the file extension you choose and cant really delete all files without shooting themselves in the foot if they want money from you.

I still think implementing this together with a transient technologically controlled air-gapped server/repository via hypervisor (or other) script to disable the NIC or shutdown the repository VM when not explicitly in use during a backup run is an even better solution. Of course if the bad guys are already on your network playing Doom for weeks or months, you probably have bigger problems anyway. This is more for the random ransomware attack that will come in via the unsuspecting user and kick off a mass encryption job across anything it can touch or reach on the network.

PS: If you interested in how I setup FSRM, this is what I did:

Block all files:
*.*

Exclude the following:
*.vbk
*.vbm* (note the trailing *)
*.vib
*.vrb
heartbeat.bin (This may be unique to my environment only)

I am only doing this on Copy jobs but I suspect it will work fine on normal jobs too.
evander
Enthusiast
 
Posts: 62
Liked: 4 times
Joined: Thu Nov 17, 2011 7:55 am

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby erbr » Mon Mar 27, 2017 7:35 am

Hi.


My Backup drives are only attached to the Veeam Server and is not shared on the network at all. Im guessing the reasoning behind having an SMB is that you have proxies?
erbr
Novice
 
Posts: 7
Liked: never
Joined: Thu Jan 28, 2016 8:20 am
Full Name: Erik Brödje

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Dave1981 » Mon Mar 27, 2017 7:41 am

I've been recently looking at the replication part of Veeam for another project and believe that it may help in this scenario.

At the moment we have an offsite server on it's own work group (different login credentials) that the Primary site performs a Backup Copy job to every night to a Repository located on that server.

We have then installed an additional copy of VB&R on the offisite server as well as enabling Hyper-V and have set up Replication jobs with the source being the VBK files in that repository.

Whist this wasn't set up with Ransomware in mind my thinking is that if this did hit us and the backup files in the offsite repository were encrypted then the daily Replication jobs would also fail due to Veeam being unable to read them. This would mean that the Replicas stored would be left in the state they were in on the final day before the attack and could easily be restored to the live environment.
Dave1981
Lurker
 
Posts: 2
Liked: never
Joined: Fri Mar 14, 2014 9:51 am
Full Name: Dave Thomas

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby hercous » Mon Mar 27, 2017 8:22 am

cbc-tgschultz wrote:That's why I personally like the "offline" Veeam copy. A physically separate non-domain server that is on its own network that is firewalled off from your other networks such that only an outbound connection from that server to your backup repository is allowed. That server just does Veeam Backup Copy Jobs against the repo and stores them locally. Cost is only the server and disk, configuration time, and testing time.


Hi,

thanks for very insteresting posts! But how do you configure that "offline" backup server only for outbound connection? Veeam Backup Copy Jobs are managed by the Veeam backup server or not?

Regards
hercous
Lurker
 
Posts: 2
Liked: never
Joined: Fri Nov 30, 2012 8:47 am
Full Name: Petr Herzig

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby cbc-tgschultz » Mon Mar 27, 2017 1:47 pm

ashman70 wrote:I have a couple of questions:

1. If you use an account that is only used to login to the Veeam backup server, is not a member of the domain admin group and there are no shares on your Veeam backup server. How can any ransomeware delete your backups? Unless of course it gets onto your network and sniffs credentials, but is it then logging onto computers with those sniffed credentials, or only deleting files in shares its accessing using those sniffed credentials?

2. If you have a VMWare host that you are replicating to, are your replicas safe from this kind of ransomware attack? Obviously a VM that's had files encrypted or deleted will replicate with those said files encrypted or deleted but previous replicas should be fine, no?


1. Theoretically that might work, but it has some issues in practice. Namely, if your B&R server is accessible, human beings being lazy, they'll likely remote into it (keylogger can grab this) from time to time to do restores. Once the attacker has access to the B&R server, they just tell it to delete all the data and you're hosed. Additionally, since B&R has domain credentials, it must be storing them somewhere and the attacker could theoretically pull them out (encrypted or not, they have to be decrypted by B&R for use, so there's potentially a way to get B&R to give them up or give up the decryption key because that has to be somewhere too).

2. Remember this whole discussion started because Ransomware isn't just about low hanging fruit anymore, we have actual documented instances of breaches where the attacker has bided their time to gather credentials and enough data to know where to strike to do the most damage. Veeam B&R will let you simply delete a replica from disk using a right-click, so if it is compromised, game over for replicas too.


evander wrote:I setup File Server Resource Manager (FSRM) on my Veeam copy job top level folder. I followed a different route to other suggestions on this thread and instead of trying to prevent known ransomware extensions on the drive I blocked ALL files and made an exception just for Veeam files. In this way I don't have to continuously update with new file extensions as they become available.
Many ransomware attacks will encrypt the file and change the extension. Changing the extension in this scenario will fail as FSRM will block it.

I suppose the Veeam files could still be deleted but I would rather try and recover a deleted file than an encrypted file, right?


Different ransomware strains do things differently. Some will create a new copy of a file during encryption and then delete the old one when completed, and these might be stopped by blocking file extensions, but others will encrypt in-place and rename after the fact. In case of those, all you're doing is preventing the file from being renamed, making it hard to identify which files were hit; the data is still encrypted.

If you're doing FSRM, make sure you're taking advantage of its ability to send alerts. I personally use it to run scripts as well as alerting me. That way, under the right conditions, it will take automated action to limit the damage.


hercous wrote:Hi,

thanks for very insteresting posts! But how do you configure that "offline" backup server only for outbound connection? Veeam Backup Copy Jobs are managed by the Veeam backup server or not?

Regards



I haven't had time to dig into it, so I might be missing something, but here's how I'm thinking of doing it: The "offline" server would sit on a private VLAN completely separate from all your other network resources, or (as in my case) it could be connected to its own private physical interface on the storage hardware (A NAS in my case). Let's call this private network B, and your normal network A. Using either a hardware firewall, or Windows firewall if you can't swing that, configure rules like so:

Inbound (From A to B): Deny all.
Outbound (From B to A): Allow only to storage hardware.

Firewalls are smart enough to understand that a connection allowed from B to A will have to be able to receive reply packets that match the socket from B to A (and only if B initiated the conversation), but otherwise will deny all attempts to talk to A from B.

On the "offline" server, install a new instance of Veeam B&R. This "Veeam Environment" will not be connected to your regular Veeam infrastructure, meaning anyone who manged to compromise that wouldn't even know it existed, which is a bonus. Add your storage as a new backup repository on the "offline" B&R server. If I understand the workings correctly, which hopefully someone can clarify, you can then import backups from your storage and run Backup Copy jobs on those backups to the local storage in the server (or an "offline" NAS that goes with this new server). As I understand it, you won't require any licenses for this since you only need to license for servers you back up or replicate from.

The sticking point in this plan, I think, is if Backup Copy jobs can work under these conditions, which I haven't yet tested.
cbc-tgschultz
Enthusiast
 
Posts: 45
Liked: 9 times
Joined: Fri May 13, 2016 1:48 pm
Full Name: Tanner Schultz

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby yasuda » Mon Mar 27, 2017 4:43 pm 1 person likes this post

frankj wrote:ok why would an auto loader be bad ?

there should be an username/password to access veeam ( off domain ) like a passworded app..

then...i guess the auto loader has some some or management.. off domain as well.. you make it use tap 1-4 week one, 5-8 week 2 , 9-12 week 3, 13-16 week 4....

veeam never knows any better.. it sees same share/drive k: for say... its empty on week 2, it writes, it's empty on week 3. it writes.. etc etc..

never can touch original data till your sets returns to week 1.. tape 1-4.. then it sees it and can delete it if it does.. then you actually detect a loss of data, and remedy with backup of week -1 ....

no credentials can delete backups since they are on a data set not seen by veeam / network..

no ?


It's not bad; it's just not as good as tapes on a shelf because, given enough time, the attacker can gain the credentials to the machine managing the autoloader, and wipe your tapes. Unless your opsec is exceptional, and your tape manager is itself air gapped and you walk over to a console to manage it.

That said, the attacker could have been encrypting your backups all along, but with tapes on a shelf, you have a greater chance of detecting that by needing to do a restore before he can cycle through all the tapes.
yasuda
Enthusiast
 
Posts: 44
Liked: 9 times
Joined: Thu May 15, 2014 3:29 pm
Full Name: Peter Yasuda

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby tfelice » Mon Mar 27, 2017 8:25 pm

aporter wrote:One approach I haven't seen mentioned on this thread yet is to configure a pull using something like rsync, or perhaps another instance of Veeam, from your backup repository to another server which is on a different physical and logical network... This way nothing that is online would have access to the last resort archive server, but it would have access to pull from the primary backup repository (one way, read only).

I've used this approach. I think it has a lot of merit in some situations. As mentioned by cbc-tgschultz, it's critical to make sure that the credentials of the "last resort" machine cannot be obtained by keyloggers or other subterfuge. In my case, the host that is pulling backups is a Linux server. There is no Samba installation, so there is no relationship between the Linux box and the Windows domain. Thus, no compromised domain credentials or host, including the Veeam server, can access the "last resort" box. There are a lot of potential tools to use, but I happen to use use rsync. If the source repository is a Windows machine, you need to install an implementation of rsync on it. I use the native rsync protocol rather than ssh if I am copying across a LAN, so as avoid the unnecessary CPU overhead. You can use rsync's --compare-dest and --link-dest to economically create a directory for each backup iteration. You'd have to script the pull so that you get the retention schedule you wish, allowing yourself enough restore points to go back to a good backup.

For administration purposes I do access the "last resort" host via SSH, but only with an RSA key and passphrase and never with password only. Even key logging the passphrase doesn't get an intruder access without the private key. Since the last-resort backup job is pulled rather than pushed by the Veeam repository, then even if the Veeam server is compromised, there is no way for the bad guys to figure out how to access the last-resort host. There isn't even any local evidence that it exists.

One gotcha is that if you are using reverse incremental backups, Veeam by default renames the .VBK file every night. This would cause rsync (or anything trying to capture incremental file changes) to have to copy the entire .VBK every time, which would be expensive both in time and disk space. For a solution, see Veeam https://www.veeam.com/kb1076 on disabling the renaming of the VBK.
tfelice
Influencer
 
Posts: 10
Liked: never
Joined: Wed May 23, 2012 12:28 pm
Full Name: Tony Felice

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Gerald » Tue Mar 28, 2017 7:02 pm

I have so far only used the free Endpoint Backup (now called Veeam Agent) with SAMBA shares on a NAS as destination.

I deliberately don't grant the Windows User accounts access to the backup share, but the Veeam Client does of course need the authentication details so of course a crypto-trojan could read where Veeam stores this information (DB File? Registry? Config File?) and use it to access the share and delete/encrypt the backups also.

My question: Would I be more secure with a non-free Veeam Server instead of a NAS as backup target? When backing up to a Veeam Server instead of a NAS, who manages retention?
Does the client do this? If yes, then of course the client could also delete everything. If retention is configured and handled server-side only on a Veeam Backup server, then it would be more secure than a NAS, provided the l/p authentication for the server cannot be found on any one of the clients?
Gerald
Novice
 
Posts: 6
Liked: never
Joined: Fri Sep 04, 2015 5:56 pm
Full Name: Gerald

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: adapterer and 21 guests