Yes, Ransomware can delete your Veeam backups.

Availability for the Always-On Enterprise

Yes, Ransomware can delete your Veeam backups.

Veeam Logoby kbossak » Mon Mar 06, 2017 11:05 pm

On 2/7 we were hit with Samas Ransomware. Of course I freaked but I felt confident driving into work that I was ok with backups. I used Veeam to backup all my servers to two CIFS folders on 2 different Drobos on campus. We are a Private School with a small Tech budget and we get by with what we can. The backup location on each of them was a folder called Veeam_Backups and only 1 account had write access to the share and that same account was connected to it as a backup repository on the server.

The server itself got wiped with Samas, but I still felt confident. I looked in the Veeam_Backups folder a few times on both Drobos and both were empty, but I figured it was just a permission issue or something. I wasn't that worried.

I called Veeam support and the tech said he had never seen Ransomware delete any backups, so again I felt ok as I reinstalled Veeam on a new server.

Later that day we had a call with someone from the FBI (that we knew through an employee). He said he had never seen Ransomware delete backups, they usually encrypted them.

Fast forward a day when I'm on the phone with Veeam engineers getting my backup repositories reconnected.

As they reattached, I saw where they scanned and said 0 backups found. I knew at that point they were gone. The tech didn't believe me, which is fair. I was able to find config logs from my dead server, they verified in fact that Veeam had written over 200GB just 2 days before we were hit to both repositories.

So, yes, let me be the first to tell you, Ransomware can delete your Veeam backups, it can wipe out entire backup repositories.

I say this to only try and help as many people as I can. Get some offline backups of your files. Take the most important databases at your work, back them up, and send them to cloud storage. Write a script to change the extension of your important databases to .bak1 or .zip1 or whatever. Ransomware searches for file extensions, so you can mask your databases with this simple script. Samas didn't touch Windows 10, and it didn't touch Server 2012 R2. Malwarebytes was 1 of only 12 A/V programs that could still detect the variant we received 5 days after we got it. Microsoft Security Essentials that we used still couldn't detect it as of 2 weeks ago.

As an IT Director for over 14 years with 2 different companies, I always thought about backups as a means of protecting data from fires, or environmental disasters. I just never thought of it strongly from the Malware standpoint, thinking that onsite backups would be enough. Change your thinking, these are tough times for IT folks.

Feel free to ask if you have any follow up questions, I'm happy to provide info.
kbossak
Novice
 
Posts: 4
Liked: never
Joined: Mon Mar 06, 2017 10:48 pm
Full Name: Kerry Bossak

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Gostev » Mon Mar 06, 2017 11:09 pm 1 person likes this post

kbossak wrote:Get some offline backups of your files.

Ditto.
Gostev
Veeam Software
 
Posts: 21250
Liked: 2317 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby mkretzer » Tue Mar 07, 2017 5:39 am

@Gostev: Is there a way to disable all admin shares on a Veeam repo server? Or does Veeam use these?
mkretzer
Expert
 
Posts: 281
Liked: 61 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby shakala » Tue Mar 07, 2017 5:49 am

You are not the first... :|
shakala
Lurker
 
Posts: 1
Liked: never
Joined: Sat Jan 29, 2011 6:59 am
Full Name: Luigi

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Mike Resseler » Tue Mar 07, 2017 6:01 am

First, Kerry, I am really sorry to hear about this. If there are updates through your investigation with the FBI, please let us know. The more the FBI can research, the more change the security community has to work on countermeasures.

Just as an update. Samas is indeed a very painful variant of Ransomware. It used to target the healthcare vertical at first but it seems it is growing and attacking other verticals now also. It is known to search for backup files and basically can find backup files of most of the backup vendors and deletes them. The worse part is that the malware that does it deletes itself after this so it is difficult to trace it :-(

Get some offline backups of your files is indeed a way to go. Whether this is rotating devices, tapes, a cloud service provider. I do feel your pain though when you are a small shop. I have been there, done that as an IT admin and my solution against that was to take my tapes to me at home (with permission of the boss). It is not fun, but then there is a day you will be happy that you went through that pain...

@mkretzer look at the ports that you need in our guides. There is certainly different measures that you can take (a search on this forum should give you enough discussions on the topic) but in the end, it will give you a false sense of security. By the time you are done with those measures, there is a new variant of malware that will easily bypass your measures. Only when there is a "air-gapped" copy, you have a safe copy (for now, who knows what can be done in the future)

As an example, When researchers figured out how to battle cryptolocker version 2 (at least I believe it was 2) it took the bad guys 48 hours to write version 3 and we were back at the start... So as long as there is a connection, there is a danger. You can do your best to protect it as much as possible (and yes, please do so!) but have that additional air-gapped copy also... Just in case
Mike Resseler
Veeam Software
 
Posts: 2997
Liked: 354 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby lukejf » Tue Mar 07, 2017 9:56 am

That is a very scary post. Do you know who got the infection? Was it a domain admin. I would be interested to find out how it got to the storage if it wasn't an admin account without permission.
I always recommend tape to our customers and often get scoffed at for old technology. Helps me sleep at nigh that's for sure.
lukejf
Service Provider
 
Posts: 35
Liked: 2 times
Joined: Tue Jul 10, 2012 8:15 am
Full Name: Luke

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby lando_uk » Tue Mar 07, 2017 10:39 am

edit: by the looks of it, it seems like this particular nasty sits on your systems for a while, harvesting usernames/pws with key loggers and using a variety of other injected binaries, it has a human on the controls conducting reconnaissance of the whole infrastructure before striking. Offline tapes would have helped, cloud may not if admins regularly logged into the cloud provider with creds that could delete the cloud backups.
lando_uk
Expert
 
Posts: 235
Liked: 17 times
Joined: Thu Oct 17, 2013 10:02 am
Full Name: Mark

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby HendersonD » Tue Mar 07, 2017 12:20 pm

This post scared the hell out of me when I read it this morning. First, sorry to hear about the loss. I also work for a school district and funds are always tight.

Are there ways to do cloud backup that would prevent this? I ask because we use Veeam to backup from a Nimble array on one side of campus to a Nimble on the other side of campus. We also backup to an Exagrid offsite. The Nimble in our DR site and the Exagrid offsite are of course online sources. We have very strong passwords but now I am thinking of going the additional step off cloud based backups or tape. It sounds like even the cloud based backups could be an issue since they are online as well.
HendersonD
Enthusiast
 
Posts: 57
Liked: 3 times
Joined: Sat Jul 23, 2011 12:35 am

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby HendersonD » Tue Mar 07, 2017 12:30 pm

Does Veeam Cloud Connect help in this regard or because it is online can this type of Ransomware wipe it out as well?
What about this type of service
http://docs.aws.amazon.com/amazonglacie ... ction.html
HendersonD
Enthusiast
 
Posts: 57
Liked: 3 times
Joined: Sat Jul 23, 2011 12:35 am

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby v.Eremin » Tue Mar 07, 2017 12:49 pm

Does Veeam Cloud Connect help in this regard or because it is online can this type of Ransomware wipe it out as well?

Only if software is taught to utilize VB&R own commands, but it might as well be taught to erase online tapes and remove from catalog offline ones making restores significantly harder.

I don't think that such situation is realistic, though.
v.Eremin
Veeam Software
 
Posts: 12881
Liked: 936 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby kjstech » Tue Mar 07, 2017 12:59 pm

If the backup repository is connected via SSH like Veeam to Exagrid for example, could this malware still get to it? I could certainly understand CIFS/SMB, but is the malware becoming os independent enough to cross that Windows boundary over to Linux based?

Is it worth Veeam's research and development to make a ovf deployable Veeam Linux based virtual appliance version of Veeam Backup and Recovery? The only thing I see that getting you is some isolation from windows based malware, and as long as its repositories are also hosted on non-windows systems / protocols maybe that could help?
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby tdewin » Tue Mar 07, 2017 1:49 pm 1 person likes this post

Well you could use almost any Linux distribution to run as a repository. For example Centos or Ubuntu. The requirements are quite minimal. For example, there is a Centos 7 discussion here:
vmware-vsphere-f24/using-a-centos-7-linux-as-a-backup-repository-how-to-t26092.html

Of course malware could become smarter to be cross platform but it does add another layer of "complexity". Again if you want to be 100% sure, put it on tape, get it out of the library and create an air gap between your network and your backups
tdewin
Veeam Software
 
Posts: 1020
Liked: 351 times
Joined: Fri Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby kbossak » Tue Mar 07, 2017 4:31 pm

I don't think it can cross over into Linnux. None of our ESXi servers were effected, just the file types within the VMDK's.

As I mentioned, it didn't touch our Server 2012 R2 servers which happened to be both of our domain controllers, so that was very helpful in rebuilding.

I connected a cloud storage provider to Veeam last night, and I'm working on connecting our old LTO5 tape drive to one of our ESxi servers.
kbossak
Novice
 
Posts: 4
Liked: never
Joined: Mon Mar 06, 2017 10:48 pm
Full Name: Kerry Bossak

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby dellock6 » Tue Mar 07, 2017 10:51 pm 2 people like this post

HendersonD wrote:Are there ways to do cloud backup that would prevent this?

We are seeing several service providers offering air-gapped copies of Veeam Cloud Connect backups, so that even if a hacker gets the credentials to login to your VCC tenant and delete the backups, there's another copy of the same backup that is offline and can be used. It's not stored on premises, and it's offline.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 4934
Liked: 1293 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby MOBO » Wed Mar 08, 2017 7:59 am 1 person likes this post

how about if the repository is a standalone server and veeam program is the only system that has the credentials , am i still at risk for malware to delete backups ?
MOBO
Influencer
 
Posts: 10
Liked: 2 times
Joined: Sat Jan 24, 2015 7:26 am
Full Name: Morten Boegeskov

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: No registered users and 34 guests