Availability for the Always-On Enterprise
Post Reply
kbossak
Novice
Posts: 5
Liked: never
Joined: Mar 06, 2017 10:48 pm
Full Name: Kerry Bossak
Contact:

Yes, Ransomware can delete your Veeam backups.

Post by kbossak » Mar 06, 2017 11:05 pm

On 2/7 we were hit with Samas Ransomware. Of course I freaked but I felt confident driving into work that I was ok with backups. I used Veeam to backup all my servers to two CIFS folders on 2 different Drobos on campus. We are a Private School with a small Tech budget and we get by with what we can. The backup location on each of them was a folder called Veeam_Backups and only 1 account had write access to the share and that same account was connected to it as a backup repository on the server.

The server itself got wiped with Samas, but I still felt confident. I looked in the Veeam_Backups folder a few times on both Drobos and both were empty, but I figured it was just a permission issue or something. I wasn't that worried.

I called Veeam support and the tech said he had never seen Ransomware delete any backups, so again I felt ok as I reinstalled Veeam on a new server.

Later that day we had a call with someone from the FBI (that we knew through an employee). He said he had never seen Ransomware delete backups, they usually encrypted them.

Fast forward a day when I'm on the phone with Veeam engineers getting my backup repositories reconnected.

As they reattached, I saw where they scanned and said 0 backups found. I knew at that point they were gone. The tech didn't believe me, which is fair. I was able to find config logs from my dead server, they verified in fact that Veeam had written over 200GB just 2 days before we were hit to both repositories.

So, yes, let me be the first to tell you, Ransomware can delete your Veeam backups, it can wipe out entire backup repositories.

I say this to only try and help as many people as I can. Get some offline backups of your files. Take the most important databases at your work, back them up, and send them to cloud storage. Write a script to change the extension of your important databases to .bak1 or .zip1 or whatever. Ransomware searches for file extensions, so you can mask your databases with this simple script. Samas didn't touch Windows 10, and it didn't touch Server 2012 R2. Malwarebytes was 1 of only 12 A/V programs that could still detect the variant we received 5 days after we got it. Microsoft Security Essentials that we used still couldn't detect it as of 2 weeks ago.

As an IT Director for over 14 years with 2 different companies, I always thought about backups as a means of protecting data from fires, or environmental disasters. I just never thought of it strongly from the Malware standpoint, thinking that onsite backups would be enough. Change your thinking, these are tough times for IT folks.

Feel free to ask if you have any follow up questions, I'm happy to provide info.

Gostev
Veeam Software
Posts: 23116
Liked: 2917 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Gostev » Mar 06, 2017 11:09 pm 1 person likes this post

kbossak wrote:Get some offline backups of your files.
Ditto.

mkretzer
Expert
Posts: 403
Liked: 80 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by mkretzer » Mar 07, 2017 5:39 am

@Gostev: Is there a way to disable all admin shares on a Veeam repo server? Or does Veeam use these?

shakala
Lurker
Posts: 1
Liked: never
Joined: Jan 29, 2011 6:59 am
Full Name: Luigi
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by shakala » Mar 07, 2017 5:49 am

You are not the first... :|

Mike Resseler
Veeam Software
Posts: 4993
Liked: 528 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Mike Resseler » Mar 07, 2017 6:01 am

First, Kerry, I am really sorry to hear about this. If there are updates through your investigation with the FBI, please let us know. The more the FBI can research, the more change the security community has to work on countermeasures.

Just as an update. Samas is indeed a very painful variant of Ransomware. It used to target the healthcare vertical at first but it seems it is growing and attacking other verticals now also. It is known to search for backup files and basically can find backup files of most of the backup vendors and deletes them. The worse part is that the malware that does it deletes itself after this so it is difficult to trace it :-(

Get some offline backups of your files is indeed a way to go. Whether this is rotating devices, tapes, a cloud service provider. I do feel your pain though when you are a small shop. I have been there, done that as an IT admin and my solution against that was to take my tapes to me at home (with permission of the boss). It is not fun, but then there is a day you will be happy that you went through that pain...

@mkretzer look at the ports that you need in our guides. There is certainly different measures that you can take (a search on this forum should give you enough discussions on the topic) but in the end, it will give you a false sense of security. By the time you are done with those measures, there is a new variant of malware that will easily bypass your measures. Only when there is a "air-gapped" copy, you have a safe copy (for now, who knows what can be done in the future)

As an example, When researchers figured out how to battle cryptolocker version 2 (at least I believe it was 2) it took the bad guys 48 hours to write version 3 and we were back at the start... So as long as there is a connection, there is a danger. You can do your best to protect it as much as possible (and yes, please do so!) but have that additional air-gapped copy also... Just in case

lukejf
Service Provider
Posts: 54
Liked: 2 times
Joined: Jul 10, 2012 8:15 am
Full Name: Luke
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by lukejf » Mar 07, 2017 9:56 am

That is a very scary post. Do you know who got the infection? Was it a domain admin. I would be interested to find out how it got to the storage if it wasn't an admin account without permission.
I always recommend tape to our customers and often get scoffed at for old technology. Helps me sleep at nigh that's for sure.

lando_uk
Expert
Posts: 282
Liked: 21 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by lando_uk » Mar 07, 2017 10:39 am

edit: by the looks of it, it seems like this particular nasty sits on your systems for a while, harvesting usernames/pws with key loggers and using a variety of other injected binaries, it has a human on the controls conducting reconnaissance of the whole infrastructure before striking. Offline tapes would have helped, cloud may not if admins regularly logged into the cloud provider with creds that could delete the cloud backups.

HendersonD
Enthusiast
Posts: 99
Liked: 4 times
Joined: Jul 23, 2011 12:35 am

Re: Yes, Ransomware can delete your Veeam backups.

Post by HendersonD » Mar 07, 2017 12:20 pm

This post scared the hell out of me when I read it this morning. First, sorry to hear about the loss. I also work for a school district and funds are always tight.

Are there ways to do cloud backup that would prevent this? I ask because we use Veeam to backup from a Nimble array on one side of campus to a Nimble on the other side of campus. We also backup to an Exagrid offsite. The Nimble in our DR site and the Exagrid offsite are of course online sources. We have very strong passwords but now I am thinking of going the additional step off cloud based backups or tape. It sounds like even the cloud based backups could be an issue since they are online as well.

HendersonD
Enthusiast
Posts: 99
Liked: 4 times
Joined: Jul 23, 2011 12:35 am

Re: Yes, Ransomware can delete your Veeam backups.

Post by HendersonD » Mar 07, 2017 12:30 pm

Does Veeam Cloud Connect help in this regard or because it is online can this type of Ransomware wipe it out as well?
What about this type of service
http://docs.aws.amazon.com/amazonglacie ... ction.html

v.Eremin
Veeam Software
Posts: 15234
Liked: 1146 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by v.Eremin » Mar 07, 2017 12:49 pm

Does Veeam Cloud Connect help in this regard or because it is online can this type of Ransomware wipe it out as well?
Only if software is taught to utilize VB&R own commands, but it might as well be taught to erase online tapes and remove from catalog offline ones making restores significantly harder.

I don't think that such situation is realistic, though.

kjstech
Expert
Posts: 148
Liked: 14 times
Joined: Jan 17, 2014 4:12 pm
Full Name: Keith S
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by kjstech » Mar 07, 2017 12:59 pm

If the backup repository is connected via SSH like Veeam to Exagrid for example, could this malware still get to it? I could certainly understand CIFS/SMB, but is the malware becoming os independent enough to cross that Windows boundary over to Linux based?

Is it worth Veeam's research and development to make a ovf deployable Veeam Linux based virtual appliance version of Veeam Backup and Recovery? The only thing I see that getting you is some isolation from windows based malware, and as long as its repositories are also hosted on non-windows systems / protocols maybe that could help?

tdewin
Veeam Software
Posts: 1338
Liked: 439 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by tdewin » Mar 07, 2017 1:49 pm 1 person likes this post

Well you could use almost any Linux distribution to run as a repository. For example Centos or Ubuntu. The requirements are quite minimal. For example, there is a Centos 7 discussion here:
vmware-vsphere-f24/using-a-centos-7-lin ... 26092.html

Of course malware could become smarter to be cross platform but it does add another layer of "complexity". Again if you want to be 100% sure, put it on tape, get it out of the library and create an air gap between your network and your backups

kbossak
Novice
Posts: 5
Liked: never
Joined: Mar 06, 2017 10:48 pm
Full Name: Kerry Bossak
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by kbossak » Mar 07, 2017 4:31 pm

I don't think it can cross over into Linnux. None of our ESXi servers were effected, just the file types within the VMDK's.

As I mentioned, it didn't touch our Server 2012 R2 servers which happened to be both of our domain controllers, so that was very helpful in rebuilding.

I connected a cloud storage provider to Veeam last night, and I'm working on connecting our old LTO5 tape drive to one of our ESxi servers.

dellock6
Veeam Software
Posts: 5511
Liked: 1524 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by dellock6 » Mar 07, 2017 10:51 pm 2 people like this post

HendersonD wrote:Are there ways to do cloud backup that would prevent this?
We are seeing several service providers offering air-gapped copies of Veeam Cloud Connect backups, so that even if a hacker gets the credentials to login to your VCC tenant and delete the backups, there's another copy of the same backup that is offline and can be used. It's not stored on premises, and it's offline.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com/en/
vExpert 2011-2012-2013-2014-2015-2016-2017-2018
Veeam VMCE #1

MOBO
Influencer
Posts: 14
Liked: 3 times
Joined: Jan 24, 2015 7:26 am
Full Name: Morten Boegeskov
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by MOBO » Mar 08, 2017 7:59 am 1 person likes this post

how about if the repository is a standalone server and veeam program is the only system that has the credentials , am i still at risk for malware to delete backups ?

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 36 guests