Comprehensive data protection for all workloads
kbossak
Novice
Posts: 6
Liked: never
Joined: Mar 06, 2017 10:48 pm
Full Name: Kerry Bossak
Contact:

Yes, Ransomware can delete your Veeam backups.

Post by kbossak » Mar 06, 2017 11:05 pm

On 2/7 we were hit with Samas Ransomware. Of course I freaked but I felt confident driving into work that I was ok with backups. I used Veeam to backup all my servers to two CIFS folders on 2 different Drobos on campus. We are a Private School with a small Tech budget and we get by with what we can. The backup location on each of them was a folder called Veeam_Backups and only 1 account had write access to the share and that same account was connected to it as a backup repository on the server.

The server itself got wiped with Samas, but I still felt confident. I looked in the Veeam_Backups folder a few times on both Drobos and both were empty, but I figured it was just a permission issue or something. I wasn't that worried.

I called Veeam support and the tech said he had never seen Ransomware delete any backups, so again I felt ok as I reinstalled Veeam on a new server.

Later that day we had a call with someone from the FBI (that we knew through an employee). He said he had never seen Ransomware delete backups, they usually encrypted them.

Fast forward a day when I'm on the phone with Veeam engineers getting my backup repositories reconnected.

As they reattached, I saw where they scanned and said 0 backups found. I knew at that point they were gone. The tech didn't believe me, which is fair. I was able to find config logs from my dead server, they verified in fact that Veeam had written over 200GB just 2 days before we were hit to both repositories.

So, yes, let me be the first to tell you, Ransomware can delete your Veeam backups, it can wipe out entire backup repositories.

I say this to only try and help as many people as I can. Get some offline backups of your files. Take the most important databases at your work, back them up, and send them to cloud storage. Write a script to change the extension of your important databases to .bak1 or .zip1 or whatever. Ransomware searches for file extensions, so you can mask your databases with this simple script. Samas didn't touch Windows 10, and it didn't touch Server 2012 R2. Malwarebytes was 1 of only 12 A/V programs that could still detect the variant we received 5 days after we got it. Microsoft Security Essentials that we used still couldn't detect it as of 2 weeks ago.

As an IT Director for over 14 years with 2 different companies, I always thought about backups as a means of protecting data from fires, or environmental disasters. I just never thought of it strongly from the Malware standpoint, thinking that onsite backups would be enough. Change your thinking, these are tough times for IT folks.

Feel free to ask if you have any follow up questions, I'm happy to provide info.

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Gostev » Mar 06, 2017 11:09 pm 1 person likes this post

kbossak wrote:Get some offline backups of your files.
Ditto.

Shared folders are just too easy to wipe remotely in general due to their content easily accessible via the standard SMB protocol. So, using ANY other repository type out of many that Veeam supports is safer, because it would require ransomware to implement proprietary Veeam protocols to access backup files sitting there.

In particular, since vast majority of ransomware is Windows-based, using a Linux-based backup repository adds an extra layer of protection.

mkretzer
Expert
Posts: 566
Liked: 127 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by mkretzer » Mar 07, 2017 5:39 am

@Gostev: Is there a way to disable all admin shares on a Veeam repo server? Or does Veeam use these?

shakala
Lurker
Posts: 1
Liked: never
Joined: Jan 29, 2011 6:59 am
Full Name: Luigi
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by shakala » Mar 07, 2017 5:49 am

You are not the first... :|

Mike Resseler
Product Manager
Posts: 5760
Liked: 616 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Mike Resseler » Mar 07, 2017 6:01 am

First, Kerry, I am really sorry to hear about this. If there are updates through your investigation with the FBI, please let us know. The more the FBI can research, the more change the security community has to work on countermeasures.

Just as an update. Samas is indeed a very painful variant of Ransomware. It used to target the healthcare vertical at first but it seems it is growing and attacking other verticals now also. It is known to search for backup files and basically can find backup files of most of the backup vendors and deletes them. The worse part is that the malware that does it deletes itself after this so it is difficult to trace it :-(

Get some offline backups of your files is indeed a way to go. Whether this is rotating devices, tapes, a cloud service provider. I do feel your pain though when you are a small shop. I have been there, done that as an IT admin and my solution against that was to take my tapes to me at home (with permission of the boss). It is not fun, but then there is a day you will be happy that you went through that pain...

@mkretzer look at the ports that you need in our guides. There is certainly different measures that you can take (a search on this forum should give you enough discussions on the topic) but in the end, it will give you a false sense of security. By the time you are done with those measures, there is a new variant of malware that will easily bypass your measures. Only when there is a "air-gapped" copy, you have a safe copy (for now, who knows what can be done in the future)

As an example, When researchers figured out how to battle cryptolocker version 2 (at least I believe it was 2) it took the bad guys 48 hours to write version 3 and we were back at the start... So as long as there is a connection, there is a danger. You can do your best to protect it as much as possible (and yes, please do so!) but have that additional air-gapped copy also... Just in case

lukejf
Service Provider
Posts: 54
Liked: 3 times
Joined: Jul 10, 2012 8:15 am
Full Name: Luke
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by lukejf » Mar 07, 2017 9:56 am 1 person likes this post

That is a very scary post. Do you know who got the infection? Was it a domain admin. I would be interested to find out how it got to the storage if it wasn't an admin account without permission.
I always recommend tape to our customers and often get scoffed at for old technology. Helps me sleep at nigh that's for sure.

lando_uk
Expert
Posts: 312
Liked: 27 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by lando_uk » Mar 07, 2017 10:39 am

edit: by the looks of it, it seems like this particular nasty sits on your systems for a while, harvesting usernames/pws with key loggers and using a variety of other injected binaries, it has a human on the controls conducting reconnaissance of the whole infrastructure before striking. Offline tapes would have helped, cloud may not if admins regularly logged into the cloud provider with creds that could delete the cloud backups.

HendersonD
Expert
Posts: 120
Liked: 4 times
Joined: Jul 23, 2011 12:35 am

Re: Yes, Ransomware can delete your Veeam backups.

Post by HendersonD » Mar 07, 2017 12:20 pm

This post scared the hell out of me when I read it this morning. First, sorry to hear about the loss. I also work for a school district and funds are always tight.

Are there ways to do cloud backup that would prevent this? I ask because we use Veeam to backup from a Nimble array on one side of campus to a Nimble on the other side of campus. We also backup to an Exagrid offsite. The Nimble in our DR site and the Exagrid offsite are of course online sources. We have very strong passwords but now I am thinking of going the additional step off cloud based backups or tape. It sounds like even the cloud based backups could be an issue since they are online as well.

HendersonD
Expert
Posts: 120
Liked: 4 times
Joined: Jul 23, 2011 12:35 am

Re: Yes, Ransomware can delete your Veeam backups.

Post by HendersonD » Mar 07, 2017 12:30 pm

Does Veeam Cloud Connect help in this regard or because it is online can this type of Ransomware wipe it out as well?
What about this type of service
http://docs.aws.amazon.com/amazonglacie ... ction.html

veremin
Product Manager
Posts: 17005
Liked: 1455 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by veremin » Mar 07, 2017 12:49 pm

Does Veeam Cloud Connect help in this regard or because it is online can this type of Ransomware wipe it out as well?
Only if software is taught to utilize VB&R own commands, but it might as well be taught to erase online tapes and remove from catalog offline ones making restores significantly harder.

I don't think that such situation is realistic, though.

kjstech
Expert
Posts: 148
Liked: 14 times
Joined: Jan 17, 2014 4:12 pm
Full Name: Keith S
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by kjstech » Mar 07, 2017 12:59 pm

If the backup repository is connected via SSH like Veeam to Exagrid for example, could this malware still get to it? I could certainly understand CIFS/SMB, but is the malware becoming os independent enough to cross that Windows boundary over to Linux based?

Is it worth Veeam's research and development to make a ovf deployable Veeam Linux based virtual appliance version of Veeam Backup and Recovery? The only thing I see that getting you is some isolation from windows based malware, and as long as its repositories are also hosted on non-windows systems / protocols maybe that could help?

tdewin
Veeam Software
Posts: 1447
Liked: 462 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by tdewin » Mar 07, 2017 1:49 pm 1 person likes this post

Well you could use almost any Linux distribution to run as a repository. For example Centos or Ubuntu. The requirements are quite minimal. For example, there is a Centos 7 discussion here:
vmware-vsphere-f24/using-a-centos-7-lin ... 26092.html

Of course malware could become smarter to be cross platform but it does add another layer of "complexity". Again if you want to be 100% sure, put it on tape, get it out of the library and create an air gap between your network and your backups

kbossak
Novice
Posts: 6
Liked: never
Joined: Mar 06, 2017 10:48 pm
Full Name: Kerry Bossak
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by kbossak » Mar 07, 2017 4:31 pm

I don't think it can cross over into Linnux. None of our ESXi servers were effected, just the file types within the VMDK's.

As I mentioned, it didn't touch our Server 2012 R2 servers which happened to be both of our domain controllers, so that was very helpful in rebuilding.

I connected a cloud storage provider to Veeam last night, and I'm working on connecting our old LTO5 tape drive to one of our ESxi servers.

dellock6
Veeam Software
Posts: 5753
Liked: 1644 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by dellock6 » Mar 07, 2017 10:51 pm 2 people like this post

HendersonD wrote:Are there ways to do cloud backup that would prevent this?
We are seeing several service providers offering air-gapped copies of Veeam Cloud Connect backups, so that even if a hacker gets the credentials to login to your VCC tenant and delete the backups, there's another copy of the same backup that is offline and can be used. It's not stored on premises, and it's offline.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

MOBO
Influencer
Posts: 15
Liked: 5 times
Joined: Jan 24, 2015 7:26 am
Full Name: Morten Boegeskov
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by MOBO » Mar 08, 2017 7:59 am 1 person likes this post

how about if the repository is a standalone server and veeam program is the only system that has the credentials , am i still at risk for malware to delete backups ?

Mike Resseler
Product Manager
Posts: 5760
Liked: 616 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Mike Resseler » Mar 08, 2017 8:02 am

Morten,

Unfortunately yes... You make the risk very small, but the risk is there. Ransomware really is getting very good. Maybe you will be safe for a while, but they adapt their stuff pretty quickly. I've read in this thread also somewhere to use Linux as the typical Windows Ransomware cannot touch that but there were already attempts (OK, bad ones but still) in creating those for the Linux platform so air-gap is really the only solution at this moment which can be considered safe.

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Gostev » Mar 08, 2017 2:10 pm

There are plenty of Linux ransomware these days... KillDisk and FairWare come to my mind from the recent ones.

Mike Resseler
Product Manager
Posts: 5760
Liked: 616 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Mike Resseler » Mar 08, 2017 4:35 pm

True, luckily some of them are not that good and can be decrypted (albeit difficult) but it is just a matter of time before they will get worse :-(

larry
Expert
Posts: 387
Liked: 92 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by larry » Mar 08, 2017 6:35 pm 2 people like this post

I have a powershell script that will email me if Veeam backup files go missing or renamed. I run it every hour as a task. You will need to have a c:\temp directory.
if you run as a different user than the task be sure to delete temp files. first run it will create baseline after that it just runs. A file change of 50 percent in a repository trips the email. not pretty code as I never planned on sharing. Normally I use it to monitor all user directories for big changes in case of rasomware or employee quitting. Hope it helps someone.

Code: Select all

Function SendEmail
{

# Email configuration

$emailHost = "exchange"
$emailUser = ""
$emailPass = ""
$emailFrom = "FileCountAlert@chelseagroton.com"
$emailTo = "lxxxxx@xxxxroton.com"
# Send report as attachment - $true or $false
$emailAttach = $true
# Email Subject 
$emailSubject = "Large File Change"

If ($sendEmail) {
        $smtp = New-Object System.Net.Mail.SmtpClient $emailHost
$smtp.port = 25
#port 26 because mcafee blocks 25 on win 7 Exception calling "Send" with "1" argument(s): "Failure sending mail
        $smtp.Credentials = New-Object System.Net.NetworkCredential($emailUser, $emailPass);
$msg = New-Object System.Net.Mail.MailMessage($emailFrom, $emailTo)
$msg.Subject = $emailSubject
If ($emailAttach) {
#$body = "Report Attached"
                      $body = $htmlOutput
$msg.Body = $body
                        $msg.isBodyhtml = $true
$tempfile = "$env:TEMP\$emailSubject.htm"
$htmlOutput | Out-File $tempfile
$attachment = new-object System.Net.Mail.Attachment $tempfile
      $msg.Attachments.Add($attachment)

} Else {
$body = $htmlOutput
$msg.Body = $body
$msg.isBodyhtml = $true
}       
        $smtp.send($msg)
If ($emailAttach) {
$attachment.dispose()
Remove-Item $tempfile
  }
    
       }
} #email Function

$sendEmail = $false
$pathArray = @("E:\backup","E:\Backups","F:\Backups","g:\Backups","h:\Backups","t:\Backups")
$Dirlist = @()

#get dir names
for ($i=0; $i -lt $pathArray.length; $i++) { 
$originalPath =$pathArray[$i]
$tempDirlist = Get-ChildItem $originalPath\* | ?{ $_.PSIsContainer } | Select-Object FullName
$Dirlist += $tempDirlist
}

$OutputArray = @()


$File="C:\temp\content.txt"
if (Test-Path $File) 
{$DirList2 = import-clixml $File  }
else
{ 
#should only happen on install or first run
$DirList2 = $Dirlist 
}

foreach ($i in $Dirlist)
{
$originalPath = $I.FullName
#$filecount = (Get-ChildItem $originalPath\* -Include *.gif, *.jpg, *.xls*, *.doc*,*.png*, *.pdf*, *.wav*, .ppt* -recurse).count
#top one was to see if docs got encrypted or deleted, below switched to veeam backups
$filecount = (Get-ChildItem $originalPath\* -Include *.vbk, *.vrb, *.vbm*, *.vib -recurse).count

$I |Add-Member -MemberType NoteProperty -Name 'files' -Value ($filecount)
$I |Add-Member -MemberType NoteProperty -Name 'Note' -Value ("")
}


$File="C:\temp\content.txt"
if (Test-Path $File) 
{$DirList2 = import-clixml $File  }
else
{ 
#should only happen on install or first run
$DirList2 = $Dirlist 
}



$compare = compare-object $DirList2 $Dirlist  -Property Fullname,files
$compare = $compare | sort FullName


$diffCount = $compare.Count
foreach ($i in $compare)
  {
     $i |Add-Member -MemberType NoteProperty -Name 'Note' -Value ("")
   }
$compare | foreach  { 
      if ($_.sideindicator -eq '<=')
        {$_.sideindicator = "Was"}

      if ($_.sideindicator -eq '=>')
        {$_.sideindicator = "Is"}
     }

$body = $compare| ConvertTo-HTML -fragment
For ($i=0; $i -lt $diffCount; $i++) {
    
   if ($compare[$i].FullName -eq $compare[$i + 1 ].FullName)
    {
       write-host $compare[$i].FullName
       $diff = $compare[$i].files - $compare[$i + 1].files
         $t = $i 
         $t2 = $i +1 
         # var t is used in math not sure which order
        if ($compare[$i].SideIndicator -eq "Was" )
           {$diff = $diff * -1
           $t = $i +1
           $t2 = $I
            }

         write-host $diff
    #lt 5 files who cares
        if ($diff -lt 5 )
           {$percentDown = 100 - (($compare[$t].files + 1)/($compare[$t2].files +1) * 100)
     # add one to not worry div zero
             Write-host "File Count went down " $percentDown

            }
        if ($percentDown -gt 50 )
            {
              $sendEmail = $true
              Write-host "File Count went down more than 50 percent"
              $compare[$i].Note = "File Count went down more than 50 percent"
             }

#write-host $c[$i + 1].FullName

     }
 }

$body = $compare| ConvertTo-HTML -fragment

# HTML Stuff
$headerObj = @"
<html>
        <head>
        <center>
           <h1>Warning Veeam File Changes  </h1> 
           <h3>Report generated at $(Get-Date -format g) on $((gc env:computername).ToLower()) </h3>   
       </center>
        </head>
"@
 
$bodyTop = @"
        <body>
               
"@
 


$footerObj = @"
</body>
</html>
"@


#$htmlOutput = $headerObj + $bodyTop + $body + $filelist + $footerObj
$htmlOutput = $headerObj + $bodyTop + $body + $footerObj

$Dirlist |  Select-Object FullName, files | Export-Clixml -Path $File
$htmlOutput > c:\temp\test.html
SendEmail

csinetops
Expert
Posts: 113
Liked: 15 times
Joined: Jun 06, 2014 2:45 pm
Full Name: csinetops
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by csinetops » Mar 08, 2017 9:21 pm

Is there a reason why the backup servers need to talk to anything other than the other backup servers,DC's and VCenter? I was thinking of using IP security policies to lock my Veeam servers down to only talk to each other,the DC's and Vcenter.

So far all I can thing we'd lose out on is failover to network mode ( we use hot add).

Does Veeam need to talk to the production servers in a restore scenario? IE, restoring a file using FLR or Exchange item?

Am I missing anything?

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Gostev » Mar 08, 2017 11:20 pm

Yes, it needs to talk to production servers to restore files and application items there. Although for files specifically, we don't need the direct network connection because we can also inject them through the host connection (albeit much slower).

frankj
Service Provider
Posts: 29
Liked: 2 times
Joined: May 27, 2016 4:53 pm
Full Name: FRANK Jacques
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by frankj » Mar 12, 2017 11:23 pm

you have this part in double is that normal ?

Code: Select all

$File="C:\temp\content.txt"
if (Test-Path $File) {
    $DirList2 = import-clixml $File  
} else { 
    #should only happen on install or first run
    $DirList2 = $Dirlist 
}
larry wrote:I have a powershell script that will email me if Veeam backup files go missing or renamed. I run it every hour as a task. You will need to have a c:\temp directory.
if you run as a different user than the task be sure to delete temp files. first run it will create baseline after that it just runs. A file change of 50 percent in a repository trips the email. not pretty code as I never planned on sharing. Normally I use it to monitor all user directories for big changes in case of rasomware or employee quitting. Hope it helps someone.

Code: Select all

Function SendEmail
{

# Email configuration

$emailHost = "exchange"
$emailUser = ""
$emailPass = ""
$emailFrom = "FileCountAlert@chelseagroton.com"
$emailTo = "lxxxxx@xxxxroton.com"
# Send report as attachment - $true or $false
$emailAttach = $true
# Email Subject 
$emailSubject = "Large File Change"

If ($sendEmail) {
        $smtp = New-Object System.Net.Mail.SmtpClient $emailHost
$smtp.port = 25
#port 26 because mcafee blocks 25 on win 7 Exception calling "Send" with "1" argument(s): "Failure sending mail
        $smtp.Credentials = New-Object System.Net.NetworkCredential($emailUser, $emailPass);
$msg = New-Object System.Net.Mail.MailMessage($emailFrom, $emailTo)
$msg.Subject = $emailSubject
If ($emailAttach) {
#$body = "Report Attached"
                      $body = $htmlOutput
$msg.Body = $body
                        $msg.isBodyhtml = $true
$tempfile = "$env:TEMP\$emailSubject.htm"
$htmlOutput | Out-File $tempfile
$attachment = new-object System.Net.Mail.Attachment $tempfile
      $msg.Attachments.Add($attachment)

} Else {
$body = $htmlOutput
$msg.Body = $body
$msg.isBodyhtml = $true
}       
        $smtp.send($msg)
If ($emailAttach) {
$attachment.dispose()
Remove-Item $tempfile
  }
    
       }
} #email Function

$sendEmail = $false
$pathArray = @("E:\backup","E:\Backups","F:\Backups","g:\Backups","h:\Backups","t:\Backups")
$Dirlist = @()

#get dir names
for ($i=0; $i -lt $pathArray.length; $i++) { 
$originalPath =$pathArray[$i]
$tempDirlist = Get-ChildItem $originalPath\* | ?{ $_.PSIsContainer } | Select-Object FullName
$Dirlist += $tempDirlist
}

$OutputArray = @()


$File="C:\temp\content.txt"
if (Test-Path $File) 
{$DirList2 = import-clixml $File  }
else
{ 
#should only happen on install or first run
$DirList2 = $Dirlist 
}

foreach ($i in $Dirlist)
{
$originalPath = $I.FullName
#$filecount = (Get-ChildItem $originalPath\* -Include *.gif, *.jpg, *.xls*, *.doc*,*.png*, *.pdf*, *.wav*, .ppt* -recurse).count
#top one was to see if docs got encrypted or deleted, below switched to veeam backups
$filecount = (Get-ChildItem $originalPath\* -Include *.vbk, *.vrb, *.vbm*, *.vib -recurse).count

$I |Add-Member -MemberType NoteProperty -Name 'files' -Value ($filecount)
$I |Add-Member -MemberType NoteProperty -Name 'Note' -Value ("")
}


$File="C:\temp\content.txt"
if (Test-Path $File) 
{$DirList2 = import-clixml $File  }
else
{ 
#should only happen on install or first run
$DirList2 = $Dirlist 
}



$compare = compare-object $DirList2 $Dirlist  -Property Fullname,files
$compare = $compare | sort FullName


$diffCount = $compare.Count
foreach ($i in $compare)
  {
     $i |Add-Member -MemberType NoteProperty -Name 'Note' -Value ("")
   }
$compare | foreach  { 
      if ($_.sideindicator -eq '<=')
        {$_.sideindicator = "Was"}

      if ($_.sideindicator -eq '=>')
        {$_.sideindicator = "Is"}
     }

$body = $compare| ConvertTo-HTML -fragment
For ($i=0; $i -lt $diffCount; $i++) {
    
   if ($compare[$i].FullName -eq $compare[$i + 1 ].FullName)
    {
       write-host $compare[$i].FullName
       $diff = $compare[$i].files - $compare[$i + 1].files
         $t = $i 
         $t2 = $i +1 
         # var t is used in math not sure which order
        if ($compare[$i].SideIndicator -eq "Was" )
           {$diff = $diff * -1
           $t = $i +1
           $t2 = $I
            }

         write-host $diff
    #lt 5 files who cares
        if ($diff -lt 5 )
           {$percentDown = 100 - (($compare[$t].files + 1)/($compare[$t2].files +1) * 100)
     # add one to not worry div zero
             Write-host "File Count went down " $percentDown

            }
        if ($percentDown -gt 50 )
            {
              $sendEmail = $true
              Write-host "File Count went down more than 50 percent"
              $compare[$i].Note = "File Count went down more than 50 percent"
             }

#write-host $c[$i + 1].FullName

     }
 }

$body = $compare| ConvertTo-HTML -fragment

# HTML Stuff
$headerObj = @"
<html>
        <head>
        <center>
           <h1>Warning Veeam File Changes  </h1> 
           <h3>Report generated at $(Get-Date -format g) on $((gc env:computername).ToLower()) </h3>   
       </center>
        </head>
"@
 
$bodyTop = @"
        <body>
               
"@
 


$footerObj = @"
</body>
</html>
"@


#$htmlOutput = $headerObj + $bodyTop + $body + $filelist + $footerObj
$htmlOutput = $headerObj + $bodyTop + $body + $footerObj

$Dirlist |  Select-Object FullName, files | Export-Clixml -Path $File
$htmlOutput > c:\temp\test.html
SendEmail

tacioandrade
Enthusiast
Posts: 29
Liked: 3 times
Joined: Nov 17, 2016 2:04 am
Full Name: Tácio Andrade
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by tacioandrade » Mar 12, 2017 11:27 pm

Without a doubt, backups made with Veeam can be encrypted, so much so that I contacted the Samba developers to see if there was something like Veto Files, however that would work in reverse, a Allow Files, releasing the writing files only Of the released extensions (to record my Veeam backups and other solutions in this directory).

However, they said that this functionality was not done and that if someone created it and did a pull to the samba repository, it would certainly be accepted, but the developers themselves would not program it.

A pity as I see this, because without this feature, I believe the only real way to solve this problem is as a friend has done in FreeNAS, mounting different LUNs for different days on the same mount point, so that if one of them Be committed, the other is not.


Sincerely, Tácio Andrade.

frankj
Service Provider
Posts: 29
Liked: 2 times
Joined: May 27, 2016 4:53 pm
Full Name: FRANK Jacques
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by frankj » Mar 12, 2017 11:30 pm

woudnt a user that is veeam credentials only having r/w and global admins just read be a way ?

this way an infected server that is affected coudnt touch those repos right ? unless it's the veeam user account ?

AlexLeadingEdge
Expert
Posts: 323
Liked: 39 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by AlexLeadingEdge » Mar 13, 2017 1:43 am

One of our clients were hit by the cryptolocker virus a few months back, it wiped out a days worth of work. The thing that saved them was that they rotate their USB drives every morning, there are five of them for each day of the working week, and they're stored in a fire-proof safe. The cryptolocker virus could only encrypt that which was physically attached, so the other air-gapped USB backup drives were all fine.

One of the reasons we have at a bare minimum three USB drives is because users will often think there is something wrong with the first drive, then replace it with another drive, and when that too doesn't work they realise something is wrong with their system and call us, which is when we then can tell what has happened and we still have one good backup drive to restore from, but three days worth of lost backups is better than having none at all.

final
Enthusiast
Posts: 32
Liked: 11 times
Joined: Aug 14, 2016 7:19 pm
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by final » Mar 13, 2017 7:26 am

@kbossak: can you tell us (or play a guessgame with us) how the ransomware got access to the shares with the backups on them? Did it comprimise a server that had access permissions?

Your post made me overthink our backup strategy. We already do backups to tape on a weekly basis, but it would still be quite bad if our backups to disk were gone on a friday afternoon.

In our case, we use Synology NAS which are connected via iSCSI to the servers. But if the ransomware were to run under a domain admin account and tries all the default shares (c$, d$. e$ ...) it'd eventually find them as well.

smartsys
Enthusiast
Posts: 30
Liked: 4 times
Joined: Sep 14, 2010 8:27 am
Full Name: Jeroen Leeflang
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by smartsys » Mar 13, 2017 9:53 am

Hmmm. This complicates things a LOT.

There needs to be some sort of isolation between the backup "network" and the production network.
Would it make sense to place the entire backup environment in some sort of DMZ and control all traffic through a firewall?
Most backups are made using directsan, so it is only management traffic between the Veeam server, production servers requiring vss, hyper-v hosts, SCCM VMM or ESXi hosts, vCenter server.
We know what ports are used so we can easily make rules to only allow this traffic between the backup and production network.

When should also place Hyper-V hosts, ESXi hosts, VMM of vCenter in a separate management network (VLAN) to prevent Ransomware to easily discover these servers and monitor them.

When backup servers and storage cannot be reached from computers in the production network and from the management network only the protocols required can access the backup network and the other way around should this prevent the Ransomware from damaging any backup files?

I don't know how "smart" this Ransomware is?

netmask
Novice
Posts: 5
Liked: never
Joined: Aug 24, 2009 2:06 pm
Full Name: Rob Koetsier
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by netmask » Mar 13, 2017 10:30 am

You should Always password protect the backup share with an account that only you and veeam B&R knows. I even do it this way at home.

zfs
Novice
Posts: 6
Liked: never
Joined: Nov 13, 2014 9:03 pm
Full Name: Data Integrity
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by zfs » Mar 13, 2017 10:58 am

Hello, I would suggest using ZFS or similar as underlying storage for your Veeam backups and have snapshotting/replication enabled. This saved me about a month back. I would even recommend that you have this kind of storage capabilities for your VM storage, because in that case you can probably do snapshotting more frequent than with Veeam since the snapshot is underneath the hypervisor layer and is done without any snapshotting overhead in the hypervisor.

smartsys
Enthusiast
Posts: 30
Liked: 4 times
Joined: Sep 14, 2010 8:27 am
Full Name: Jeroen Leeflang
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by smartsys » Mar 13, 2017 10:59 am

netmask wrote:You should Always password protect the backup share with an account that only you and veeam B&R knows. I even do it this way at home.
This is possible if using a NAS device as the backup share.

When using a Windows server as a backup repository the local admin, all domain admins, system have potential access to the data.
If this Ransomware is capable of hopping from an infected client computer to other computers, it can also attack data that is permission protected by a single, maybe even local user account, simply by going around or by forcing ownership.

Maybe we should use a NAS to store backup data, for example a Netgear ReadyNAS, with snapshots enabled. Only thing is, this kind of storage often is not fast enough for random IO loads such as Veeam generates with synthetic backups.
If Ransomware somehow deletes the data, one can simply revert to an earlier snapshot and use this for restores after first making sure there is no more sign of ransomware activity.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 26 guests