Yes, Ransomware can delete your Veeam backups.

Availability for the Always-On Enterprise

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Mike Resseler » Wed Mar 08, 2017 8:02 am

Morten,

Unfortunately yes... You make the risk very small, but the risk is there. Ransomware really is getting very good. Maybe you will be safe for a while, but they adapt their stuff pretty quickly. I've read in this thread also somewhere to use Linux as the typical Windows Ransomware cannot touch that but there were already attempts (OK, bad ones but still) in creating those for the Linux platform so air-gap is really the only solution at this moment which can be considered safe.
Mike Resseler
Veeam Software
 
Posts: 3381
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Gostev » Wed Mar 08, 2017 2:10 pm

There are plenty of Linux ransomware these days... KillDisk and FairWare come to my mind from the recent ones.
Gostev
Veeam Software
 
Posts: 21603
Liked: 2405 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Mike Resseler » Wed Mar 08, 2017 4:35 pm

True, luckily some of them are not that good and can be decrypted (albeit difficult) but it is just a matter of time before they will get worse :-(
Mike Resseler
Veeam Software
 
Posts: 3381
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby larry » Wed Mar 08, 2017 6:35 pm 2 people like this post

I have a powershell script that will email me if Veeam backup files go missing or renamed. I run it every hour as a task. You will need to have a c:\temp directory.
if you run as a different user than the task be sure to delete temp files. first run it will create baseline after that it just runs. A file change of 50 percent in a repository trips the email. not pretty code as I never planned on sharing. Normally I use it to monitor all user directories for big changes in case of rasomware or employee quitting. Hope it helps someone.

Code: Select all
Function SendEmail
{

# Email configuration

$emailHost = "exchange"
$emailUser = ""
$emailPass = ""
$emailFrom = "FileCountAlert@chelseagroton.com"
$emailTo = "lxxxxx@xxxxroton.com"
# Send report as attachment - $true or $false
$emailAttach = $true
# Email Subject
$emailSubject = "Large File Change"

If ($sendEmail) {
        $smtp = New-Object System.Net.Mail.SmtpClient $emailHost
$smtp.port = 25
#port 26 because mcafee blocks 25 on win 7 Exception calling "Send" with "1" argument(s): "Failure sending mail
        $smtp.Credentials = New-Object System.Net.NetworkCredential($emailUser, $emailPass);
$msg = New-Object System.Net.Mail.MailMessage($emailFrom, $emailTo)
$msg.Subject = $emailSubject
If ($emailAttach) {
#$body = "Report Attached"
                      $body = $htmlOutput
$msg.Body = $body
                        $msg.isBodyhtml = $true
$tempfile = "$env:TEMP\$emailSubject.htm"
$htmlOutput | Out-File $tempfile
$attachment = new-object System.Net.Mail.Attachment $tempfile
      $msg.Attachments.Add($attachment)

} Else {
$body = $htmlOutput
$msg.Body = $body
$msg.isBodyhtml = $true
}       
        $smtp.send($msg)
If ($emailAttach) {
$attachment.dispose()
Remove-Item $tempfile
  }
   
       }
} #email Function

$sendEmail = $false
$pathArray = @("E:\backup","E:\Backups","F:\Backups","g:\Backups","h:\Backups","t:\Backups")
$Dirlist = @()

#get dir names
for ($i=0; $i -lt $pathArray.length; $i++) {
$originalPath =$pathArray[$i]
$tempDirlist = Get-ChildItem $originalPath\* | ?{ $_.PSIsContainer } | Select-Object FullName
$Dirlist += $tempDirlist
}

$OutputArray = @()


$File="C:\temp\content.txt"
if (Test-Path $File)
{$DirList2 = import-clixml $File  }
else
{
#should only happen on install or first run
$DirList2 = $Dirlist
}

foreach ($i in $Dirlist)
{
$originalPath = $I.FullName
#$filecount = (Get-ChildItem $originalPath\* -Include *.gif, *.jpg, *.xls*, *.doc*,*.png*, *.pdf*, *.wav*, .ppt* -recurse).count
#top one was to see if docs got encrypted or deleted, below switched to veeam backups
$filecount = (Get-ChildItem $originalPath\* -Include *.vbk, *.vrb, *.vbm*, *.vib -recurse).count

$I |Add-Member -MemberType NoteProperty -Name 'files' -Value ($filecount)
$I |Add-Member -MemberType NoteProperty -Name 'Note' -Value ("")
}


$File="C:\temp\content.txt"
if (Test-Path $File)
{$DirList2 = import-clixml $File  }
else
{
#should only happen on install or first run
$DirList2 = $Dirlist
}



$compare = compare-object $DirList2 $Dirlist  -Property Fullname,files
$compare = $compare | sort FullName


$diffCount = $compare.Count
foreach ($i in $compare)
  {
     $i |Add-Member -MemberType NoteProperty -Name 'Note' -Value ("")
   }
$compare | foreach  {
      if ($_.sideindicator -eq '<=')
        {$_.sideindicator = "Was"}

      if ($_.sideindicator -eq '=>')
        {$_.sideindicator = "Is"}
     }

$body = $compare| ConvertTo-HTML -fragment
For ($i=0; $i -lt $diffCount; $i++) {
   
   if ($compare[$i].FullName -eq $compare[$i + 1 ].FullName)
    {
       write-host $compare[$i].FullName
       $diff = $compare[$i].files - $compare[$i + 1].files
         $t = $i
         $t2 = $i +1
         # var t is used in math not sure which order
        if ($compare[$i].SideIndicator -eq "Was" )
           {$diff = $diff * -1
           $t = $i +1
           $t2 = $I
            }

         write-host $diff
    #lt 5 files who cares
        if ($diff -lt 5 )
           {$percentDown = 100 - (($compare[$t].files + 1)/($compare[$t2].files +1) * 100)
     # add one to not worry div zero
             Write-host "File Count went down " $percentDown

            }
        if ($percentDown -gt 50 )
            {
              $sendEmail = $true
              Write-host "File Count went down more than 50 percent"
              $compare[$i].Note = "File Count went down more than 50 percent"
             }

#write-host $c[$i + 1].FullName

     }
 }

$body = $compare| ConvertTo-HTML -fragment

# HTML Stuff
$headerObj = @"
<html>
        <head>
        <center>
           <h1>Warning Veeam File Changes  </h1>
           <h3>Report generated at $(Get-Date -format g) on $((gc env:computername).ToLower()) </h3>   
       </center>
        </head>
"@
 
$bodyTop = @"
        <body>
               
"@
 


$footerObj = @"
</body>
</html>
"@


#$htmlOutput = $headerObj + $bodyTop + $body + $filelist + $footerObj
$htmlOutput = $headerObj + $bodyTop + $body + $footerObj

$Dirlist |  Select-Object FullName, files | Export-Clixml -Path $File
$htmlOutput > c:\temp\test.html
SendEmail
larry
Expert
 
Posts: 383
Liked: 90 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby csinetops » Wed Mar 08, 2017 9:21 pm

Is there a reason why the backup servers need to talk to anything other than the other backup servers,DC's and VCenter? I was thinking of using IP security policies to lock my Veeam servers down to only talk to each other,the DC's and Vcenter.

So far all I can thing we'd lose out on is failover to network mode ( we use hot add).

Does Veeam need to talk to the production servers in a restore scenario? IE, restoring a file using FLR or Exchange item?

Am I missing anything?
csinetops
Expert
 
Posts: 111
Liked: 15 times
Joined: Fri Jun 06, 2014 2:45 pm
Full Name: csinetops

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby Gostev » Wed Mar 08, 2017 11:20 pm

Yes, it needs to talk to production servers to restore files and application items there. Although for files specifically, we don't need the direct network connection because we can also inject them through the host connection (albeit much slower).
Gostev
Veeam Software
 
Posts: 21603
Liked: 2405 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby frankj » Sun Mar 12, 2017 11:23 pm

you have this part in double is that normal ?

Code: Select all
$File="C:\temp\content.txt"
if (Test-Path $File) {
    $DirList2 = import-clixml $File 
} else {
    #should only happen on install or first run
    $DirList2 = $Dirlist
}

larry wrote:I have a powershell script that will email me if Veeam backup files go missing or renamed. I run it every hour as a task. You will need to have a c:\temp directory.
if you run as a different user than the task be sure to delete temp files. first run it will create baseline after that it just runs. A file change of 50 percent in a repository trips the email. not pretty code as I never planned on sharing. Normally I use it to monitor all user directories for big changes in case of rasomware or employee quitting. Hope it helps someone.

Code: Select all
Function SendEmail
{

# Email configuration

$emailHost = "exchange"
$emailUser = ""
$emailPass = ""
$emailFrom = "FileCountAlert@chelseagroton.com"
$emailTo = "lxxxxx@xxxxroton.com"
# Send report as attachment - $true or $false
$emailAttach = $true
# Email Subject
$emailSubject = "Large File Change"

If ($sendEmail) {
        $smtp = New-Object System.Net.Mail.SmtpClient $emailHost
$smtp.port = 25
#port 26 because mcafee blocks 25 on win 7 Exception calling "Send" with "1" argument(s): "Failure sending mail
        $smtp.Credentials = New-Object System.Net.NetworkCredential($emailUser, $emailPass);
$msg = New-Object System.Net.Mail.MailMessage($emailFrom, $emailTo)
$msg.Subject = $emailSubject
If ($emailAttach) {
#$body = "Report Attached"
                      $body = $htmlOutput
$msg.Body = $body
                        $msg.isBodyhtml = $true
$tempfile = "$env:TEMP\$emailSubject.htm"
$htmlOutput | Out-File $tempfile
$attachment = new-object System.Net.Mail.Attachment $tempfile
      $msg.Attachments.Add($attachment)

} Else {
$body = $htmlOutput
$msg.Body = $body
$msg.isBodyhtml = $true
}       
        $smtp.send($msg)
If ($emailAttach) {
$attachment.dispose()
Remove-Item $tempfile
  }
   
       }
} #email Function

$sendEmail = $false
$pathArray = @("E:\backup","E:\Backups","F:\Backups","g:\Backups","h:\Backups","t:\Backups")
$Dirlist = @()

#get dir names
for ($i=0; $i -lt $pathArray.length; $i++) {
$originalPath =$pathArray[$i]
$tempDirlist = Get-ChildItem $originalPath\* | ?{ $_.PSIsContainer } | Select-Object FullName
$Dirlist += $tempDirlist
}

$OutputArray = @()


$File="C:\temp\content.txt"
if (Test-Path $File)
{$DirList2 = import-clixml $File  }
else
{
#should only happen on install or first run
$DirList2 = $Dirlist
}

foreach ($i in $Dirlist)
{
$originalPath = $I.FullName
#$filecount = (Get-ChildItem $originalPath\* -Include *.gif, *.jpg, *.xls*, *.doc*,*.png*, *.pdf*, *.wav*, .ppt* -recurse).count
#top one was to see if docs got encrypted or deleted, below switched to veeam backups
$filecount = (Get-ChildItem $originalPath\* -Include *.vbk, *.vrb, *.vbm*, *.vib -recurse).count

$I |Add-Member -MemberType NoteProperty -Name 'files' -Value ($filecount)
$I |Add-Member -MemberType NoteProperty -Name 'Note' -Value ("")
}


$File="C:\temp\content.txt"
if (Test-Path $File)
{$DirList2 = import-clixml $File  }
else
{
#should only happen on install or first run
$DirList2 = $Dirlist
}



$compare = compare-object $DirList2 $Dirlist  -Property Fullname,files
$compare = $compare | sort FullName


$diffCount = $compare.Count
foreach ($i in $compare)
  {
     $i |Add-Member -MemberType NoteProperty -Name 'Note' -Value ("")
   }
$compare | foreach  {
      if ($_.sideindicator -eq '<=')
        {$_.sideindicator = "Was"}

      if ($_.sideindicator -eq '=>')
        {$_.sideindicator = "Is"}
     }

$body = $compare| ConvertTo-HTML -fragment
For ($i=0; $i -lt $diffCount; $i++) {
   
   if ($compare[$i].FullName -eq $compare[$i + 1 ].FullName)
    {
       write-host $compare[$i].FullName
       $diff = $compare[$i].files - $compare[$i + 1].files
         $t = $i
         $t2 = $i +1
         # var t is used in math not sure which order
        if ($compare[$i].SideIndicator -eq "Was" )
           {$diff = $diff * -1
           $t = $i +1
           $t2 = $I
            }

         write-host $diff
    #lt 5 files who cares
        if ($diff -lt 5 )
           {$percentDown = 100 - (($compare[$t].files + 1)/($compare[$t2].files +1) * 100)
     # add one to not worry div zero
             Write-host "File Count went down " $percentDown

            }
        if ($percentDown -gt 50 )
            {
              $sendEmail = $true
              Write-host "File Count went down more than 50 percent"
              $compare[$i].Note = "File Count went down more than 50 percent"
             }

#write-host $c[$i + 1].FullName

     }
 }

$body = $compare| ConvertTo-HTML -fragment

# HTML Stuff
$headerObj = @"
<html>
        <head>
        <center>
           <h1>Warning Veeam File Changes  </h1>
           <h3>Report generated at $(Get-Date -format g) on $((gc env:computername).ToLower()) </h3>   
       </center>
        </head>
"@
 
$bodyTop = @"
        <body>
               
"@
 


$footerObj = @"
</body>
</html>
"@


#$htmlOutput = $headerObj + $bodyTop + $body + $filelist + $footerObj
$htmlOutput = $headerObj + $bodyTop + $body + $footerObj

$Dirlist |  Select-Object FullName, files | Export-Clixml -Path $File
$htmlOutput > c:\temp\test.html
SendEmail
frankj
Service Provider
 
Posts: 20
Liked: 1 time
Joined: Fri May 27, 2016 4:53 pm
Full Name: FRANK Jacques

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby tacioandrade » Sun Mar 12, 2017 11:27 pm

Without a doubt, backups made with Veeam can be encrypted, so much so that I contacted the Samba developers to see if there was something like Veto Files, however that would work in reverse, a Allow Files, releasing the writing files only Of the released extensions (to record my Veeam backups and other solutions in this directory).

However, they said that this functionality was not done and that if someone created it and did a pull to the samba repository, it would certainly be accepted, but the developers themselves would not program it.

A pity as I see this, because without this feature, I believe the only real way to solve this problem is as a friend has done in FreeNAS, mounting different LUNs for different days on the same mount point, so that if one of them Be committed, the other is not.


Sincerely, Tácio Andrade.
tacioandrade
Enthusiast
 
Posts: 29
Liked: 3 times
Joined: Thu Nov 17, 2016 2:04 am
Full Name: Tácio Andrade

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby frankj » Sun Mar 12, 2017 11:30 pm

woudnt a user that is veeam credentials only having r/w and global admins just read be a way ?

this way an infected server that is affected coudnt touch those repos right ? unless it's the veeam user account ?
frankj
Service Provider
 
Posts: 20
Liked: 1 time
Joined: Fri May 27, 2016 4:53 pm
Full Name: FRANK Jacques

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby AlexLeadingEdge » Mon Mar 13, 2017 1:43 am

One of our clients were hit by the cryptolocker virus a few months back, it wiped out a days worth of work. The thing that saved them was that they rotate their USB drives every morning, there are five of them for each day of the working week, and they're stored in a fire-proof safe. The cryptolocker virus could only encrypt that which was physically attached, so the other air-gapped USB backup drives were all fine.

One of the reasons we have at a bare minimum three USB drives is because users will often think there is something wrong with the first drive, then replace it with another drive, and when that too doesn't work they realise something is wrong with their system and call us, which is when we then can tell what has happened and we still have one good backup drive to restore from, but three days worth of lost backups is better than having none at all.
AlexLeadingEdge
Expert
 
Posts: 161
Liked: 19 times
Joined: Mon Dec 14, 2015 9:42 pm

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby final » Mon Mar 13, 2017 7:26 am

@kbossak: can you tell us (or play a guessgame with us) how the ransomware got access to the shares with the backups on them? Did it comprimise a server that had access permissions?

Your post made me overthink our backup strategy. We already do backups to tape on a weekly basis, but it would still be quite bad if our backups to disk were gone on a friday afternoon.

In our case, we use Synology NAS which are connected via iSCSI to the servers. But if the ransomware were to run under a domain admin account and tries all the default shares (c$, d$. e$ ...) it'd eventually find them as well.
final
Enthusiast
 
Posts: 25
Liked: 8 times
Joined: Sun Aug 14, 2016 7:19 pm

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby smartsys » Mon Mar 13, 2017 9:53 am

Hmmm. This complicates things a LOT.

There needs to be some sort of isolation between the backup "network" and the production network.
Would it make sense to place the entire backup environment in some sort of DMZ and control all traffic through a firewall?
Most backups are made using directsan, so it is only management traffic between the Veeam server, production servers requiring vss, hyper-v hosts, SCCM VMM or ESXi hosts, vCenter server.
We know what ports are used so we can easily make rules to only allow this traffic between the backup and production network.

When should also place Hyper-V hosts, ESXi hosts, VMM of vCenter in a separate management network (VLAN) to prevent Ransomware to easily discover these servers and monitor them.

When backup servers and storage cannot be reached from computers in the production network and from the management network only the protocols required can access the backup network and the other way around should this prevent the Ransomware from damaging any backup files?

I don't know how "smart" this Ransomware is?
smartsys
Influencer
 
Posts: 21
Liked: 4 times
Joined: Tue Sep 14, 2010 8:27 am
Full Name: Jeroen Leeflang

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby netmask » Mon Mar 13, 2017 10:30 am

You should Always password protect the backup share with an account that only you and veeam B&R knows. I even do it this way at home.
netmask
Novice
 
Posts: 5
Liked: never
Joined: Mon Aug 24, 2009 2:06 pm
Full Name: Rob Koetsier

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby zfs » Mon Mar 13, 2017 10:58 am

Hello, I would suggest using ZFS or similar as underlying storage for your Veeam backups and have snapshotting/replication enabled. This saved me about a month back. I would even recommend that you have this kind of storage capabilities for your VM storage, because in that case you can probably do snapshotting more frequent than with Veeam since the snapshot is underneath the hypervisor layer and is done without any snapshotting overhead in the hypervisor.
zfs
Novice
 
Posts: 6
Liked: never
Joined: Thu Nov 13, 2014 9:03 pm
Full Name: Data Integrity

Re: Yes, Ransomware can delete your Veeam backups.

Veeam Logoby smartsys » Mon Mar 13, 2017 10:59 am

netmask wrote:You should Always password protect the backup share with an account that only you and veeam B&R knows. I even do it this way at home.

This is possible if using a NAS device as the backup share.

When using a Windows server as a backup repository the local admin, all domain admins, system have potential access to the data.
If this Ransomware is capable of hopping from an infected client computer to other computers, it can also attack data that is permission protected by a single, maybe even local user account, simply by going around or by forcing ownership.

Maybe we should use a NAS to store backup data, for example a Netgear ReadyNAS, with snapshots enabled. Only thing is, this kind of storage often is not fast enough for random IO loads such as Veeam generates with synthetic backups.
If Ransomware somehow deletes the data, one can simply revert to an earlier snapshot and use this for restores after first making sure there is no more sign of ransomware activity.
smartsys
Influencer
 
Posts: 21
Liked: 4 times
Joined: Tue Sep 14, 2010 8:27 am
Full Name: Jeroen Leeflang

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot], Yahoo [Bot] and 5 guests