Availability for the Always-On Enterprise
Post Reply
miconib
Lurker
Posts: 1
Liked: never
Joined: Jun 22, 2018 9:49 am
Full Name: Brandon Miconi
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by miconib » Jun 22, 2018 10:07 am

What we have done and i'm not sure if anyone wants to add in there own snip about it is, the following

Since we mostly have small to mid size customers

For lager customers
Separate VMWARE server
Virtual Machine Running Veeam "Not on the domain, Completely difrent set of usernames and passwords for the admin account and there is only one admin account"
It is however on the same network, but it's completely locked unless you know the admin password "Which is different"
We make a veeam local backup
If the Server gets hit or has a hardware problem we spin it up on the secondary vmware server, we also use cloud connect as well but this is more for quick restore times "The local backup is"

For Smaller customers
On the same VMWARE server
Virtual Machine Running Veeam "Not on the domain, Completely difrent set of usernames and passwords for the admin account and there is only one admin account"
It is however on the same network, but it's completely locked unless you know the admin password "Which is different"
We make a veeam local backup
If the Server gets hit we spin it up on the same vmware server, we also use cloud connect as well but this is more for quick restore times "The local backup is"

what do you guys think ?

michaelyou
Novice
Posts: 4
Liked: 1 time
Joined: Jul 06, 2018 3:19 pm
Full Name: michael
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by michaelyou » Jul 06, 2018 6:00 pm

I think Veeam need to provide a proactive protection for ransomware issue.
This is because Veeam is image-based backup solution on Windows OS. (Just like Backup Exec or Ghost)
Therefore, vbk and vib files become targets of viruses or malware.
Currently, we use a powershell script to change the file extension to a custom string on each backup job. (prescript and postscript)

However
Commvault : Provide many methods to protect backup data from ransomware.
Unitrends and Rubrik : Hardened linux-based appliance.

https://documentation.commvault.com/com ... p=7722.htm
https://www.unitrends.com/solutions/ran ... protection

How about Veeam ?

mikeely
Enthusiast
Posts: 67
Liked: 10 times
Joined: Nov 07, 2016 7:39 pm
Full Name: Mike Ely
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by mikeely » Jul 06, 2018 6:36 pm

Our backups are being written (via NFS) to a ZFS share which takes regular snapshosts. Pretty sure the ransomware wouldn't be able to access those as the Linux vbr/headend can't see them, although if we somehow missed the attack until after the snaps expired...
Unless otherwise specified, I am asking about something pertaining to Linux. We use Windows as infrequently as possible, and enthusiastically seek ways to reduce that usage further.

Gostev
Veeam Software
Posts: 22808
Liked: 2801 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Gostev » Jul 06, 2018 11:37 pm

Right, Veeam supported Linux-based repositories since v1. So if you believe using hardened Linux-based backup repository actually helps against real-world attacks - then by all means, you should use them. I personally recommend air gapping backup instead, seeing how almost all successful attacks we've seen in support were carried out from inside by the hacker having sniffed/keylogged credentials to all critical IT systems... the only role actual ransomware had in all these cases was letting the hacker into the environment.

michaelyou
Novice
Posts: 4
Liked: 1 time
Joined: Jul 06, 2018 3:19 pm
Full Name: michael
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by michaelyou » Jul 07, 2018 4:36 am

Hi mikeely ,

Thank you so much for your sharing about your scenario for the backup repository.

Hi Gostev ,

Thank you so much for your reply. I know the "air gap" is the ideal solution for data security and ransomware attack.
However if we implement air gap solution in the real world that means the RTO will be reduce due to isolated from the production environment. that is why I hope Veeam can do more proactive protection about this.
This is not only the "leader supplier" of Veeam in the Gartner report! However, like other competitors like Acronis, there is also "Active Protection". Why is Veeam not ? I hope Veeam can solve this issue from the root.

Gostev
Veeam Software
Posts: 22808
Liked: 2801 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by Gostev » Jul 07, 2018 8:06 pm 1 person likes this post

Veeam does not have it because it is a marketing gimmick that does nothing besides giving you a false sense of protection. Such features are an actual disservice to the users, because they make them think they are well protected and can skip on implementing the proper solution (air gapped backups). It's not until after the actual attack when these users learn that solutions of this kind take seconds to be uninstalled (or simply disabled) by the hacker, thus adding zero extra protection to your backups.

I don't know if you followed Veeam forum digests in the past couple of years, because I've been sharing there many of the actual attack stories that we saw in our support. All of them clearly showed, in particular, how this type of "backup protection solutions" have zero value against real-world cyberattacks.

F182
Service Provider
Posts: 18
Liked: 2 times
Joined: Jun 03, 2018 3:13 pm
Full Name: Farzon David Almaneih
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by F182 » Jul 08, 2018 2:32 pm 1 person likes this post

I am very sorry for those that have been attacked by these ransom/viruses.

Though, this thread is quite surprising to me. We have been implementing 3-2-1 backup practices for almost 30 years and the concept was well established in the industry before then. See summary of 3-2-1: https://www.veeam.com/blog/how-to-follo ... ation.html.

Offsites are critical and not a want-have they are a must-have. There are a lot of options available. You can use portal media. Cloud repository. Etc etc.

For example, we are much beyond 3-2-1. We have the main backup repository. We have a copy job to portable media repository. We have a copy job to cloud repository. Our cloud repository has a backup of it's entire box to Azure backup (these backups go a long long way back).

It may seem like overkill, but if you want to sleep at night and have the confidence that the OP put in his first post, these are the steps that are needed.

Good luck everyone

mvalpreda
Enthusiast
Posts: 38
Liked: 1 time
Joined: May 06, 2015 10:57 pm
Full Name: Mark Valpreda
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by mvalpreda » Jul 10, 2018 1:58 am 1 person likes this post

While not a replacement for air-gapped off-site backups.....if nothing else get a Synology (or other NAS) that does snapshots. Send backups to Synology, schedule a snapshot after your backups, and make sure you are not presenting snapshots to the OS. Make sure the login for your NAS is a different password than the admin password and not stored in the browser cache or password manager. We typically do iSCSI to take advantage of 9.5 + 2016 + ReFS.

If your backups get hosed, apply a snapshot. It's like nothing happened. Tested a few times and works great. Short of someone getting into your NAS and deleting the snapshots, you're in pretty good shape.

CloudMSP
Service Provider
Posts: 30
Liked: 11 times
Joined: Jul 16, 2017 5:39 am
Full Name: Veeam MSP
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by CloudMSP » Jul 22, 2018 1:52 pm 1 person likes this post

Yeah but who actually recommends writing their backups to Synology in the first place?

csydas
Expert
Posts: 117
Liked: 27 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by csydas » Jul 22, 2018 5:34 pm 1 person likes this post

michaelyou wrote: However if we implement air gap solution in the real world that means the RTO will be reduce due to isolated from the production environment. that is why I hope Veeam can do more proactive protection about this.
This is not only the "leader supplier" of Veeam in the Gartner report! However, like other competitors like Acronis, there is also "Active Protection".
Out of curiosity, why does it have to reduce RTO? The "oh shit" backups should be beyond tertiary in my mind.

Backups are tough, and doing it "right" is basically throwing as much redundancy at the situation as you possibly can. In our setup, we just do absolutely everything we can to move the files into redundant and safe locations with the goal of eventually vaulting them, either with rotated drives or tapes. Yeah, it costs money, but at the same time, what is the cost of losing everything?

I don't buy the Active Protection stuff as it's basically just reliant on account permissions, and the entire attack vector for malware is that a malicious actor has gotten privileged credentials. You can bet against it as much as you want, but ultimately, it's not 0-days or clever privilege elevation attacks you're having to worry about, it's your coworker Bob getting an email that says "Hey Bob! look at this!".

We've had more infections because of otherwise intelligent and very talented admins falling for a spoofed email after long day than any actual security flaws. I'm not saying "don't patch it's pointless", but instead just understand the threat you're dealing with and also what frustrates these attacks.

We got to the size that the math for an all-down scenario took us way too long and too much money to get back to operational, and suddenly the cost of an LTO7 library wasn't so bad.

That's how we see it; we backup copy to rotated drives with a short-term "oh shit" retention, same with our tapes.in the event that we do get taken over, sure, it's going to be a long couple of days, but for the most part we have an RTO of 24 hour for our primary servers and upwards of 48 hours for our secondary. We've done this with one client before in an actual ransomware situation, and by hour 5, they were up and running with the essentials, by hour 30 everything was back.

mvalpreda
Enthusiast
Posts: 38
Liked: 1 time
Joined: May 06, 2015 10:57 pm
Full Name: Mark Valpreda
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by mvalpreda » Jul 23, 2018 1:41 am

CloudMSP wrote:Yeah but who actually recommends writing their backups to Synology in the first place?
Not everyone has unlimited budgets for backup repositories. A DS918+ with 4x 8TB drives and Veeam B&R for a customer with 2-4TB of VMs....it works well when properly configured. We also recommend at a minimum doing a USB drive as an offsite. Did you have another recommendation?

michaelyou
Novice
Posts: 4
Liked: 1 time
Joined: Jul 06, 2018 3:19 pm
Full Name: michael
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by michaelyou » Jul 23, 2018 6:49 am

Thank you all for your good idea and reply.

Honestly speaking.
After compared various backup solution (PoC), we decided to deploy Veeam to our environment.
However this issue is big concern for us. That is why I really care about the feedback from Veeam.

Yes , I totally agreed the 3-2-1 rules is the best practice for backup. no doubt.
But I feel Veeam just keep emphasize how importantce of 3-2-1 rules when Veeam discussing with this kind of topic.

I really hope Veeam have better solution or suggestion besides 3-2-1 rules.

Currently I planned to created Linux-based repository with ZFS file system.
I have a HP DL180 G6 server with 8TB hard disk x 8 (RAID 6)
Veeam server will access repository via NFS protocol.
Is it possible to restrict an account (linux local account) to Read / Write the backup reposiotry only ?
Any suggestion will be apprecaited.
Thank you very much.

doum
Novice
Posts: 3
Liked: never
Joined: Feb 15, 2018 10:45 pm
Full Name: Benoit Machiavello
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by doum » Jul 23, 2018 7:15 am

+1

it's a big problem and it's not always possible to use external backup (slow internet connection for example)

Does Veeam provide a whitepaper with all the best practices to secure its veeam infrastructure against this type of attack?

afokkema
Service Provider
Posts: 21
Liked: 1 time
Joined: Feb 13, 2009 2:00 pm
Full Name: Arne Fokkema
Location: Netherlands
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by afokkema » Jul 24, 2018 12:01 pm 1 person likes this post

You could take a look at the best practices guide: https://bp.veeam.expert/proof-of-concep ... -hardening or take a look at this whitepaper: https://www.veeam.com/wp-backup-replica ... ening.html

SimonS
Novice
Posts: 7
Liked: 3 times
Joined: Jan 26, 2018 11:19 am
Full Name: Simon Setina
Location: Slovenia
Contact:

Re: Yes, Ransomware can delete your Veeam backups.

Post by SimonS » Jul 24, 2018 12:35 pm 1 person likes this post

michaelyou wrote:
Currently I planned to created Linux-based repository with ZFS file system.
I have a HP DL180 G6 server with 8TB hard disk x 8 (RAID 6)
Veeam server will access repository via NFS protocol.
Is it possible to restrict an account (linux local account) to Read / Write the backup reposiotry only ?
Any suggestion will be apprecaited.
Instead using NFS/CIFS you can consider using Linux repository:

Add Repository -> Type -> Linux Server

In this case Data mover is running on linux repository, and backup files are not visible to Windows world
Examples:
http://blog.dewin.me/2013/05/veeam-and- ... itory.html
https://www.virtualtothecore.com/en/per ... ositories/
https://www.virtualtothecore.com/en/vee ... -centos-7/

Post Reply

Who is online

Users browsing this forum: No registered users and 21 guests