Yes, Ransomware can delete your Veeam backups.

[CloudMSP]

All Veeam credentials are easily obtained by PowerShell via a script that support can run for you, in fact I still have a copy of the script. So there goes that "protection".

Guys I am a MSP with 40+ clients running Veeam to a local dedicated BDR server with local storage, and then cloudconnect to our datacenter. Can someone break it down, what would be the most cost effective way to ensure these backups are as safe as possible for all my clients?

[itrabbit]

Hi all,

With the recent post of virus and malware and even hackers deleting veeam backups. The one thing that bothers me, is an administrator can logon to the Veeam console and simply go to the repository and click delete and it uses the Veeam stored credentials.

Is there anyway to stop Veeam console from allow a user to directly delete backups. I am find if veeam is doing its retention policy and all magical stuff. But how do I stop simple users?

I would rather a dedicated password that must be entered prior to deleting any backups manually.

[veremin]

Hi, Matt,

Have you thought about adding tapes or rotated drives to your backup strategy?

Thanks,

[cbc-tgschultz]

Hi all,

With the recent post of virus and malware and even hackers deleting veeam backups. The one thing that bothers me, is an administrator can logon to the Veeam console and simply go to the repository and click delete and it uses the Veeam stored credentials.

Is there anyway to stop Veeam console from allow a user to directly delete backups. I am find if veeam is doing its retention policy and all magical stuff. But how do I stop simple users?

I would rather a dedicated password that must be entered prior to deleting any backups manually.

I'm afraid that given:

All Veeam credentials are easily obtained by PowerShell via a script that support can run for you, in fact I still have a copy of the script. So there goes that "protection".

This feature wouldn't actually offer that much protection. Veeam stores the credentials, so one must assume that if someone has access to the Veeam server, then they have access to those credentials and are a few simple steps from being able to delete your backups.

The solutions that come to mind:

1) Offline backups, such as the suggested tape backups. I personally don't like this as tape is quite inconvenient and rarely do people take the time to test them like they should.
2) Rely on features of the storage system. For instance, Nimble storage arrays can take block-delta snapshots as frequently as you like, with pretty much any retention period you like. If your files are deleted, just go back to the last known good snapshot. The problem here is that any attacker determined enough to get your Veeam server can probably take the time to get at your storage system too and just delete the snapshots.
3) Not-really-offline backups. I described this in an earlier post in this thread. Basically, set up another storage system that is completely inaccessible over the network, only being accessible through a physical monitor+keyboard in the server room, that can form one-way connections to the primary storage server for the purposes of copying backups to itself from there.

[aporter]

1.5) Offline backups using rotating external hard drives.

[Jacv]

Can existing vbk/vib files be set as read only?

My thinking is I can add another layer of security to my offsite DR (Linux SMB share over VPN) by changing RW access from the share user to RO by giving write permissions to root only after every backup.

I know I'd have to manually control retention policy and I still have airgapped backups but thought the extra defence layer would be worth it.

[DGrinev]

Hi,

In order to provide the best data protection plan you should follow the 3-2-1 Rule.
Also, you can use Veeam Cloud Connect to store your data in the cloud on a Service Provider site.
Please review this thread with best approaches of backup file protection against deletion. Thanks!

[chrsm]

I am designing a new veeam solution for a customer. We design the solution to do a daily backup copy job to a repository on a remote site.
What is best practices for securing the data on the remote site to protect the data from being deleted from a hacker attack (ransomware etc.)
I can see that 9.3 U3 have a new feature to cloud connect customers, that keep the data on the cloud provider even if the data have been deleted. Do we have similar possibilities even if we are not cloud connect customer.

[veremin]

Do we have similar possibilities even if we are not cloud connect customer.

Nope, and you wouldn't benefit from such feature, even if we had it, as everything would still be controlled with the same access identity. Which is not the case with Cloud Connect scenario, where a customer is unable to reach a location where deleted data is stored.

So, without CC in equation, the most reliable solutions turned to be tapes and removable drives (think real "offline", "air-gapped" backups).

Thanks.

[dellock6]

One possibility however is VCC-E, that is Cloud Connect for Enterprise. It's technically the same solution as "regular" Cloud Connect, but it has specific requirements in terms of existing Enterprise License agreements to be eligible.
Otherwise, the bets option is to either deploy VCC and become yourself a service provider for your customer, or partner (as a reseller/broker...) with an existing Veeam service provider and resell their Cloud Connect.

[yowmemperor]

So this discussion has gone more than one page since I last read it. I apologize ahead of time for not taking time to read the other 6 pages.

So a script was mentioned to rename the backup files. Sounds like a good idea, however, I assume this effects Veeam's ability to run Synthetic full's, and backup maps? To run those, the script would have to be re-run to revert to the original file name just as we would with a restore? The Syth fulls and maps take a significant amount of time for us, are there other ideas aside from offline copies?

[lxzndr]

What connections are required for a "disconnected" server to obtain backups, or backup copies if that system is on the other side of a physical firewall device?

I saw mention of something like that where connections "To" that device are denied, and only allow connections "From" that device are allowed.
I would actually lean towards running backup jobs from it, instead of copy - that way it becomes an additional independent backup and avoids any issues if the original repository is unavailable.

Would that server only need access to vCenter? (doing Network backup) or I could include a Proxy for HotAdd?

[kewnev]

Hi,

In order to provide the best data protection plan you should follow the 3-2-1 Rule.
Also, you can use Veeam Cloud Connect to store your data in the cloud on a Service Provider site.
Please review this thread with best approaches of backup file protection against deletion. Thanks!

Hello, is using Veeam Cloud Connect (VCC) as secure as using rotated drives? Let's say a hacker/malware gets in to my network and wiped out all my data and on-site backups. Could they also extract VCC credentials from my Veeam configuration, then connect to my VCC provider and wipe out my backups there?
(I have never used VCC, apologies if I sound ignorant..!)

[Gostev]

Not if your service provider has Insider Protection feature enabled.

[Phate1989]

Unitrends is using this post in their marketing.

[miconib]

What we have done and i'm not sure if anyone wants to add in there own snip about it is, the following

Since we mostly have small to mid size customers

For lager customers
Separate VMWARE server
Virtual Machine Running Veeam "Not on the domain, Completely difrent set of usernames and passwords for the admin account and there is only one admin account"
It is however on the same network, but it's completely locked unless you know the admin password "Which is different"
We make a veeam local backup
If the Server gets hit or has a hardware problem we spin it up on the secondary vmware server, we also use cloud connect as well but this is more for quick restore times "The local backup is"

For Smaller customers
On the same VMWARE server
Virtual Machine Running Veeam "Not on the domain, Completely difrent set of usernames and passwords for the admin account and there is only one admin account"
It is however on the same network, but it's completely locked unless you know the admin password "Which is different"
We make a veeam local backup
If the Server gets hit we spin it up on the same vmware server, we also use cloud connect as well but this is more for quick restore times "The local backup is"

what do you guys think ?

[michaelyou]

I think Veeam need to provide a proactive protection for ransomware issue.
This is because Veeam is image-based backup solution on Windows OS. (Just like Backup Exec or Ghost)
Therefore, vbk and vib files become targets of viruses or malware.
Currently, we use a powershell script to change the file extension to a custom string on each backup job. (prescript and postscript)

However
Commvault : Provide many methods to protect backup data from ransomware.
Unitrends and Rubrik : Hardened linux-based appliance.

https://documentation.commvault.com/com ... p=7722.htm
https://www.unitrends.com/solutions/ran ... protection

How about Veeam ?

[mikeely]

Our backups are being written (via NFS) to a ZFS share which takes regular snapshosts. Pretty sure the ransomware wouldn't be able to access those as the Linux vbr/headend can't see them, although if we somehow missed the attack until after the snaps expired...

[Gostev]

Right, Veeam supported Linux-based repositories since v1. So if you believe using hardened Linux-based backup repository actually helps against real-world attacks - then by all means, you should use them. I personally recommend air gapping backup instead, seeing how almost all successful attacks we've seen in support were carried out from inside by the hacker having sniffed/keylogged credentials to all critical IT systems... the only role actual ransomware had in all these cases was letting the hacker into the environment.

[michaelyou]

Hi mikeely ,

Thank you so much for your sharing about your scenario for the backup repository.

Hi Gostev ,

Thank you so much for your reply. I know the "air gap" is the ideal solution for data security and ransomware attack.
However if we implement air gap solution in the real world that means the RTO will be reduce due to isolated from the production environment. that is why I hope Veeam can do more proactive protection about this.
This is not only the "leader supplier" of Veeam in the Gartner report! However, like other competitors like Acronis, there is also "Active Protection". Why is Veeam not ? I hope Veeam can solve this issue from the root.

[Gostev]

Veeam does not have it because it is a marketing gimmick that does nothing besides giving you a false sense of protection. Such features are an actual disservice to the users, because they make them think they are well protected and can skip on implementing the proper solution (air gapped backups). It's not until after the actual attack when these users learn that solutions of this kind take seconds to be uninstalled (or simply disabled) by the hacker, thus adding zero extra protection to your backups.

I don't know if you followed Veeam forum digests in the past couple of years, because I've been sharing there many of the actual attack stories that we saw in our support. All of them clearly showed, in particular, how this type of "backup protection solutions" have zero value against real-world cyberattacks.

[F182]

I am very sorry for those that have been attacked by these ransom/viruses.

Though, this thread is quite surprising to me. We have been implementing 3-2-1 backup practices for almost 30 years and the concept was well established in the industry before then. See summary of 3-2-1: https://www.veeam.com/blog/how-to-follo ... ation.html.

Offsites are critical and not a want-have they are a must-have. There are a lot of options available. You can use portal media. Cloud repository. Etc etc.

For example, we are much beyond 3-2-1. We have the main backup repository. We have a copy job to portable media repository. We have a copy job to cloud repository. Our cloud repository has a backup of it's entire box to Azure backup (these backups go a long long way back).

It may seem like overkill, but if you want to sleep at night and have the confidence that the OP put in his first post, these are the steps that are needed.

Good luck everyone

[mvalpreda]

While not a replacement for air-gapped off-site backups.....if nothing else get a Synology (or other NAS) that does snapshots. Send backups to Synology, schedule a snapshot after your backups, and make sure you are not presenting snapshots to the OS. Make sure the login for your NAS is a different password than the admin password and not stored in the browser cache or password manager. We typically do iSCSI to take advantage of 9.5 + 2016 + ReFS.

If your backups get hosed, apply a snapshot. It's like nothing happened. Tested a few times and works great. Short of someone getting into your NAS and deleting the snapshots, you're in pretty good shape.

[CloudMSP]

Yeah but who actually recommends writing their backups to Synology in the first place?

[csydas]

However if we implement air gap solution in the real world that means the RTO will be reduce due to isolated from the production environment. that is why I hope Veeam can do more proactive protection about this.
This is not only the "leader supplier" of Veeam in the Gartner report! However, like other competitors like Acronis, there is also "Active Protection".

Out of curiosity, why does it have to reduce RTO? The "oh shit" backups should be beyond tertiary in my mind.

Backups are tough, and doing it "right" is basically throwing as much redundancy at the situation as you possibly can. In our setup, we just do absolutely everything we can to move the files into redundant and safe locations with the goal of eventually vaulting them, either with rotated drives or tapes. Yeah, it costs money, but at the same time, what is the cost of losing everything?

I don't buy the Active Protection stuff as it's basically just reliant on account permissions, and the entire attack vector for malware is that a malicious actor has gotten privileged credentials. You can bet against it as much as you want, but ultimately, it's not 0-days or clever privilege elevation attacks you're having to worry about, it's your coworker Bob getting an email that says "Hey Bob! look at this!".

We've had more infections because of otherwise intelligent and very talented admins falling for a spoofed email after long day than any actual security flaws. I'm not saying "don't patch it's pointless", but instead just understand the threat you're dealing with and also what frustrates these attacks.

We got to the size that the math for an all-down scenario took us way too long and too much money to get back to operational, and suddenly the cost of an LTO7 library wasn't so bad.

That's how we see it; we backup copy to rotated drives with a short-term "oh shit" retention, same with our tapes.in the event that we do get taken over, sure, it's going to be a long couple of days, but for the most part we have an RTO of 24 hour for our primary servers and upwards of 48 hours for our secondary. We've done this with one client before in an actual ransomware situation, and by hour 5, they were up and running with the essentials, by hour 30 everything was back.

[mvalpreda]

Yeah but who actually recommends writing their backups to Synology in the first place?

Not everyone has unlimited budgets for backup repositories. A DS918+ with 4x 8TB drives and Veeam B&R for a customer with 2-4TB of VMs....it works well when properly configured. We also recommend at a minimum doing a USB drive as an offsite. Did you have another recommendation?

[michaelyou]

Thank you all for your good idea and reply.

Honestly speaking.
After compared various backup solution (PoC), we decided to deploy Veeam to our environment.
However this issue is big concern for us. That is why I really care about the feedback from Veeam.

Yes , I totally agreed the 3-2-1 rules is the best practice for backup. no doubt.
But I feel Veeam just keep emphasize how importantce of 3-2-1 rules when Veeam discussing with this kind of topic.

I really hope Veeam have better solution or suggestion besides 3-2-1 rules.

Currently I planned to created Linux-based repository with ZFS file system.
I have a HP DL180 G6 server with 8TB hard disk x 8 (RAID 6)
Veeam server will access repository via NFS protocol.
Is it possible to restrict an account (linux local account) to Read / Write the backup reposiotry only ?
Any suggestion will be apprecaited.
Thank you very much.

[doum]

+1

it's a big problem and it's not always possible to use external backup (slow internet connection for example)

Does Veeam provide a whitepaper with all the best practices to secure its veeam infrastructure against this type of attack?

[afokkema]

You could take a look at the best practices guide: https://bp.veeam.expert/proof-of-concep ... -hardening or take a look at this whitepaper: https://www.veeam.com/wp-backup-replica ... ening.html

[SimonS]

Currently I planned to created Linux-based repository with ZFS file system.
I have a HP DL180 G6 server with 8TB hard disk x 8 (RAID 6)
Veeam server will access repository via NFS protocol.
Is it possible to restrict an account (linux local account) to Read / Write the backup reposiotry only ?
Any suggestion will be apprecaited.

Instead using NFS/CIFS you can consider using Linux repository:

Add Repository -> Type -> Linux Server

In this case Data mover is running on linux repository, and backup files are not visible to Windows world
Examples:
http://blog.dewin.me/2013/05/veeam-and- ... itory.html
https://www.virtualtothecore.com/en/per ... ositories/
https://www.virtualtothecore.com/en/vee ... -centos-7/