How are customers implemententing "offline" backups in their enviroments?

[nd39475]

I was posed a question/scenario by management to conceptualize an "offline backup" for the purposes of ransomware prevention. How are you all implementing offline backups? We have moved beyond tape drives and we do not want to re-implement such.

One proposed option was to periodically disable and re-enable network connectivity to a repository, but such does not sit well with me, since backups are scheduled daily, not manually upon need.

What are you dong to adhere to potential state mandated edu/gov requirements?

Thank you!

[AlexLeadingEdge]

- A backup server not on the domain.
- Use a backup server Administrator password that isn't the domain Administrator password. Make sure the password is long and strong.
- The firewall on the backup server set to block all but the VeeamB&R ports.
- File and printer sharing turned off.
- Multiple USB / tape backup drives that are rotated daily, and therefore air-gapped.

Basically the aim is to make the backup server only pull information, because the ability to push information to the backup server means a route for the viruses.

[richkovach]

On our install I noticed that there is a VBRCatalog folder shared out over smb. Can file sharing truly be disabled, or can the ports be blocked at the OS firewall, or is external access to that share needed for product functionality?

[nd39475]

The good news is that my Veeam Repositories are Non-Domain & Linux. That bad news is management may require quote "offline", but my target datastore(s) are not rotatable nor air-gap-able.
62TB on Dell VRTX Chassis is my primary Backup Copy Target. (61 days)
About 9TB for my secondary, offsite. (3-5 days)

I'm now wondering if older Monthlies and Weeklies on the larger array could constitute "offline".

[tomnewman]

I'm intrigued as to what "moved beyond tape drives" means to you.

We find they are reliable and are an easy way to achieve air-gap secondary backup that can easily be removed from site for disaster recovery.

[Gostev]

Technically speaking, as a storage media, tape is between 10 to 1000 times more reliable than disk. Where the difference comes from the disk grade: Consumer vs. Enterprise SATA vs. Enterprise SAS. To remind, storage media reliability is measured in URE (Unrecoverable Read Error). For example, consumer-grade hard drives are typically rated with an average of 1 URE per 10TB of data read.

Mind you, these numbers are under perfectly "normal" handling and operational conditions. And I'm not just talking of obvious hard drive killing stuff such as external vibration or drop. For example, there's so called "Battelle evaluation" for tape media vendors, which includes placing a tape into corrosive gas environment for 14 days. I mean, I don't want to even think what this will do to a hard drive

[agrob]

i would also recommend tape for complete offline backup. it is the only media i know of a the moment which you can take complete offline and store it elsewhere. for smaller environment you could probably use usb drives. but for more data and for better performance i would still recommend tape. even if you use cloud offload, if someone gets access to your backup server and have admin privileges in your backup software, cloud backups can be erased. that is not possible with offline media... so in fact it depends on the requirement you have... if cloud backups are enough or not, which risk you need to address

[ChrisGundry]

There are some cloud vendors that are offering 'insider protection' and things like 'offline recycle bins' for deleted backups. This might help with what you want.

[mbroaders]

Insider Protection helps with Cloud Connect but be sure to include a GFS point to stop someone from corrupting the chain. We usually size to include at least a Weekly point.

[zoltank]

Our primary offline back is tape. There's really nothing that works as well for offline and air gapped backups.

We also have a secondary offline back which I doubt you'll see anywhere else. It's a second Veeam server which does backups three times a week. It's connected through a small 5 port gigabit switch which sits on a simple programmable timer, and the switch is only turned on during the backup window.

Then we have a repository server that copies the backups from the second Veeam server onto local storage. It's directly connected through another 5 port switch directly to the Veeam server, and the switch is also on a timer, but it's only active when the Veeam server's switch is powered off. This way the repository server is never connected to the second Veeam server while the Veeam server is connected to the network.

In addition each of the servers sits behind an inexpensive firewall which only allow online connections and not online connections.

[chi-ltd]

3-2-1 model = tape i believe.

[MurkyBuffalo]

An exciting, meaning new, option is to use tape for Short term (3-6months) offline retention and Cloud object storage in a scale out repository for long term retention.

If you use a private cloud, you can use some of the offlining tricks mentioned in this thread (firewalls, timers, off-domain) with the added simplicity of not depending on Veeam binaries in the offline environment. You can use something any object storage container or appliance which makes scaling and maintenance simpler.

[TGacs]

We use an 8-drive RDX device as a backup copy destination. For a small company like ours, the cartridges are fairly expensive, though, and the backup times can be long (merges can take nearly a day). That gives us weekly offsite/air-gapped, but its not a great RPO.

I'm wondering about a virtual tape library onsite (we have a lot of hardware already) like Starwind. That could give us a better RPO for ransomware, etc. while the RDXs can continue to provide offsite for disaster (fire, etc.). Anyone have experience w/VTLs for this purpose?

[daniel.farrelly]

Also curious about AWS virtual tape libraries, if that is considered air-gapped. Particularly if you "eject media upon completion" and "export to archive." If anybody is willing, would love to hear some real-world stories about if/how this may have helped during a ransomware attack.

[xudaiqing]

We currently use repository vm in the cloud with daily snapshot for offsite/air-gapped backup. The snapshot is protected by 2FA and isn't accessible by backup servers.
For long term archive we just upload local backup files to glacier

[stmux]

I have mostly small clients using mostly physical servers and workstations. I have added external, line powered USB hard drives controlled by a 7 day, auto DST timer. The timer is set for a one hour window of time needed for differential backups which for my clients run between 15 and 45 minutes. That means the USB hard drives are offline for 23 hours per day. I am experimenting with using the timer to control a 5 port switch connected to a NAS to do the same. This solution is obviously for small installations, but it works well and is very inexpensive. My cost for parts and perpetual license is $200/computer which is very attractive to small businesses. I have tested the above with Veeam's Microsoft Agent and it works well, however, Veeam does not currently offer a perpetual license for the Microsoft Agent.
mux

[aman4God]

I have been researching through the forums and the interwebs and currently cannot find a Veeam best practices document for protection against ransomware that is newer than 2017. Is there something that can be provided as a typical best practice? When you read through the forums there are all manner of tricks and ideas, but short of a tape library, few cover larger enterprise implementations. Any Veeam employee feedback that can be offered would be fantastic.

[HannesK]

Hello,
I merged your question with one of the existing topics.

Most of the older topics did not change and there is a chapter for hardening in the best practice guide https://www.veeambp.com/infrastructure_hardening

Additionally to that, you might want to keep an eye on the "S3 immutable" feature with AWS and V10 (currently in beta status).



Best regards,
Hannes

[yakamoneye18]

Hey everyone,

I just wanted to remined of a air-gapped offline backup that Gostev presented in one of his weekly digests (I think it was sometime this year...). If I remember correctly he was talking about a Veeam customer who created a Windows server as secondary backup repository. After they copied the backups to this repository, the server is shutdown via post-job tasks, and everytime just before a backup starts it is booted up via BIOS. Sure, the server itself is not really air-gapped, but with the server completly shut down, the backups themselves are not reachable, only during the short period of the backup copy.

We are using a tape library with daily and weekly tapes that stay in the library, and a monthly tape that is carried to a secondary site for the worst-worst-worst case scenario. In my view tapes are really the best solution, but if we would not have a tape library, I would really try to implement the solution with the auto-shutdown - simple, afforable and still better than nothing.

Regards,

Tobias

[DerOest]

Hey Tobias and others who talk about shutting the servers down:

(At least "back in the days"...) storage/HDDs/servers in general really didn't like being powered up/down that often - that's mostly when hardware fails.

So a more hardware-friendly version would be: instead of powering down the whole server, just write a little script to enable/disable the network adapter!
This is also safer than storing a script somewhere with ILO access credentials included to remotely power it on - a hacker could find that script and use it himself

Yet, i don't like the timed approach at all, because it's no protection if someone takes a look at what you actually do/backup/have configured.
Anton Gostev wrote in some newsletters about sophisticated hackers that first manually scout out the backup servers - they would figure that out easily


I still look for a proper Veeam-engineered version of "onprem Cloud service provider with Insider Protection" - aka being able to set up a secondary Veeam server that takes replicas but disables all other network-access - while providing insider protection that could be accessed via KVM.
We all can engineer this ourselves, but not all customers are able to do so themselves - so a proper solution would be cool.
Also a great value proposition:
- Get rid of on-prem tape (which requires my precious time and fails when i'm sick at home or want to take a day@homeoffice)
- Ransomware protected
- it's truely on-prem - Cloud is not an option for many reasons for lots of companies out there

Well, just recently in Copenhagen we learned about some upcoming appliance with Linux repositories underneath... Maybe that's already digging in the right direction

[nd39475]

Lots of insightful feedback here, but i hoped to "only" include enterprise-level feedback (but with diminished state-education level budget) The USB tech does not fit us. Our tape drives, unfortunately could not retain enough of our data so we decommissioned them. (and always gave hardware issues [dell powervault]). The most interesting reply regarded datastore snapshots on cloud. The caveat is that it was a 1 day snap. Also, we are not on cloud, but may look into it (GovCloud).

Recall, this is specifically to stop ransomware infection to our backups. The repo is non-domain Linux, therefore current veeam users, admins, nor domain any other domain credentials can access it.

This is what we are currently considering. Our Veeam-Server has a secondary 10Gb NIC. We will make it connect to the offsite repo using a newly created & private subnet so that no other access is possible except the veeam-server. So now what we have is a virtual island. (repo with private bridge from server).

This will stop outside access, so now i am left with questions: can the linux stored backups still be rasomeware-encrypted from the windows server? Does it only infect the last deltas, or the entire chain?

[aman4God]

nd39475,
This is the same issue that I am trying to resolve in our organization. The vast majority of the proposed ransomware solutions are not tenable for an enterprise solution. We are a large healthcare organization and are trying to tackle the ransomware beast as well. I am surprised at the lack of direction from Veeam regarding best practices for this issue as well. I am not sure if it is because of legal backlash or just because it is a daunting task. Either way, I would appreciate something newer than 2017 from Veeam for ransomware best practices.

Our take on this is just going to be as many layers as are feasible in a mostly automated fashion. We have implemented Veeam with 2 primary datacenter sites. Backups from each site are backup copied to the secondary site. We are doing snapshots on the primary backup storage (Nimble CS-500) every hour and then daily snaps as well. I am working on bringing in a disparate storage platform for the backup copies that will have snapshots of the BCJ repos at each site as well. We are doing some replication jobs for the critical servers, backup server and DCs, as well as backing them up in encrypted jobs and using the Veeam file copy to move the backup files to a non-domain joined Linux repo that has minimal access. The 3rd offsite copy will be Wasabi immutable buckets once we get all the paperwork, risk assessments and contracting paperwork done. We are not a cloud company yet, but are working on it with VMC on AWS, Wasabi, and minimal Azure.

The additional steps we have/are implementing are strict firewall rules for all the Veeam servers in the environment. There are some hardening documents from Veeam that we are using to increase the defense in layers principle. In addition to that we are also using separate limited rights service accounts for the different backup functionalities.

My problem is I am not a security engineer and honestly don't know if these are the correct steps to take or not. In a recent ransomware event I heard about, the Veeam repos were all encrypted but they were able to fully recover using the restored snapshot from their NetApp FAS device. However apparently the FAS system files themselves were encrypted but because everything was running in Cache it didn't cause an outage. They had to work with NetApp support to get the system files recovered and reboot the array but this was after the storage snapshot was recovered and the VMs were all restored from Veeam.

If anyone identifies glaring holes in this thought process, feel free to respond.

Security Recommendations
https://helpcenter.veeam.com/docs/backu ... l?ver=95u4
Veeam Best Practices Guide (Hardening)
https://www.veeambp.com/infrastructure_hardening

[skrause]

While not an "offline" copy, having your backups stored on a SAN with snapshot capabilities and doing a snapshot of the SAN volumes shortly after your backup window ends (obviously only keeping a few days worth) is a decent approach that scales into the "enterprise" level better than USB drives.

There still is nothing better than tape (and likely isn't going to be anytime soon) for true "cold" backup data for a variety of reasons. And while even a small library is not inexpensive, tape cartridges are dirt cheap when you compare them to even the most inexpensive consumer hard drives.

Also, Rick Vanover and Joe Marton have given a presentation about ways to protect yourself from ransomware at the last 2 (maybe even 3?) VeeamON conferences and I believe it has been a webinar at least once since. Maybe they could make the slides/info from that available?

[jandrewartha]

With v10 offering immutable AWS S3 backups, what are people's thoughts on that vs tapes? I'll have to do some cost comparisons.

[Gostev]

Tape will win it