Airgap Backup ?

[gary.miles]

Hi All,

I'm just after finding out how other people perform airgapped backups. We currently use Veeam to backup to local repositories and then offsite backups to take these offsite.

With the ransomware threat etc we are looking for a solution to airgap backups as well but I cant decide on a solution, do we use tape or maybe an airgap switch with a couple of storage arrays behind it. each week the airgap gets switched over etc ?

thanks in advance

[ejenner]

One suggestion is to use a different type of OS for a separate repository. So if you mostly use Windows for your repositories you can create a Linux repository to store a copy on as well. The theory behind this is that writers of malware or attacks on your network will find it difficult to jump platforms as the exploitation will be through some flaw or using files which won't be common to both. i.e. the attack you're trying to protect your data from is unlikely to be compatible with all kinds of OS.

[pkelly_sts]

Depending on your budget you could use WORM tapes too, at least for some of the backup copies.

[gary.miles]

Thanks for the responses.

We've ruled out the other OS idea as our backup servers are locked down already however as long as they are "on the network" we worry that given time worst case scenario they can still be got at.

Only will airgapped give us that peace of mind. WORM tapes are probably out of the loop as we need to reuse the media.

We currently backup 10's of TB of data so my worry with tapes is that it will be too slow, which is why we thought about the NAS & airgapped switch idea?

[pkelly_sts]

Bear in mind that tape will always ever be a *copy* of a first-stage backup, you can't back up straight to tape even if you wanted to.

Tape also supports GFS/synthetic full backups too so daily backups can be quick (incremental) and it's only the synthetic fulls that take longer (generally at weekends).

Beware of NAS as without having an agent local to the storage, NAS backups can be very poor performing.

One compromise depending on how hands-off you need to be is that you can simply flip the write-protect tab on tapes as they fill up but you'd be reliant on people doing that, plus relient on them reversing it when the tape is due to be overwritten.

That leaves tapes in the library and available for restore, but you could also simply remove tapes & put them on the shelf/in a cupboard as dictated by your policy but, again, a hands-on process.

Finally one thing that helps with tape speed is having fast storage where the synthetic full maps are blocked out.

[csydas]

I'm not a fan of air gapped drives unless you have a very good retention/test method set up. I've worked with too many customers who thought it was just fine to toss an HDD into a safe and call it a day, only to find that years later when they needed to recover data, they plugged the drive in and it refused to spin up. I wouldn't even dare test an SSD in this scenario either.

Tape is a solid answer because at the end of the day, it has the history behind it to back the use case. Pop the tape in and be sure to eject the tapes after every write. It'll be slow (both write and read), but at the end of the day, would you rather spend a day restoring data to get your entire infra back, or start over from scratch?

Just read the best practices on Veeam Tape so you know what you're getting into. There aren't __too__ many gotchas but there are a few Veeam-isms to deal with, but it's nothing too bad. Just ask here on the forums or in a support ticket before you go all-in on a strategy.

[skrause]

Tape is your friend for pretty much any airgap situation.

[gary.miles]

Thanks all, just one last thing, what do you think to using aws as a solution ?

[csydas]

Just be ready for the bill.

I'm always suspicious of Amazon, Google, and Microsoft, cause it's easy to get your data up to them, but it's hard to get it out. Sure, they're reliable for keeping the data, but I've seen those bills add up really fast. For regular backups, a misconfiguration can easily rack up a x*$1000 bill. With a VTL, it's usable, but again, you should just be sure you understand the costs you're getting into and check what your potential DR is going to cost you when you need to do it (cause you will eventually need to)

[maryhollen]

Business located in tsunami evacuation zone: Air gap backups to rotating media, using CrystalDisk to check the integrity of each external disk before/during backup, have been very successful. I highly recommend air gaps (using a different external hard drive every week). It's a great a way of thumbing your nose at ransomware attacks, too.

How do you set the backup to make a full backup and delete the old backups when the media runs out of space? The control panel dialog is less than clear. Thank you.

[squarv]

Has anyone used some sort of out of band (dial-up) solution to enable/disable network ports or use a remote power switch to turn power on/off to a switch that would connect NAS device to the network. Turn on power to switch, copy backup files, tun power off.

[ianbutton1]

Funny, I was just thinking the same, though obviously the vulnerability is active while a backup/restore is running. Or perhaps "twin" storage - one unit off, one on, alternating daily/weekly - perhaps using onsite/offsite storage this way.
One alternative mentioned - different OS for repositories - _may_ work against an automated ransomware attack, but won't work against live hackers when they elevate their privileges to admin level. If I can go to the Files page in Veeam and see (delete) repository contents, so could a hacker. Whatever we can do, they can do - except physical access to a power-switch or network cable.
Much depends on the volume of data being protected - a few TB is OK for a removeable disk, but 50TB is a different ball-game.

[ianbutton1]

More thoughts -
1. A remote power-switch (controlled by software) could be accessed by a hacker who knows what to do - but a manual switch is safe. Similarly, software disabling of server or switch ports can be reversed by a hacker, but pulling cables gives the air-gap
2. A diskpart script after backup could offline the drive containing the repository - probably adequate protection against automated ransomware attack but not fully hacker-proof
3. Dual repositories may be an avenue to explore - with (manual) flipover power so that if one is on the other is off (and inaccessible to hackers). Dual backup jobs required (assuming units must have different ids/IP addresses). And to support that, Veeam enhancement request . . . please allow more advanced scheduling, allowing choice of weeks (plural) for "weekly" (actually fortnightly) backups - 1st + 3rd + 5th, 2nd + 4th xxxday each month (for example).