Reply to Gostev, Air-gapped backup

[hyvokar]

Just as I was thinking about this, I had to go to my computer BIOS settings to make some changes, and spotted one interesting functionality there that could be the answer – the ability to power on the computer at the scheduled time (time and day of week). I don't know how common is this functionality is between motherboard manufacturers, but I assume it should be common. So, my idea here is to have the dedicated "air-gapped repository" server with JBOD that serves as the target for, say, weekly backup copy jobs of most important backups. You will use this BIOS setting to power the server on right before the weekly backup copy job is scheduled to start, and then power it off with the post-job script command. This way, it will remain powered off for the entire week with absolutely no way to control it remotely. Could this be the cheapest "fire-and-forget" air-gap solution for everyone?

Hi,
Couldn't find how to reply Gostev's weekly email, so I'll just post here.
Setting up computer to start at scheduled time is ancient tech and widely used. The main problem with a weekly backup is, that I'm sure there are ton of users who cannot lose a week's worth of data. At least we cannot. Then again, running this backup daily, would expose the remote machine for attacker (depends on how long your backup takes), but still, it's better than nothing

My feature request for VBR would be 'write protected' backup, so you could not delete the backups from backup server console until set period of time.

How I've implemented our 'air-gap' at the moment, is far from ideal. I run a backup copy job on site1 and target it to site2 server1. Site2 server1 is in different domain (would be way cool if the target computer would not need to ba a part of a domain....) and has a completely different set of credentials than anything on site1. Server2 on site2 is running a script which copies the backup daily from server1. First of all, this is VERY slow (copy takes ~15hours, since it has to copy full vkbs every time), and secondly it wastes several terabytes of space.

'Write protected' -backup would greatly improve the situation.

[Rick.Vanover]

Cheers, hyvokar - one thing I'll note is that WORM tape media support is coming in Update 4, so that is an option also.

[hyvokar]

We have just given up using tapes, so that's why I'm looking for another air-gapped solution

[Rick.Vanover]

I need an enterprise equivalent, but at home, I use a managed power switch. It is on only for specified times.

Maybe PDUs can do this for switches or drive systems.

[hyvokar]

Still, this is far from ideal. Let's say your VBR server is compromised. A smart attacker would then wait for "air-gapped" system to come online and do his bad magic.
As we see, people have come up with many different solutions for "air-gap", some better, some worse than other. I think there's a real need for Veeam to implement this.

[Rick.Vanover]

I would rather see things like shares or LUNs be able to be taken offline via a Veeam call when writes are done - and brought online when writes are soon to come.

[Gostev]

My feature request for VBR would be 'write protected' backup, so you could not delete the backups from backup server console until set period of time.

Unfortunately, such feature would be utterly useless, because backup files can still be easily deleted using standard OS tools. In fact, this is how it's usually done anyway - most hackers don't even bother starting the backup console, when it's way faster to just run rm /rf or format d:

And, needless to say, cryptolockers don't bother going through the backup console to do their thing either so, with this feature, not only you will still get your backups encrypted and unusable, but you also won't be able to delete them through the backup console to free up disk space for the new ones!

[Gostev]

I would rather see things like shares or LUNs be able to be taken offline via a Veeam call when writes are done - and brought online when writes are soon to come.

But basically, that's exactly what I suggested in the digest then? Importantly, it must not be Veeam bringing them online, because it means there's an API that "bad guys" will be able to use as well - which is why my solution uses "self-contained power" on approach.

I do agree it's not a bulletproof solution, since if the fish is big enough, smart hacker will monitor the environment for weeks before executing the attack. But time is money, so they won't bother wasting it on smaller environments which are unlikely to pay big buck.

Anyway, thanks to all the feedback I think I found a way better solution that should protect against any attack except from insiders with physical access! And it's even more secure than storage snapshots, albeit less "convenient". Just need to polish and confirm a few more things... stay tuned for the next digest

[hyvokar]

Unfortunately, such feature would be utterly useless, because backup files can still be easily deleted using standard OS tools. In fact, this is how it's usually done anyway - most hackers don't even bother starting the backup console, when it's way faster to just run rm /rf or format d:

And, needless to say, cryptolockers don't bother going through the backup console to do their thing either so, with this feature, not only you will still get your backups encrypted and unusable, but you also won't be able to delete them through the backup console to free up disk space for the new ones!

Hi,

Please re-read my post.
You cannot use standard OS tools to delete backup files from remote system, because you don't know the credentials (I assume they cannot be stealed from the VBR database), and firewall is blocking all but VBR communications.

[Gostev]

Well, that's just a wrong assumption then. You cannot protect stored credentials which are actively used by apps - be it Veeam Backup & Replication, Google Chrome or even Windows own stored credentials. Because if the application itself is somehow able to retrieve and decrypt them, then any other code running on the same machine with local system privileges will be able to do so too. Not to mention that when the application uses those credentials, they stay encrypted in memory and can potentially be intercepted there, even though this is usually the hardest way to get a hold of them.

[mma]

@Gostev

Just a quote from your diggest: "This way, it will remain powered off for the entire week with absolutely no way to control it remotely."
If you already are in your BIOS settings, turn off WOL! Otherwise someone is able to start your computer with nothing else than a bit of magic...

[Gostev]

Yup, quite literally "magic packet" however, I thought WOL does not work on powered off computers, and can only wake computers in sleeping states (S1 through S4)?

[skrause]

WoL can work on powered off machines. It depends upon how you configure the BIOS/UEFI on the targets.

We used it to turn on public use machines at the library I worked at every morning 30 minutes before open as they shut down every night at closing time.

[MichelZ]

I think the most important and (IMHO) best solution gets overlooked here. Online disk storage with WORM support (e.g. NetApp, many others), Snaplock and the likes.
If the solutions are properly designed, then you can't delete any data from them, even if you wanted to (as admin). This would require some cooperation with veeam though (to set the retention period on the files), is this something you support? (Haven't looked into veeam in a while...)

[wla]

the only way to populate one with backup copies will be by pulling backups FROM your primary backup repository

But, when the primary backup will be compromised (encryption malware), then the secondary repository is compromised as well.

[hannisch]

Hi,

I good choice if the backup size is max. 4TB is using RDX tapes. They are recognized like normal usb drives. I use the Freeware tool freeeject to eject the tape after backup.
There is one disadvantage regarding the restore points, so I have a feature request. In the restore wizzard, when using rdx or rotating usb drives, only the last restorepoint is shown. Isn´t it possible to show all restorepoint, even the offline?

b/r

Sven

[Elemer.gazda]

I like the idea of a server which is not allowing any traffic in, and can only be managed from the local console, and dismounts the disks and disables the network interface as well.
One feature request for this:
It should interact with the original Veeam server somehow so that from the veeam backup server console (Or event Log, or just log files) you are able to see if the "Air-Gapped" backup had finished successfully or not.
Otherwise it would be a real pain to have to physically check the Air-Gap server and see if the files have been copied over or not. Major issue if you remotely manage tens of Veeam backup servers.

[mma]

If you do all the effort to have a physical server, do some scripting for file copy, firewall restrictions, monitoring all the stuff....
Why you don't just buy a LTO library / drive? Eject the drive after the job is finished and you are all good.

[pkelly_sts]

Because re-inserting the tape/drive requires manual (human) intervention so can be missed.

I'm glad I came to this topic as I was discussing similar scenarios with a colleague and we had initially settled on WORM as being most attractive but I hadn't realised that Veeam doesn't actually support it yet - I just assumed it would work.

Our solution uses two libraries, different sites with tapes just auto-rotating but I did point out to the business that a malicious internal user could theoretically format/erase all tapes before doing the same to production storage but the risk was deemed low enough at the time to not need to mitigate.

However times are changing & the risk is up for discussion again...

[Gostev]

I'm glad I came to this topic as I was discussing similar scenarios with a colleague and we had initially settled on WORM as being most attractive but I hadn't realised that Veeam doesn't actually support it yet - I just assumed it would work.

Actually, you can go ahead with the procurement as WORM tape support is a part of Update 4.
Also, that other thing in your signature is there as well so, start thinking about the new one!

[pkelly_sts]

Actually, you can go ahead with the procurement as WORM tape support is a part of Update 4.
Also, that other thing in your signature is there as well so, start thinking about the new one!

YAY! Double-celebration!

[Mikejden]

Anton,
Your proposal makes sense and there is precedence for it. Companies with dependencies on OT (think SCADA systems) use one-way communications to limit access to them. The key piece in all of this is that the OT environments, in addition to pulling patches where needed or any other information, need to push files to the monitoring environments so that they can properly be watched in case of any issues. I would recommend you add that piece to your proposal. The isolated environment, in addition to pulling backup snapshots, needs to send log information out to the designated Syslog (Linux) or Centralized Windows event servers to allow for proper monitoring of the environments.

[Frosty]

Finding this discussion fascinating. I oversee a fairly small environment (50 VMs and about 4TB of daily backup data). 100% virtual, apart from a physical Veeam backup server. We back up every VM every night to that server, keeping the last week's worth of backups onsite for quick restore if required. But we also copy everything every night to sets of removable HDDs and have them stored offsite in a secure facility. Touch wood, that has worked well as a strategy so far. We perform regular DR test restores of that data from the HDDs.

About 12 months back I considered also having an offsite over-the-network backup. I intended setting that remote location up so that it could copy data in from our backup server, but would have prevented any ability to connect to that facility via the network. So connected, but blocked via firewalls. We ended up not going ahead for two reasons: (1) cost comparison with portable HDDs was poor; and (2) if someone gets control of your firewall, then they could remove the blocks and still gain access, so you'd still need the portable HDDs air-gapped backups anyway.

[mma]

[snip] when using rdx or rotating usb drives, only the last restorepoint is shown. Isn´t it possible to show all restorepoint, even the offline?

Wow, I'm pretty sure there was an answer to this from Gostev, on which I made a comment myself.
The post from Gostev is gone, mine too.
Feel free to delete your own post, but what happened to mine?

[Gostev]

@Marcel I've cleaned up some off-topic, otherwise the discussion has really started to deviate from the main topic. And I sort of started it myself with my original response, so I apologize for that. So, since there's already an existing thread that you linked, I've deleted our entire exchange.

@Frosty actually my proposed solution solves issue (2), did you have a chance to read last week's forum digest?

[davidwatts71]

instead of powering off the machine we could just power off the network switch using a NETIO 4 smart power socket. I think a network switch should not have a problems with a sudden power off and on but I might be wrong. You could then use powershell\lua scripts to power on the switch, copy the data and power off the switch again. The Netio 4 also has built-in scheduled task to switch the power on so that would be the same as using the BIOS to power on the machine.

[Frosty]

@Gostev yes, I read all your digest emails; they're an excellent source of information and they're required reading for me.
I have a high level of paranoia. So *any* form of network connectivity is a risk that I am unwilling to absorb. I did like the proposed solution, to summarise: pull the backups from another network-connected location and lock all that down very tightly (e.g. console access only, etc). It is a very similar idea to one that I considered implementing a year ago.
But if the data is on a connected network,anywhere, then to some degree it will remain vulnerable. Firewall rules can be changed if the firewall is hacked. Console-only access can be changed if the environment around that console-only server is breached. Paranoid? Sure, I'll wear that with distinction.
We're not a high-value target (I work for a not-for-profit). Touch wood, we've never been breached AFAIK and regardless of all the over-the-network backup solutions, I will *still* always want an air-gapped backup as my "last resort" get-out-of-jail-free card. I'm lucky that our environment is small enough to be copied onto portable HDDs. Big environments would not have that luxury.

[xudaiqing]

My current solution is have is setup a isolated hyper-v server (no network connection to root partition only local console) and run a vm on it as backup repository.
Then create daily checkpoint/snapshot for the it. When use with refs it has good enough performance for our size (around 4TB).
As long as the isolation between VMs isn't breached, it should be safe.

[Mengisman]

But, when the primary backup will be compromised (encryption malware), then the secondary repository is compromised as well.

I think this is an excellent point that surely needs consideration and my solution will be to ensure that I have enough storage for at least 2 or ideally, more full backup copies and make sure I am not overwriting the last most current backup with the new incoming backup.

[caztor]

There are so many good points being made on this topic - For most of us, I think we would like an online solution that would still satisfy the requirement of being air-gapped. That way we could still automate things (without investing in a tape autoloader), get reports and status back from the device.

We have had this discussion many times internally and with customers, but I think the best solution would require some extra features being added by the software vendor to really make it effective.

The obvious option going forward is utilising the "pull" method - we need to make the "air-gapped" as impregnable as possible, so we would need it not to respond to anything and preferably only accessible using a console. But we need the to know what to copy and be intelligent about it - this is where we need something from the software vendor, maybe an agent of sorts. I'm picturing something certificate based, that could talk securely with the veeambr repository, get information on what to pull and also being able to check integrity before storing and respond to abnormalities like a high percentage of changes or missing VM's etc.

I'm thinking that we can already build something like this using the API to get the information we need, but I haven't begun exploring that yet.

These are just my 5 cents - I'm curious if someone has already built something like this?