A poor man's air gap

[robg]

I'm setting up a new veeam backup server, and was thinking about other ways to protect the local backups without the use of tape, which is impractical for my clients, especially in data centers.

Then I thought wow, all you really have to do is not make the backup server a domain member. With a unique admin password, no server or workstation can access the administrative shares such as D$.

Very simple. You can't be stupid on the backup server itself of course, but this isolates it effectively from the rest of the environment. I've thought about some other methods, such as manually mounting and dismounting the backups for every job, but I wasn't sure if this would mess with Veeam because the repository would be missing. Does anyone have other techniques to air-gap?

[Regnor]

If you remove the backup server from the domain, you add an additional security layer, but it's certainly not air-gapped. If someone steals/gets the credentials of your server, then you will still lose your backup files.
Air-gapped means that you backups are technically inaccessible.
What you could so, is disabling any kind of remote access and remote management, then it would still not be air-gapped but much more secure.

Also keep in mind, that you shouldn't rely on a single system/storage for your backups. Keep at least an additional copy of your backups, on a different storage media and get it off-site(3-2-1).

[robg]

It's still "air gapped" in the sense that the rest of the network cannot access the files, only the backup server.

I'm fully of aware of everything you have raised, there are also replicas in place (which are completely out of the reach of ransomware). And no one can steal the admin credentials of the backup server.

That's why I called this a poor man's air gap, because "it's separate enough" that it is safe from ransomware.

[Mildur]

I'm fully of aware of everything you have raised, there are also replicas in place (which are completely out of the reach of ransomware). And no one can steal the admin credentials of the backup server.

Veeam Replica Job? If yes, then there reachable over network and therefore in the reach of ransomware.
Or are you talking about Veeam Backup Copys to Tape, rotated disk or Capacity Tier.

Credentials - If you have used windows to access the backup server, then the credentials could be cached somewhere. There are tools to read out the credentials.
Don‘t trust the windows security.

My personal recommendation from me: „a backup solution should not be a poor solution“. If it‘s in a homelab, I understand. If it‘s for a company, then ask yourself, how much can you loose, if someone attacks your company. Is it worth to spend some more money to secure the company and the people who work there?

A tape solution or rotated usb disk solution doesn‘t cost to much. If it‘s a very small company, capacity tier in public cloud is not that expensive.

[robg]

Thanks for your input - but I wasn't looking for a personal recommendation (but somehow I knew that somebody here would give me one anyway).

A backup solution doesn't have to be expensive, it only has to be well thought-out with the proper layers of security.

I would like to address what you raised though.

Just because a Veeam replica is reachable over the network doesn't mean that it's in the reach of ransomware. How would they get to it? They can't. Not without exploiting the ESXi server (good luck with that).

I have not used windows to access the backup server, I connect to it from a mac. My passwords do not float around the windows infrastructure at all.

The point of my post was to ask if anyone else isolated their backups in clever ways besides just physical media.

[Mildur]

Replicas can be deleted from inside the veeam console.
If an admin (insider attack, mad coworker, hacker) has access to the veeam console, the replica is gone for good
They are not „air gapped“

[robg]

Yes, and a flock of wild pigeons can suddenly come through the window and attack me in my own home.

I laid out the scenario here very specifically. There are no insiders with the password, and no one has access to the veeam console. It is safe from the rest of the network because it's not a domain member, therefore the backups are unreachable. They would have to exploit it. Good luck.

From the perspective of the rest of the network, it's a poor man's air gap. Don't take "poor" literally, that's an expression.

Please don't assume that I am incompetent.

[Regnor]

Thanks for your input - but I wasn't looking for a personal recommendation (but somehow I knew that somebody here would give me one anyway).

Then why did you post your setup here anyway, if you didn't want any recommendation or feedback?
There are pros and cons for everything, also depending on different scenarios.

Not without exploiting the ESXi server (good luck with that).

https://www.zdnet.com/article/ransomwar ... ard-disks/

[robg]

"Does anyone have other techniques to air-gap?"

^^^ This is the only question that I asked.

I'm aware of the ESXi vulnerability. That was from february, it was patched recently. So again, not without exploiting the ESXi server (good luck with that).

[soncscy]

Just my $.02, let's call your strategy what it is; another zone in the security model: https://bp.veeam.com/vbr/VBP/Security/H ... Zones.html

I'm not trying to dump on your idea, it's just from my perspective, there's no real benefit to invoking the title air-gap, even with the caveat and for me anyways I feel it introduces confusion. I prefer clean definitions.

It's still "air gapped" in the sense that the rest of the network cannot access the files, only the backup server.

I wouldn't really accept this definition to be honest, since remember, the model for ransomware is get in, and sit. "Smash and Grab" runs of course happen, but ransomware attackers happily sit for weeks, months, and in huge cases, year+ to wait to get the environment in a situation where they can introduce the highest amount of pain when the attack is launched. It's been shown by lower quality ransomwares that the ransomware community is well aware of backups and even has some detection to catch various backup products backups/services (Veeam included), and they enter it into their strategy.

Your strategy of removing from domain is a great one, don't get me wrong, it should be done. But, if we study how previous ransomware attacks have gone, then we see this is not an insurmountable obstacle for attackers by any means, which is why the redundant off-site copies and air-gapped copies are properly secured.

Tape, physically disconnect the server, rotated drives, these are all fine and they're the best way to do this. I remember seeing something about some remote physical network switch, but I don't recall the details, but it seemed like a great thing -- you press a real button and there is a physical disconnect somewhere, which I like the idea of, but not sure how reliable or easily implemented it can be.

[Gostev]

Since no one mentioned this yet: did you consider implementing the V11 Hardened Repository with immutable backups? Properly deployed, this will provide the protection level equivalent to air-gapped backups. It won't provide protection against malicious insiders, but you already said this is not something you worry about.

[robg]

Just my $.02, let's call your strategy what it is; another zone in the security model: https://bp.veeam.com/vbr/VBP/Security/H ... Zones.html

I'm not trying to dump on your idea, it's just from my perspective, there's no real benefit to invoking the title air-gap, even with the caveat and for me anyways I feel it introduces confusion. I prefer clean definitions.

That's why I called it a "poor man's" air gap. If the backup server is properly secured and completely unreachable from the outside, there's no way they're seeing the backups. That's also why I said "you can't be stupid on the backup server itself"

I wouldn't really accept this definition to be honest, since remember, the model for ransomware is get in, and sit. "Smash and Grab" runs of course happen, but ransomware attackers happily sit for weeks, months, and in huge cases, year+ to wait to get the environment in a situation where they can introduce the highest amount of pain when the attack is launched. It's been shown by lower quality ransomwares that the ransomware community is well aware of backups and even has some detection to catch various backup products backups/services (Veeam included), and they enter it into their strategy.

Your strategy of removing from domain is a great one, don't get me wrong, it should be done. But, if we study how previous ransomware attacks have gone, then we see this is not an insurmountable obstacle for attackers by any means, which is why the redundant off-site copies and air-gapped copies are properly secured.

I get all that, but it's a little bit irritating that everyone just assumes that "it's just a matter of time before you get hit." I see marketing like this, and even the higher ups say things like "If and when" - just because it's in the news so often.

No, strongly disagree. I'm not exaggerating when I say that the only way they're getting into the backup server is if they find out who I am (not an employee), know where I live, break into my house, and manage to get on my screen before it locks. But there's no way you could know this of course.

I work with small organizations. Yes you're right, it's easy to miss something in bigger ones, and this usually comes from phishing (poor email security, poor user training), weak passwords to open RDP ports, or careless admins infecting themselves with full domain admin rights. Not impossible things to control, but harder for some vs. others let's just say..

Tape, physically disconnect the server, rotated drives, these are all fine and they're the best way to do this. I remember seeing something about some remote physical network switch, but I don't recall the details, but it seemed like a great thing -- you press a real button and there is a physical disconnect somewhere, which I like the idea of, but not sure how reliable or easily implemented it can be.

Tape and rotate drives are impractical for some.. Your suggestion is sound, though.

Since no one mentioned this yet: did you consider implementing the V11 Hardened Repository with immutable backups? Properly deployed, this will provide the protection level equivalent to air-gapped backups. It won't provide protection against malicious insiders, but you already said this is not something you worry about.

And Gostev, that's a great idea. I haven't looked into this but I'll read about it.

[robg]

Gostev, how does the hardened repository stop the physical backup files from being encrypted? On a windows server.. I'm sure I'm missing something obvious, I don't see how this is possible without removing the OS's access to the files somehow.

[Gostev]

Yes, essentially that's what we do through the immutability flag. However, it's Linux-based technology. Just search the forum, or Google for Veeam Hardened Repository, and you will find tons of information that answers all imaginable questions

[robg]

Ok, so it requires a linux repository..

[Priit]

Besides our main domain connected backup servers, we use a poor mans Linux equivalent repository, that is just a home/small business grade NAS box with only local auth and no domain integration and have a copy replica of all backups copied there on cheap SATA drives. We keep a month worth on there. We also have a slightly less poor mans option in use as an extra layer to keep 2 weeks worth of backups on Azure block blob. This is not accessible with domain credentials either, but access to its credential strings is available with Azure tenancy Global Admin credentials. We know none of those are air gaped, however they are 3 separate methods of authentication and storage not accessible for general or even admin accounts on the domain, so the attack needs to be very sophisticated and targeted to the specific repositories and systems. Though I agree that an attack that can breach the Veeam software itself to use its methods of accessing those repositories may be able to scramble them, as Veeam does have the access credentials and methods in its credential manager.
I can think of a couple of ways to increase the security by creating partial air gap by running timed scripts on the infrastructure to disable LAN ports for the Veeam repository appliances or network switches and only enabling for the time when the replication of new backups is scheduled to happen. If a restore or testing backups is happening can enable the ports manually and disable after, but not sure it is worth the extra complexity and loss of alerting from the appliances for majority of the time. Will have to bring that up the next cyber security meeting...

[robg]

I was wondering about this too Priit. In the case of a local windows volume, to run a pre-script for the first job that mounts the backup drive, and re-mounts it after the last job is done.. this would make the window of opportunity for an attack smaller, but I wasn't sure how Veeam would react to its repository disappearing off and on..

[Mlubbe]

I get the intent and we do some things similar.

Step one is to make all of the backup images and store them long enough to make some copies.

Step two is to push copies of all of the images to a backup server (synology device) that is on a schedule to power up at the prescribed time before the job starts and then powers down once we are done moving data. (Poor man’s air-gap #1). It is only on long enough to catch the files.

Step three is to push the images to a rotating removable SSD drive that then is dismounted via script upon completion of the move. (Poor mans air-gap #2). We rotate 6 of these drives. When starting the copy, a script will wipe the drive clean.

Final step is that we push them out to the cloud into a bucket that is a write only repository (for the creds used in Veeam to authenticate) and has policies setup to not allow any file to be deleted for 12 months, at which time they automatically are removed by bucket policy. I cant delete them unless I completely killed the account and associated buckets. (Poor mans air-gap #3). Yes there is a cost to holding 12 months of backups that cant be deleted, but in the big picture, it is worth the $400/month cost for peace of mind. With the write only policy, I cant read/compare what is there, hence we just push new full images in every time.

Next round of backups, wipe the local copies first, then make new images.

[NaplesDave]

If you want a REAL AIR GAP device check out this https://www.techbyking.com/project-gallery-page. Scroll down for AIR GAP Device.
The device sits between your NAS or external backup drive and physically disconnects the ethernet from the device during your specified backup time.

[mrecob]

We have Veeam B&R installed on an older Dell server. It is not on the domain. It has a complex password. We use DUO for two factor authentication to log in. All backups are stored on a NAS box connected to a second network port that is separate from the corporate LAN and Internet. We also have our backups sent to the cloud, Wasabi. In addition, once a quarter we copy our backups to a large hard drive and put it in our fire proof safe though once a month might be better. We are a small 100 person company. This did not cost us much to implement.

[ARHT]

A simple part-solution is to have the back-up controller/initiator on a separate switch and then use a WIFI or network controlled power adapter to power-off said switch on a schedule (or manually if that works for you) outside of backup periods, essentially making it air-gaped for a portion of the day. Obviously this will only work in some scenarios and is by no means a full solution, but for small operations can reduce the attack surface a little more.

[thomas.biesmans]

As others have mentioned, not having the backup server be part of the domain is a very good first step, the first of the low hanging fruit, but it's still only step 1 though. Step 2 is using firewalls and segmenting management networks. My personal motto is "a port that's isn't available, cannot be exploited". And as was already mentioned, if they take their time and you as a target are juicy enough, then it's only a matter of time. Things get automated on the attack side as well, so I don't think our job will become easier in the future. And exploits are a big part of that job already.

We used to add our virtualized backup servers & backup hosts to vCenter, good for convenience, but a horrible SPOF from a security PoV. It's all about that risk reduction. Gostev mentioned the hardened Linux repo: if properly set up, there isn't much that can offer a better bang for buck, similar to offloading to the cloud with object lock. So it's getting introduced in all of our new projects while we eagerly await Azure immutability. No air gap, but at least we've got that immutability that a vaulted tape offers.

I support that call for clearer definitions, just removing it from AD is called operating in workgroup mode, that's not air gapping at all. Only marketeers would bend definitions like that. But a total air gap isn't practical. So that single vulnerable RDP port in the firewall would do, or a jump host with MFA as a next logical step. Perhaps complemented with timed physical disconnections for one of your copies if you want (although I'm not a fan, it might work with some fancy disk / NAS rotations, but imho hardened Linux repos took away much of their appeal). In the end it's a costs vs benefits analysis for a customer, hopefully educated on available options and their risks / downsides and aware that there is no such thing as perfect protection.

[SergS]

In my understanding "poor man airgap" is when backup server is compromised it is from very hard to nearly impossible to get access to the repository. So how can we achieve this with Veeam? We know that all credentials from all infrastructure added to backup server are stored encrypted by DPAPI in Veeam database, so we need a repository which can not be directly controlled from the backup server and its root creds are not stored on the backup server. The best way would be to use tape library with WORM tapes, but we need ultra cheap solution, so why not try Virtual Tape Library. QUADSTOR VTL supports WORM pools and its free and can run almost on any hardware and OS of your choice. So even if backup server is totally hacked, the attacker should not be able to encrypt or remove backups from VTL, since he\she can't directly control it.

[robnicholsonmalt]

I would assume that resolving this is something that is now probably on Microsoft's radar. There are still companies running their entire operation on one physical server (yes I and they know the risks) and a client recently had their entire server and Veeam backup repository killed by ransomware. What one needed was totally locked security on the Veeam repository folder that *only* the Veeam account can access with *no* backdoors. Or some totally different API for accessing them etc. Sure, still only a poor man's airgap but still harder to compromise.

[jraymond]

We have Veeam B&R installed on an older Dell server. It is not on the domain. It has a complex password. We use DUO for two factor authentication to log in. All backups are stored on a NAS box connected to a second network port that is separate from the corporate LAN and Internet. We also have our backups sent to the cloud, Wasabi. In addition, once a quarter we copy our backups to a large hard drive and put it in our fire proof safe though once a month might be better. We are a small 100 person company. This did not cost us much to implement.

This is almost exactly our set up. But, I have 80TB of internal HDD I use for the backup rather than a NAS/SAN device. And FYI, we did get hit, never touched the backup server as they couldn't see it. I had all but 1 of my 27 servers back up and running before most people came in at 8 AM. (I arrive around 5:30AM everyday).

Only difference is we use iLand for Cloud Backups, Long Term Storage and for our DR. Very affordable and are a solid partner with Veeam. Working amazing for us.

[Zew]

a poor mans "air-gap" is more what I did, grab a laptop, configure it has a Veeam Proxy, connect big enough USB drive to it, create a BCJ of all your backups and use this proxy's USB drive repo, and rotate the repo out on your own schedule. Redundancy comes from the amount of USB drives you rotate out.

*NOTE* One caveat is if you don't delete the old backup data before running the job the change block tracking has to do validation on what's changed and this I/O can make the job take forever, you are better off deleting or ensuring to only attach blank drives on a drives rotation. This way the job is nothing but sequential writes and will be MUCH faster.

[billeuze]

We are a relatively small shop, 10 - 20 servers, with a "poor man's" IT budget. We have a true air gap using rotating USB disks. Our B&R server is on the domain but our repositories are not domain joined and they are on a different subnet (not internet connected) than our domain. I have one repository that saves new full backups each day. We then have a small powered off Dell computer connected to the repository subnet. Its BIOS powers it up each night at 10:00pm, then it runs a script that robocopies everything in the repository to an external USB disk. The final line in the script is a shutdown command. By the time I get in at 7:00 am My backup results are in my email, the computer is shutdown and I unplug the USB disk to plug in the one for the next day.

So far this is working great as long as someone swaps the USB cable each day. This is a minimal 4TB backup of all essential business services. Some times the backup data gets beyond 4 TB and then the backup script is still running for 1/2 hour or so when I get in at 7:00. if our backups grow much more we will have to switch to removeable SSD like Mlubbe above.

[molan]

I hate to say this as I am sure you will react badly, but have seem to be extremely arrogant in your thinking there is no way someone could ever get in to your server without your password. Its like you want to call out all the trolls and hackers and put a big target on yourself.

Also your replica's are 100% vulnerable. because of how a replica needs to function to be useful. The bad actor doesn't have to get access to the replica storage location to corrupt them. All they need to do is subtlety corrupt the source before you know there is a problem. The replica process itself will do the rest.

This means the second you are attacked and a server is encrypted then your replica's are also encrypted and useless. Now I know you will say, but I can go back X number of days with my replica's, but it isn't good enough. Ransomware often lays dormant for this exact reason and will very probably be in those older images and will re-activate once restored encrypting your data gain. This problem exists for backups too.

But to answer your question here is what I would suggest.

1. Immutable Backups. Both for on premise and your cloud copies ( you are storing copies off site correct?)
2. MFA on your backup server. Use a service like DUO mobile to add an MFA layer onto the RDP and console access of your backup server gives you an extra layer of protection beyond your password. and its free to use DUO for up to 10 users.
3. close all inbound firewall ports on your server except RDP. this a good step, but something you will have to constantly review as both windows and Veeam love to fill the windows firewall with holes every time they update.
4. better yet put a physical firewall in front of the backup server if you can.

I agree that airgaps are not a practical solution for most. I haven't been physically in the office for over a year myself with Covid. There is no way we could have someone reliably swapping taps (or any other media) on prem these days.

[colohost]

I can't say I'd feel particularly comfortable with any of the not-actual air gap "equivalent" suggestions in the thread, if the intent is to mitigate ransomware and not just to mitigate operator error / accidental backup deletion. The frequency in which there are remotely exploitable Windows and vSphere/vCenter vulnerabilities seems like it's a several times per year thing, and add in the need for enforcing best practices on those servers' configs, whether that's windows credential cache, admins using domain admin accounts when they shouldn't be, so on and so forth. There are an awful lot of weak links in such a config and ransomware tends to know how to find those.

The same entities in need of a poor man's air gap are also extremely likely to be the ones without sufficient manpower or redundant equipment to keep services online and be applying critical updates immediately. For example, such an entity is probably not running Enterprise Plus vSphere, so doing maintenance on a vSphere host without vDS likely means hand migrating every running VM off a host since you have to pick the target network, patching, and then hand migrating them back. Or, is such an entity really taking their windows servers offline the day after critical patches are released? Probably not since they probably also don't have redundancy for whatever services the given server was providing. Next thing you know one or more systems is out of date, and ransomware gets past the poor man's air gap by way of lingering exploits.

Regarding, Hardened Repository, if that simply relies on toggling the immutable bit at the filesystem level of the linux repo to keep those backup images "safe", I don't know that I'd ever feel comfortable with that from a ransomware perspective. While not as often, linux certainly isn't invulnerable to exploit, or an authorized server admin's ssh key being in too many places, or someone's computer being hacked while having an ssh key agent running with the key still in mem, or they live on a VM on shared storage where there's other ways to get to it, etc. Once they're in, chattr the bit back off and remove.

OP, if it isn't a geographic issue, you may want to shop around among data center operators to find one that's more cost competitive for doing remote hands type work, such as swapping tapes. Most modern tape libraries have the bulk of the cost on the drive side, so if you can live with less drives, perhaps you can get an affordable unit with a fairly large import/export door that lets you eject an entire full backup into one 'container' of 8 or 10 tapes, and the data center staff can just yank one and stick it in a cabinet and replace it with another. Or, authorize a company like Iron Mountain to enter the cage/rack and swap the tapes along with taking them off site. I get that these solutions would certainly be more expensive than spinning up a repository server and hoping for the best, but in the grand scheme of things, they aren't *that* expensive. For example, I pay effectively a few hundred dollars per month for the data center remote hands to swap a bundle of ten tapes weekly. That plus the tape library is far less expensive, and less risky, than staff time developing workarounds for true air gap, along with staff time executing and monitoring that the poor man's solution is actually still functioning as intended, which it inevitably won't be when the time comes to need it.

If the full backup size is compatible, I'd go with the cheap-o USB drive option and a rock solid multi-employee plan to monitor that it's being executed properly before any network-based solution where you hope for the best.

[Gostev]

Regarding, Hardened Repository, if that simply relies on toggling the immutable bit at the filesystem level of the linux repo to keep those backup images "safe", I don't know that I'd ever feel comfortable with that from a ransomware perspective. While not as often, linux certainly isn't invulnerable to exploit, or an authorized server admin's ssh key being in too many places, or someone's computer being hacked while having an ssh key agent running with the key still in mem, or they live on a VM on shared storage where there's other ways to get to it, etc. Once they're in, chattr the bit back off and remove.

Except "properly deployed" hardened repository means disabling the SSH Server, which makes all your SSH key concerns irrelevant. With the only "way in" being the local server console, you can forget about any possibility of remote attacks, even if hackers have root credentials to your entire environment. They simply can't use them remotely, that's the whole idea!