Offline airgap backup

[jandrewartha]

From an old thread:

It'd be nice to having something that pulls the image, we're pondering ways to get an offline copy of our long-term (12mo+, 26TB atm) backup copy repository that's updated once a week, both for cryptolocker recovery and in case the whole server dies in a fire for some reason, we don't lose that backup history.

Right now I'm thinking a two stage backup, Veeam agent backup to our old netapp, then a small server that boots up automatically (BIOS/UEFI control?), copies the image over, emails success then shuts down again, but I'm sure this could be simplified.

Bringing this up again after Gostev's weekly post. I think the big barriers are how does Veeam handle a repository that is offline some of the time, and that you really want a pull model for the airgapped backup, preferably only done once it's confirmed the backup is recoverable (ie not encrypted by a cryptolocker), not an automatic push. It needs to be pull so that only the airgapped machine has the credentials to write to itself, otherwise again an attacker who gains control of Veeam can just wait for it to come online again and overwrite it. And it also allows for the user to just power it on manually when they want to make an offline copy, if they don't have a BIOS with power-on controls (although they are fairly common in my experience).

And finally the specific question from that thread, how do I get a duplicate copy of an ReFS repository will all the history in it without reinflating everything?

[staun]

Will follow this topic as I have the exact same concerns and looking for the right solution for an offline backupCopy. My current plan is a local server, but as you mention - Our main Veeam server has access to this server...

[Nils]

With a network share, you can also hide backup folders from the share.

Depending on whether the rotation uses physical drives or just folders, you can mount and unmount the drives/folders into the repository folder using scheduled diskpart or makelink (don't use symbolic links as those are resolved by the client). That way, unused and unmounted/unlinked folders are simply not accessible outside of their backup window. Veeam requires the " This repository is backed by rotating drives" to cope.

You should use Veeam's Job scripts option to create and delete status files for "job xy is running", so you don't dismount the folder that a job is currently running on.

Also, you might want to turn off the default admin shares and explicitly not grant your backup (or domain) admins local admin privileges on the repository, so an attacker can't leverage here.

[xudaiqing]

I think storage snapshot of backup repository with a air-gapped management interface is better and easier.

[olavl]

For the paranoid you need offline and offgrid.

Offline: Protects against virus/ransomware attack and protects against accidental delete. The files are not directly accessible on the network so an infection or accident cant reach the backupfiles.
- Non AD-member and locked up Veeam
- Put files on SAN/NAS and snapshot
- Azure/AWS storage (soon I hope)

Offgrid: Protects against hackers intentionally accessing and deleting backup files. Storage snapshot is fine for offline, but if the case is a hacker on your network with admin access (always assume the worst), the rogue player has access to both Veeam and storage and can delete everything.

Offgrid is the problem and I havent found a good solution. How can I put the backup files so they are accessible to me in case of restore, but impossible for me to delete them?
- LTO8 tape in safe?
- Some sort of snapshot in Azure or AWS?
- A VTL that requires console access to delete?

[wleister]

Not exactly offline, but we recommend HP storeonce as a backup target. It integrates with Veeam using a catalyst share. We have never had any customers have the storeonce get encrypted with ransomware. You can also have storeonce replicate to a storeonce at another location or the cloud if you want extra protection.

[ChrisSnell]

ExaGrid also uses the Veeam Data Mover. The backup share is therefore neither CIFS or NFS, but a Veeam share. The only way to access it is via the Veeam server using a specific username and password. It's a virtual air gap. The Data Mover also increases performance

[Gostev]

@Chris, in virtually all successful attacks we saw to date, the hacker was taking over the backup server and thus had access to all stored credentials. So unless by "virtual" air-gap you mean "non-existent" then I would disagree this provides any air-gap whatsoever.

The common confusion seems to be when people think cryptolocker attacks on their Veeam backups are about some super smart Veeam-aware cryptolockers looking around for accessible repositories and file shares holding Veeam backups. In reality though, we've NEVER seen a successful cryptolocker attack that did not involve a hacker penetrating the environment to manually find and delete (often with zeroing out disk) the backups before manually sicking a cryptolocker on the most critical production data they identified. With the initial malware being used solely to provide a hacker with the way into the network perimeter.

[DerOest]

Hello Gostev,

as always, i loved reading your Newsletter this morning!

You casually, non-saleslike mentioned "Veeam Cloud Connect with Insider Protection". Wouldn't that be the best option, to bring something like that on-prem?

Something roughly like

• User installs local repository operating system (Windows or supported Linux distribution - make sure to disable ILOM on that hardware server, not AD-joined)

• In Veeam, add it as Repository with an option like "Delete Pretected" + select how many days/weeks backups should be delete-pretected

• Show a complex password, protect the configuration of this repository with it

• Run a script on the Delete-protected Repository - lock down the Firewall [except Veeam Ports], etc.

• Disable all local Users except one with that password

That way, you could just use it like a normal repository, with the benefit of "it's all managed by Veeam".

And because only all incoming traffic [except that Veeam Rules is denied, WSUS/linux updates, monitoring (push-alerting) etc. should still work!

[jasonede]

I like the idea of a server which can only be managed from the console and pulls the backups. As far as I can see if the backup/replication server is online and remote manageable then it's vulnerable to someone with enough time and motive (and lots of money is a pretty damn good motive!). If the server is shutdown when it's not needed then it's also saving on precious electricity if it's in a data center as quite a lot of them charge by the amp. ACPI wakeup is trivial and easily scriptable (just a matter of updating /sys/class/rtc/rtc0/wakealarm or simular on linux systems).
I like the idea of only pulling the latest .vbk files and avoiding the older ones, in case they get corrupted. It is possible that a sneaky hacker could start by messing with old backups and then wait for that to pollute the chain and the older good ones to drop off the end as you'll not be able to keep all backups forever, but that would require someone to be inside your systems unnoticed for potentially months on end.
It would be nice to have a veeam script/add-on that could do this. Also I'm guessing this sort of method wouldn't be compatible with all backup strategies such as reverse incremental where a synthetic full is always generated.

[Gostev]

@DerOest "normal repository" approach will never work as it requires that the backup server is able to connect to the repository server remotely. So, it will be impossible for you to fully isolate the repository from remote access by dropping ALL incoming connections, which as you remember is the key of the proposed solution. But in any case, I want to clarify that the biggest part of the Insider Protection feature value proposition is that your data resides with the 3rd party and cannot be controlled by your own IT staff. Thus the name of the feature, Insider Protection.

@Jasonede you're correct. In theory you can make it work with all backup strategies (again, with things like rdiff-backup) but explained well why it is a bad idea. This will work best with any backup modes that allow you to copy new backup files only - so any backup mode with periodic fulls (as with these backup modes, backup files are never modified once they are created, so there's no point of copying them more than once).

[dpeach1]

What about the integrity of the Veeam Server? Seems like a chicken egg problem. if the Veeam server is crypto lockered, along with the agent based backups, what do you do? The Veeam backup on tape is useless if you cannot recover the server to access the tape library. Do we need an air gapped copy of the Veeam server backup with recovery disk files along with the Veeam configuration backup? Put it on a thumbdrive? This becomes very labor intensive.

[Gostev]

Just reinstall the Veeam server and import configuration backup, and you will be up and running with your recoveries in no time. Configuration backups should of course be pulled into the air-gapped repository along with normal backups, so there's no added labor in protecting that.

P.S. In case today's post in this thread seem a bit out of context, we're discussing the fully automated air-gap solution proposed in today's weekly Veeam forums digest.

[DerOest]

@DerOest "normal repository" approach will never work as it requires that the backup server is able to connect to the repository server remotely. So, it will be impossible for you to fully isolate the repository from remote access by dropping ALL incoming connections, which as you remember is the key of the proposed solution. But in any case, I want to clarify that the biggest part of the Insider Protection feature value proposition is that your data resides with the 3rd party and cannot be controlled by your own IT staff. Thus the name of the feature, Insider Protection.

Hello Gostev,

thanks for your reply, but i guess you misunderstood my intentions. Instead of "hacking together" a protection á la power-off-power-on, it would be easier if Veeam made it available as a configuration option for a repository.
We can't use cloud for multiple reasons, but the "Insider Protection" is what we'd want, just on-prem...

Firewall problem: that's why I mentioned "lock down the Firewall [except Veeam Ports]" - this way it can receive data. Now, the Repository is responsible for "don't delete backups newer than x days" - server locked down, no way to delete backups - cool!
Insider Protection: I wrote "Repository with an option like "Delete Protected"" - because you can't really safeguard against a physical "Insider" - he could just destroy the physical server - be it powered on or not!
But a "locked down server with delete-protection" would offer you the protection against hackers who would be considered "insiders" once they gained admin-logins.

[Gostev]

Veeam uses quite a few ports, so this leaves a huge opportunity for a hacker to execute the attack through the exposed non-Veeam components. Not to mention possible vulnerabilities in Veeam components themselves.

I was looking to provide for a solution that provide 100% solid protection, something I can stand behind for - not just another set of "half-measures" so to speak, something with lots of but-what-ifs. And even more importantly, I wanted something everyone can use today and with the shipping Veeam code.

[DerOest]

Ok, "with shipping code" is the argument i didn't grasp!

But thinking into the future, i trust more in Veeam to deliver a 99,9% solution than admins of different skill-levels across the world implementing selfbrew-security!

Maybe put your own idea of "enable customers to secure their backups on-prem" on the feature request list - I trust in you that you already did that

Meanwhile, trusty old tapes - right now i've got to retreive the tapes from our other facility to put them into the safe...

[markmu]

I was thinking about upgrading our off-site backups to off-site / off-line backups.

Will isolating the Veeam server, data store and an AD server behind an internal firewall prevent access to the backed up data? Would it be good enough? Obviously, access would be really bolted down, i.e. encryption, no routing advertisements, time, address and protocol restrictions, VPN between AD servers etc.

How much ancillary traffic is required for Veeam to communicate with VMware? Can it all be initiated from behind this firewall? Will this ancillary traffic defeat my off-line plans?

[Mike Resseler]

Hey markmu,
First: Welcome to the forums
Second: Yes, it is possible through Firewalls etc. All the ports are listed here: https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Now to answer your question: Will isolating the Veeam server, data store and an AD server behind an internal firewall prevent access... It will certainly secure it much more and will block certain types of ransomware, but not all. And it also doesn't block it from "internal user mistakes".

As you can read in this thread, and in other threads, securing your backups is not a matter of inserting one layer of defense, it is a matter of adding multiple layers. Offline backups is still the only way we know where you have 100% guarantee that it can't be encrypted. While other layers give you a very high percentage of security... There is always this little change...

For example, many people start to think about putting their backups in the cloud and see it as offline / air-gapped. It is close, but once a "bad guy" gets access to your cloud account... Well

[mr-tin]

Could disabling windows admin shares on the Veeam Backup & Replication server help to reduce exposure or would this cause issues/interfere with Veeam backup operations

[jasonede]

Hey markmu,
First: Welcome to the forums
Second: Yes, it is possible through Firewalls etc. All the ports are listed here: https://helpcenter.veeam.com/docs/backu ... tml?ver=95

Now to answer your question: Will isolating the Veeam server, data store and an AD server behind an internal firewall prevent access... It will certainly secure it much more and will block certain types of ransomware, but not all. And it also doesn't block it from "internal user mistakes".

As you can read in this thread, and in other threads, securing your backups is not a matter of inserting one layer of defense, it is a matter of adding multiple layers. Offline backups is still the only way we know where you have 100% guarantee that it can't be encrypted. While other layers give you a very high percentage of security... There is always this little change...

For example, many people start to think about putting their backups in the cloud and see it as offline / air-gapped. It is close, but once a "bad guy" gets access to your cloud account... Well

If a bad guy gets access to your backup server though they'll spot that a backup/replication happens at the same time every day and then that gives them a window to try and attack the "airgapped" backup server as it will be online for that replication. It's a small window, but still could be enough to exploit a zero day attack.