Availability for the Always-On Enterprise
jandrewartha
Influencer
Posts: 14
Liked: 1 time
Joined: Feb 13, 2017 1:49 am
Contact:

Offline airgap backup

Post by jandrewartha » Nov 05, 2018 4:49 am

From an old thread:
jandrewartha wrote:
Feb 07, 2018 5:48 am
It'd be nice to having something that pulls the image, we're pondering ways to get an offline copy of our long-term (12mo+, 26TB atm) backup copy repository that's updated once a week, both for cryptolocker recovery and in case the whole server dies in a fire for some reason, we don't lose that backup history.

Right now I'm thinking a two stage backup, Veeam agent backup to our old netapp, then a small server that boots up automatically (BIOS/UEFI control?), copies the image over, emails success then shuts down again, but I'm sure this could be simplified.
Bringing this up again after Gostev's weekly post. I think the big barriers are how does Veeam handle a repository that is offline some of the time, and that you really want a pull model for the airgapped backup, preferably only done once it's confirmed the backup is recoverable (ie not encrypted by a cryptolocker), not an automatic push. It needs to be pull so that only the airgapped machine has the credentials to write to itself, otherwise again an attacker who gains control of Veeam can just wait for it to come online again and overwrite it. And it also allows for the user to just power it on manually when they want to make an offline copy, if they don't have a BIOS with power-on controls (although they are fairly common in my experience).

And finally the specific question from that thread, how do I get a duplicate copy of an ReFS repository will all the history in it without reinflating everything?

staun
Influencer
Posts: 12
Liked: 2 times
Joined: Jun 30, 2017 8:48 am
Full Name: Jørgen Staun
Contact:

Re: Offline airgap backup

Post by staun » Nov 05, 2018 8:54 am

Will follow this topic as I have the exact same concerns and looking for the right solution for an offline backupCopy. My current plan is a local server, but as you mention - Our main Veeam server has access to this server...

Nils
Influencer
Posts: 11
Liked: 2 times
Joined: Jun 18, 2013 8:12 am
Full Name: Nils Petersen
Contact:

Pseudo air-gapping

Post by Nils » Nov 05, 2018 11:07 am

With a network share, you can also hide backup folders from the share.

Depending on whether the rotation uses physical drives or just folders, you can mount and unmount the drives/folders into the repository folder using scheduled diskpart or makelink (don't use symbolic links as those are resolved by the client). That way, unused and unmounted/unlinked folders are simply not accessible outside of their backup window. Veeam requires the " This repository is backed by rotating drives" to cope.

You should use Veeam's Job scripts option to create and delete status files for "job xy is running", so you don't dismount the folder that a job is currently running on.

Also, you might want to turn off the default admin shares and explicitly not grant your backup (or domain) admins local admin privileges on the repository, so an attacker can't leverage here.

xudaiqing
Novice
Posts: 9
Liked: 1 time
Joined: Apr 14, 2017 5:25 pm
Full Name: xudaiqing
Contact:

Re: Offline airgap backup

Post by xudaiqing » Nov 05, 2018 12:09 pm

I think storage snapshot of backup repository with a air-gapped management interface is better and easier.

olavl
Novice
Posts: 4
Liked: 1 time
Joined: Jan 23, 2018 8:21 am
Full Name: Olav Langeland
Contact:

Re: Offline airgap backup

Post by olavl » Nov 05, 2018 1:18 pm

For the paranoid you need offline and offgrid.

Offline: Protects against virus/ransomware attack and protects against accidental delete. The files are not directly accessible on the network so an infection or accident cant reach the backupfiles.
- Non AD-member and locked up Veeam
- Put files on SAN/NAS and snapshot
- Azure/AWS storage (soon I hope)

Offgrid: Protects against hackers intentionally accessing and deleting backup files. Storage snapshot is fine for offline, but if the case is a hacker on your network with admin access (always assume the worst), the rogue player has access to both Veeam and storage and can delete everything.

Offgrid is the problem and I havent found a good solution. How can I put the backup files so they are accessible to me in case of restore, but impossible for me to delete them?
- LTO8 tape in safe?
- Some sort of snapshot in Azure or AWS?
- A VTL that requires console access to delete?

wleister
Lurker
Posts: 1
Liked: 1 time
Joined: Nov 05, 2018 2:43 pm
Full Name: Wayne Leister
Contact:

Re: Offline airgap backup

Post by wleister » Nov 05, 2018 3:09 pm 1 person likes this post

Not exactly offline, but we recommend HP storeonce as a backup target. It integrates with Veeam using a catalyst share. We have never had any customers have the storeonce get encrypted with ransomware. You can also have storeonce replicate to a storeonce at another location or the cloud if you want extra protection.

ChrisSnell
Technology Partner
Posts: 123
Liked: 16 times
Joined: Feb 28, 2011 5:20 pm
Full Name: Chris Snell
Contact:

Re: Offline airgap backup

Post by ChrisSnell » Nov 06, 2018 6:02 pm

ExaGrid also uses the Veeam Data Mover. The backup share is therefore neither CIFS or NFS, but a Veeam share. The only way to access it is via the Veeam server using a specific username and password. It's a virtual air gap. The Data Mover also increases performance :)

Gostev
Veeam Software
Posts: 23189
Liked: 2955 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offline airgap backup

Post by Gostev » Nov 08, 2018 5:43 pm 1 person likes this post

@Chris, in virtually all successful attacks we saw to date, the hacker was taking over the backup server and thus had access to all stored credentials. So unless by "virtual" air-gap you mean "non-existent" ;) then I would disagree this provides any air-gap whatsoever.

The common confusion seems to be when people think cryptolocker attacks on their Veeam backups are about some super smart Veeam-aware cryptolockers looking around for accessible repositories and file shares holding Veeam backups. In reality though, we've NEVER seen a successful cryptolocker attack that did not involve a hacker penetrating the environment to manually find and delete (often with zeroing out disk) the backups before manually sicking a cryptolocker on the most critical production data they identified. With the initial malware being used solely to provide a hacker with the way into the network perimeter.

DerOest
Enthusiast
Posts: 32
Liked: 4 times
Joined: Oct 30, 2015 10:10 am
Contact:

Re: Offline airgap backup

Post by DerOest » Nov 12, 2018 8:24 am 1 person likes this post

Hello Gostev,

as always, i loved reading your Newsletter this morning!

You casually, non-saleslike mentioned "Veeam Cloud Connect with Insider Protection". Wouldn't that be the best option, to bring something like that on-prem?

Something roughly like
  • User installs local repository operating system (Windows or supported Linux distribution - make sure to disable ILOM on that hardware server, not AD-joined)
  • In Veeam, add it as Repository with an option like "Delete Pretected" + select how many days/weeks backups should be delete-pretected
  • Show a complex password, protect the configuration of this repository with it
  • Run a script on the Delete-protected Repository - lock down the Firewall [except Veeam Ports], etc.
  • Disable all local Users except one with that password
That way, you could just use it like a normal repository, with the benefit of "it's all managed by Veeam".

And because only all incoming traffic [except that Veeam Rules is denied, WSUS/linux updates, monitoring (push-alerting) etc. should still work!

jasonede
Service Provider
Posts: 17
Liked: 2 times
Joined: Jan 04, 2018 4:51 pm
Contact:

Re: Offline airgap backup

Post by jasonede » Nov 12, 2018 8:49 am

I like the idea of a server which can only be managed from the console and pulls the backups. As far as I can see if the backup/replication server is online and remote manageable then it's vulnerable to someone with enough time and motive (and lots of money is a pretty damn good motive!). If the server is shutdown when it's not needed then it's also saving on precious electricity if it's in a data center as quite a lot of them charge by the amp. ACPI wakeup is trivial and easily scriptable (just a matter of updating /sys/class/rtc/rtc0/wakealarm or simular on linux systems).
I like the idea of only pulling the latest .vbk files and avoiding the older ones, in case they get corrupted. It is possible that a sneaky hacker could start by messing with old backups and then wait for that to pollute the chain and the older good ones to drop off the end as you'll not be able to keep all backups forever, but that would require someone to be inside your systems unnoticed for potentially months on end.
It would be nice to have a veeam script/add-on that could do this. Also I'm guessing this sort of method wouldn't be compatible with all backup strategies such as reverse incremental where a synthetic full is always generated.

Gostev
Veeam Software
Posts: 23189
Liked: 2955 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offline airgap backup

Post by Gostev » Nov 12, 2018 1:06 pm

@DerOest "normal repository" approach will never work as it requires that the backup server is able to connect to the repository server remotely. So, it will be impossible for you to fully isolate the repository from remote access by dropping ALL incoming connections, which as you remember is the key of the proposed solution. But in any case, I want to clarify that the biggest part of the Insider Protection feature value proposition is that your data resides with the 3rd party and cannot be controlled by your own IT staff. Thus the name of the feature, Insider Protection.

@Jasonede you're correct. In theory you can make it work with all backup strategies (again, with things like rdiff-backup) but explained well why it is a bad idea. This will work best with any backup modes that allow you to copy new backup files only - so any backup mode with periodic fulls (as with these backup modes, backup files are never modified once they are created, so there's no point of copying them more than once).

dpeach1
Lurker
Posts: 2
Liked: never
Joined: Nov 18, 2011 3:14 pm
Contact:

what about the Veeam Server

Post by dpeach1 » Nov 12, 2018 1:39 pm

What about the integrity of the Veeam Server? Seems like a chicken egg problem. if the Veeam server is crypto lockered, along with the agent based backups, what do you do? The Veeam backup on tape is useless if you cannot recover the server to access the tape library. Do we need an air gapped copy of the Veeam server backup with recovery disk files along with the Veeam configuration backup? Put it on a thumbdrive? This becomes very labor intensive.

Gostev
Veeam Software
Posts: 23189
Liked: 2955 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offline airgap backup

Post by Gostev » Nov 12, 2018 1:45 pm

Just reinstall the Veeam server and import configuration backup, and you will be up and running with your recoveries in no time. Configuration backups should of course be pulled into the air-gapped repository along with normal backups, so there's no added labor in protecting that.

P.S. In case today's post in this thread seem a bit out of context, we're discussing the fully automated air-gap solution proposed in today's weekly Veeam forums digest.

DerOest
Enthusiast
Posts: 32
Liked: 4 times
Joined: Oct 30, 2015 10:10 am
Contact:

Re: Offline airgap backup

Post by DerOest » Nov 13, 2018 11:47 am 1 person likes this post

Gostev wrote:
Nov 12, 2018 1:06 pm
@DerOest "normal repository" approach will never work as it requires that the backup server is able to connect to the repository server remotely. So, it will be impossible for you to fully isolate the repository from remote access by dropping ALL incoming connections, which as you remember is the key of the proposed solution. But in any case, I want to clarify that the biggest part of the Insider Protection feature value proposition is that your data resides with the 3rd party and cannot be controlled by your own IT staff. Thus the name of the feature, Insider Protection.
Hello Gostev,

thanks for your reply, but i guess you misunderstood my intentions. Instead of "hacking together" a protection á la power-off-power-on, it would be easier if Veeam made it available as a configuration option for a repository.
We can't use cloud for multiple reasons, but the "Insider Protection" is what we'd want, just on-prem...

Firewall problem: that's why I mentioned "lock down the Firewall [except Veeam Ports]" - this way it can receive data. Now, the Repository is responsible for "don't delete backups newer than x days" - server locked down, no way to delete backups - cool!
Insider Protection: I wrote "Repository with an option like "Delete Protected"" - because you can't really safeguard against a physical "Insider" - he could just destroy the physical server - be it powered on or not!
But a "locked down server with delete-protection" would offer you the protection against hackers who would be considered "insiders" once they gained admin-logins.

Gostev
Veeam Software
Posts: 23189
Liked: 2955 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offline airgap backup

Post by Gostev » Nov 13, 2018 12:49 pm

Veeam uses quite a few ports, so this leaves a huge opportunity for a hacker to execute the attack through the exposed non-Veeam components. Not to mention possible vulnerabilities in Veeam components themselves.

I was looking to provide for a solution that provide 100% solid protection, something I can stand behind for - not just another set of "half-measures" so to speak, something with lots of but-what-ifs. And even more importantly, I wanted something everyone can use today and with the shipping Veeam code.

Post Reply

Who is online

Users browsing this forum: anthony_b, christian.kotze, Exabot [Bot] and 28 guests