Discussions specific to the VMware vSphere hypervisor
hannisch
Enthusiast
Posts: 34
Liked: 5 times
Joined: Dec 15, 2011 8:14 pm
Full Name: Sven Hannisch
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by hannisch » Nov 12, 2018 8:43 am

Hi,

I good choice if the backup size is max. 4TB is using RDX tapes. They are recognized like normal usb drives. I use the Freeware tool freeeject to eject the tape after backup.
There is one disadvantage regarding the restore points, so I have a feature request. In the restore wizzard, when using rdx or rotating usb drives, only the last restorepoint is shown. Isn´t it possible to show all restorepoint, even the offline?

b/r

Sven

Elemer.gazda
Influencer
Posts: 10
Liked: never
Joined: Feb 14, 2018 12:01 pm
Full Name: Elemer Gazda
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Elemer.gazda » Nov 12, 2018 10:32 am

I like the idea of a server which is not allowing any traffic in, and can only be managed from the local console, and dismounts the disks and disables the network interface as well.
One feature request for this:
It should interact with the original Veeam server somehow so that from the veeam backup server console (Or event Log, or just log files) you are able to see if the "Air-Gapped" backup had finished successfully or not.
Otherwise it would be a real pain to have to physically check the Air-Gap server and see if the files have been copied over or not. Major issue if you remotely manage tens of Veeam backup servers.

mma
Service Provider
Posts: 89
Liked: 13 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by mma » Nov 12, 2018 11:04 am 1 person likes this post

If you do all the effort to have a physical server, do some scripting for file copy, firewall restrictions, monitoring all the stuff....
Why you don't just buy a LTO library / drive? Eject the drive after the job is finished and you are all good.

pkelly_sts
Expert
Posts: 568
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by pkelly_sts » Nov 12, 2018 12:28 pm

Because re-inserting the tape/drive requires manual (human) intervention so can be missed.

I'm glad I came to this topic as I was discussing similar scenarios with a colleague and we had initially settled on WORM as being most attractive but I hadn't realised that Veeam doesn't actually support it yet - I just assumed it would work.

Our solution uses two libraries, different sites with tapes just auto-rotating but I did point out to the business that a malicious internal user could theoretically format/erase all tapes before doing the same to production storage but the risk was deemed low enough at the time to not need to mitigate.

However times are changing & the risk is up for discussion again...

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Gostev » Nov 12, 2018 1:19 pm 1 person likes this post

pkelly_sts wrote:
Nov 12, 2018 12:28 pm
I'm glad I came to this topic as I was discussing similar scenarios with a colleague and we had initially settled on WORM as being most attractive but I hadn't realised that Veeam doesn't actually support it yet - I just assumed it would work.
Actually, you can go ahead with the procurement as WORM tape support is a part of Update 4.
Also, that other thing in your signature is there as well :wink: so, start thinking about the new one!

pkelly_sts
Expert
Posts: 568
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by pkelly_sts » Nov 12, 2018 1:41 pm 1 person likes this post

Gostev wrote:
Nov 12, 2018 1:19 pm
Actually, you can go ahead with the procurement as WORM tape support is a part of Update 4.
Also, that other thing in your signature is there as well :wink: so, start thinking about the new one!
YAY! Double-celebration! :lol: :D :lol:

Mikejden
Veeam Software
Posts: 4
Liked: never
Joined: Aug 06, 2018 2:08 am
Full Name: Mike Dennehy
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Mikejden » Nov 12, 2018 3:10 pm

Anton,
Your proposal makes sense and there is precedence for it. Companies with dependencies on OT (think SCADA systems) use one-way communications to limit access to them. The key piece in all of this is that the OT environments, in addition to pulling patches where needed or any other information, need to push files to the monitoring environments so that they can properly be watched in case of any issues. I would recommend you add that piece to your proposal. The isolated environment, in addition to pulling backup snapshots, needs to send log information out to the designated Syslog (Linux) or Centralized Windows event servers to allow for proper monitoring of the environments.

Frosty
Expert
Posts: 164
Liked: 35 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Frosty » Nov 12, 2018 9:42 pm

Finding this discussion fascinating. I oversee a fairly small environment (50 VMs and about 4TB of daily backup data). 100% virtual, apart from a physical Veeam backup server. We back up every VM every night to that server, keeping the last week's worth of backups onsite for quick restore if required. But we also copy everything every night to sets of removable HDDs and have them stored offsite in a secure facility. Touch wood, that has worked well as a strategy so far. We perform regular DR test restores of that data from the HDDs.

About 12 months back I considered also having an offsite over-the-network backup. I intended setting that remote location up so that it could copy data in from our backup server, but would have prevented any ability to connect to that facility via the network. So connected, but blocked via firewalls. We ended up not going ahead for two reasons: (1) cost comparison with portable HDDs was poor; and (2) if someone gets control of your firewall, then they could remove the blocks and still gain access, so you'd still need the portable HDDs air-gapped backups anyway.

mma
Service Provider
Posts: 89
Liked: 13 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by mma » Nov 13, 2018 7:30 am

hannisch wrote:
Nov 12, 2018 8:43 am
[snip] when using rdx or rotating usb drives, only the last restorepoint is shown. Isn´t it possible to show all restorepoint, even the offline?
Wow, I'm pretty sure there was an answer to this from Gostev, on which I made a comment myself.
The post from Gostev is gone, mine too.
Feel free to delete your own post, but what happened to mine?

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Gostev » Nov 13, 2018 12:42 pm

@Marcel I've cleaned up some off-topic, otherwise the discussion has really started to deviate from the main topic. And I sort of started it myself with my original response, so I apologize for that. So, since there's already an existing thread that you linked, I've deleted our entire exchange.

@Frosty actually my proposed solution solves issue (2), did you have a chance to read last week's forum digest?

davidwatts71
Novice
Posts: 9
Liked: 3 times
Joined: Oct 30, 2017 8:05 am
Full Name: David Alexander Watts
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by davidwatts71 » Nov 14, 2018 2:34 pm

instead of powering off the machine we could just power off the network switch using a NETIO 4 smart power socket. I think a network switch should not have a problems with a sudden power off and on but I might be wrong. You could then use powershell\lua scripts to power on the switch, copy the data and power off the switch again. The Netio 4 also has built-in scheduled task to switch the power on so that would be the same as using the BIOS to power on the machine.

Frosty
Expert
Posts: 164
Liked: 35 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Frosty » Nov 15, 2018 12:28 am 1 person likes this post

@Gostev yes, I read all your digest emails; they're an excellent source of information and they're required reading for me.
I have a high level of paranoia. So *any* form of network connectivity is a risk that I am unwilling to absorb. I did like the proposed solution, to summarise: pull the backups from another network-connected location and lock all that down very tightly (e.g. console access only, etc). It is a very similar idea to one that I considered implementing a year ago.
But if the data is on a connected network,anywhere, then to some degree it will remain vulnerable. Firewall rules can be changed if the firewall is hacked. Console-only access can be changed if the environment around that console-only server is breached. Paranoid? Sure, I'll wear that with distinction.
We're not a high-value target (I work for a not-for-profit). Touch wood, we've never been breached AFAIK and regardless of all the over-the-network backup solutions, I will *still* always want an air-gapped backup as my "last resort" get-out-of-jail-free card. I'm lucky that our environment is small enough to be copied onto portable HDDs. Big environments would not have that luxury.

xudaiqing
Influencer
Posts: 10
Liked: 1 time
Joined: Apr 14, 2017 5:25 pm
Full Name: xudaiqing
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by xudaiqing » Nov 16, 2018 4:03 am 1 person likes this post

My current solution is have is setup a isolated hyper-v server (no network connection to root partition only local console) and run a vm on it as backup repository.
Then create daily checkpoint/snapshot for the it. When use with refs it has good enough performance for our size (around 4TB).
As long as the isolation between VMs isn't breached, it should be safe.

Mengisman
Lurker
Posts: 1
Liked: never
Joined: May 25, 2018 1:14 pm
Full Name: Chris Menge
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Mengisman » Nov 16, 2018 5:26 pm

wla wrote:
Nov 12, 2018 8:37 am
But, when the primary backup will be compromised (encryption malware), then the secondary repository is compromised as well.
I think this is an excellent point that surely needs consideration and my solution will be to ensure that I have enough storage for at least 2 or ideally, more full backup copies and make sure I am not overwriting the last most current backup with the new incoming backup.

caztor
Service Provider
Posts: 1
Liked: 2 times
Joined: Mar 16, 2015 10:09 pm
Full Name: Theis Andersen
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by caztor » Nov 19, 2018 10:17 am 2 people like this post

There are so many good points being made on this topic - For most of us, I think we would like an online solution that would still satisfy the requirement of being air-gapped. That way we could still automate things (without investing in a tape autoloader), get reports and status back from the device.

We have had this discussion many times internally and with customers, but I think the best solution would require some extra features being added by the software vendor to really make it effective.

The obvious option going forward is utilising the "pull" method - we need to make the "air-gapped" as impregnable as possible, so we would need it not to respond to anything and preferably only accessible using a console. But we need the to know what to copy and be intelligent about it - this is where we need something from the software vendor, maybe an agent of sorts. I'm picturing something certificate based, that could talk securely with the veeambr repository, get information on what to pull and also being able to check integrity before storing and respond to abnormalities like a high percentage of changes or missing VM's etc.

I'm thinking that we can already build something like this using the API to get the information we need, but I haven't begun exploring that yet.

These are just my 5 cents - I'm curious if someone has already built something like this?

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 6 guests