Discussions specific to the VMware vSphere hypervisor
hyvokar
Expert
Posts: 343
Liked: 21 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Reply to Gostev, Air-gapped backup

Post by hyvokar » Nov 05, 2018 9:03 am

Just as I was thinking about this, I had to go to my computer BIOS settings to make some changes, and spotted one interesting functionality there that could be the answer – the ability to power on the computer at the scheduled time (time and day of week). I don't know how common is this functionality is between motherboard manufacturers, but I assume it should be common. So, my idea here is to have the dedicated "air-gapped repository" server with JBOD that serves as the target for, say, weekly backup copy jobs of most important backups. You will use this BIOS setting to power the server on right before the weekly backup copy job is scheduled to start, and then power it off with the post-job script command. This way, it will remain powered off for the entire week with absolutely no way to control it remotely. Could this be the cheapest "fire-and-forget" air-gap solution for everyone?
Hi,
Couldn't find how to reply Gostev's weekly email, so I'll just post here.
Setting up computer to start at scheduled time is ancient tech and widely used. The main problem with a weekly backup is, that I'm sure there are ton of users who cannot lose a week's worth of data. At least we cannot. Then again, running this backup daily, would expose the remote machine for attacker (depends on how long your backup takes), but still, it's better than nothing

My feature request for VBR would be 'write protected' backup, so you could not delete the backups from backup server console until set period of time.

How I've implemented our 'air-gap' at the moment, is far from ideal. I run a backup copy job on site1 and target it to site2 server1. Site2 server1 is in different domain (would be way cool if the target computer would not need to ba a part of a domain....) and has a completely different set of credentials than anything on site1. Server2 on site2 is running a script which copies the backup daily from server1. First of all, this is VERY slow (copy takes ~15hours, since it has to copy full vkbs every time), and secondly it wastes several terabytes of space.

'Write protected' -backup would greatly improve the situation.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

Rick.Vanover
Veeam Software
Posts: 554
Liked: 118 times
Joined: Nov 30, 2010 3:19 pm
Full Name: Rick Vanover
Location: Columbus, Ohio USA
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Rick.Vanover » Nov 05, 2018 2:35 pm 1 person likes this post

Cheers, hyvokar - one thing I'll note is that WORM tape media support is coming in Update 4, so that is an option also.

hyvokar
Expert
Posts: 343
Liked: 21 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by hyvokar » Nov 06, 2018 12:50 pm

We have just given up using tapes, so that's why I'm looking for another air-gapped solution :-)
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

Rick.Vanover
Veeam Software
Posts: 554
Liked: 118 times
Joined: Nov 30, 2010 3:19 pm
Full Name: Rick Vanover
Location: Columbus, Ohio USA
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Rick.Vanover » Nov 06, 2018 9:25 pm

I need an enterprise equivalent, but at home, I use a managed power switch. It is on only for specified times.

Maybe PDUs can do this for switches or drive systems.

hyvokar
Expert
Posts: 343
Liked: 21 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by hyvokar » Nov 07, 2018 7:39 am

Still, this is far from ideal. Let's say your VBR server is compromised. A smart attacker would then wait for "air-gapped" system to come online and do his bad magic.
As we see, people have come up with many different solutions for "air-gap", some better, some worse than other. I think there's a real need for Veeam to implement this.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

Rick.Vanover
Veeam Software
Posts: 554
Liked: 118 times
Joined: Nov 30, 2010 3:19 pm
Full Name: Rick Vanover
Location: Columbus, Ohio USA
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Rick.Vanover » Nov 07, 2018 6:28 pm

I would rather see things like shares or LUNs be able to be taken offline via a Veeam call when writes are done - and brought online when writes are soon to come.

Gostev
SVP, Product Management
Posts: 23654
Liked: 3127 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Gostev » Nov 08, 2018 6:04 pm

hyvokar wrote:
Nov 05, 2018 9:03 am
My feature request for VBR would be 'write protected' backup, so you could not delete the backups from backup server console until set period of time.
Unfortunately, such feature would be utterly useless, because backup files can still be easily deleted using standard OS tools. In fact, this is how it's usually done anyway - most hackers don't even bother starting the backup console, when it's way faster to just run rm /rf or format d:

And, needless to say, cryptolockers don't bother going through the backup console to do their thing either ;) so, with this feature, not only you will still get your backups encrypted and unusable, but you also won't be able to delete them through the backup console to free up disk space for the new ones! :D

Gostev
SVP, Product Management
Posts: 23654
Liked: 3127 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Gostev » Nov 08, 2018 6:17 pm

Rick.Vanover wrote:
Nov 07, 2018 6:28 pm
I would rather see things like shares or LUNs be able to be taken offline via a Veeam call when writes are done - and brought online when writes are soon to come.
But basically, that's exactly what I suggested in the digest then? Importantly, it must not be Veeam bringing them online, because it means there's an API that "bad guys" will be able to use as well - which is why my solution uses "self-contained power" on approach.

I do agree it's not a bulletproof solution, since if the fish is big enough, smart hacker will monitor the environment for weeks before executing the attack. But time is money, so they won't bother wasting it on smaller environments which are unlikely to pay big buck.

Anyway, thanks to all the feedback I think I found a way better solution that should protect against any attack except from insiders with physical access! And it's even more secure than storage snapshots, albeit less "convenient". Just need to polish and confirm a few more things... stay tuned for the next digest ;)

hyvokar
Expert
Posts: 343
Liked: 21 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by hyvokar » Nov 09, 2018 10:34 am

Gostev wrote:
Nov 08, 2018 6:04 pm
Unfortunately, such feature would be utterly useless, because backup files can still be easily deleted using standard OS tools. In fact, this is how it's usually done anyway - most hackers don't even bother starting the backup console, when it's way faster to just run rm /rf or format d:

And, needless to say, cryptolockers don't bother going through the backup console to do their thing either ;) so, with this feature, not only you will still get your backups encrypted and unusable, but you also won't be able to delete them through the backup console to free up disk space for the new ones! :D
Hi,

Please re-read my post.
You cannot use standard OS tools to delete backup files from remote system, because you don't know the credentials (I assume they cannot be stealed from the VBR database), and firewall is blocking all but VBR communications.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

Gostev
SVP, Product Management
Posts: 23654
Liked: 3127 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Gostev » Nov 09, 2018 1:16 pm

Well, that's just a wrong assumption then. You cannot protect stored credentials which are actively used by apps - be it Veeam Backup & Replication, Google Chrome or even Windows own stored credentials. Because if the application itself is somehow able to retrieve and decrypt them, then any other code running on the same machine with local system privileges will be able to do so too. Not to mention that when the application uses those credentials, they stay encrypted in memory and can potentially be intercepted there, even though this is usually the hardest way to get a hold of them.

mma
Service Provider
Posts: 89
Liked: 13 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by mma » Nov 09, 2018 1:28 pm

@Gostev

Just a quote from your diggest: "This way, it will remain powered off for the entire week with absolutely no way to control it remotely."
If you already are in your BIOS settings, turn off WOL! Otherwise someone is able to start your computer with nothing else than a bit of magic...

Gostev
SVP, Product Management
Posts: 23654
Liked: 3127 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Gostev » Nov 09, 2018 2:08 pm

Yup, quite literally "magic packet" ;) however, I thought WOL does not work on powered off computers, and can only wake computers in sleeping states (S1 through S4)?

skrause
Expert
Posts: 368
Liked: 66 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by skrause » Nov 09, 2018 3:28 pm

WoL can work on powered off machines. It depends upon how you configure the BIOS/UEFI on the targets.

We used it to turn on public use machines at the library I worked at every morning 30 minutes before open as they shut down every night at closing time.
Steve Krause
Veeam Certified Architect

MichelZ
Novice
Posts: 3
Liked: 2 times
Joined: Jul 28, 2017 6:23 am
Full Name: Michel Zehnder
Location: Zurich, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by MichelZ » Nov 12, 2018 6:25 am

I think the most important and (IMHO) best solution gets overlooked here. Online disk storage with WORM support (e.g. NetApp, many others), Snaplock and the likes.
If the solutions are properly designed, then you can't delete any data from them, even if you wanted to (as admin). This would require some cooperation with veeam though (to set the retention period on the files), is this something you support? (Haven't looked into veeam in a while...)

wla
Novice
Posts: 6
Liked: never
Joined: Feb 17, 2011 11:23 am
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by wla » Nov 12, 2018 8:37 am

the only way to populate one with backup copies will be by pulling backups FROM your primary backup repository
But, when the primary backup will be compromised (encryption malware), then the secondary repository is compromised as well.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 7 guests