Discussions specific to the VMware vSphere hypervisor
pkelly_sts
Expert
Posts: 576
Liked: 63 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by pkelly_sts » Nov 20, 2018 9:29 am 1 person likes this post

Hmm, is this an enhancement opportunity for Veeam? Create a new kind of repository/agent combination - An agent that can ONLY "pull" backups to itself, and a repository-type that can ONLY be "pulled to" by such pull agents?

Obviously would be dependent on that machine being as utterly locked-down as possible in every other way...

yasuda
Enthusiast
Posts: 63
Liked: 10 times
Joined: May 15, 2014 3:29 pm
Full Name: Peter Yasuda
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by yasuda » Nov 30, 2018 9:17 pm

Has anyone looked into Wasabi immutable cloud storage ?
https://wasabi.com/blog/data-immutability-done-right/

Perhaps some Veeam Managed Cloud Providers will offer a service with immutable storage - maybe using Wasabi on the back end. It would be nice to not have to upload full backups.

hyvokar
Expert
Posts: 362
Liked: 23 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by hyvokar » Dec 04, 2018 11:45 am

How about using cloud gateway technology for internal use and add an option to protect the remote backups from deletion? Or would this eat too much cloud providers' markets?
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

Gostev
SVP, Product Management
Posts: 25136
Liked: 3689 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Gostev » Dec 04, 2018 8:44 pm

Not really a concern, for example we have had Cloud Connect available to regular customers for a few years now > Veeam Cloud Connect for the Enterprise

hyvokar
Expert
Posts: 362
Liked: 23 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by hyvokar » Dec 05, 2018 9:47 am

Cool! Didnt know that option existed for enterprises. That could possibly make my life a bit easier... :-)
Any change to write protect the backups on the cloud gateway server?
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

veremin
Product Manager
Posts: 17070
Liked: 1475 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by veremin » Dec 05, 2018 10:50 am 1 person likes this post

Can you elaborate what change you're talking? If you're asking for Insider Protection feature, then, it is available in Veeam Cloud Connect for the Enteprise, correct. Thanks!

jihering
Lurker
Posts: 1
Liked: never
Joined: Dec 20, 2018 7:47 pm
Contact:

[MERGED] Air gapped copies

Post by jihering » Dec 20, 2018 7:58 pm

I'm sure this has already been addressed. If so please point me in the right direction.

New VEEAM Backup and Replication user (from an OLD version of Backup Exec tape system). We are a small school district. We recently had new VM Servers, Nimble SAN and QNAP backup storage installed. migrating old VM's to new servers and storage. Running daily VM backups with weekly synthetic fulls. Everything set up and configured by an outside vendor. Been struggling getting through the 1000+ page user manual. I'm a little freaked out by what I've been reading about ransomware locking servers and deleting VEEAM backups. Is it possible to occasionally just copy off the data from a full backup to an external USB storage device for offline/offsite storage? I can see the backup files on the VEEAM server's storage drive (the QNAP).

Again, I apologize if this is a repeat. I couldn't find exactly what I was looking for searching through the forums.

wishr
Veeam Software
Posts: 1334
Liked: 133 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Air gapped copies

Post by wishr » Dec 21, 2018 9:55 am

Hi Jihering,

Welcome to Veeam Community Forums and thank you for posting your query.

Sure, you can set up a Backup Copy job to your USB drive, once completed remove the drive from the machine.

Please let us know if you have any additional questions on that. Thank you.

P.S. I've moved your post to an existing thread - definitely recommend taking a look.

bdufour
Expert
Posts: 204
Liked: 40 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by bdufour » Dec 21, 2018 5:30 pm

ive been thinking about this, we never had backup appliances tied to AD auth, but obviously veeam is installed on a production server, which is tied to AD and i would like to keep it that way bc of kerberos auth, management, and compliance. i think if veeam could enable MFA on the console that would protect us in the event that a privileged account was compromised, which could log into veeam and delete backups. we have alerting set up whenever an admin account logs into a server, but it may be too late at that point. the biggest thing is, as an admin, i can delete backups from the console - that is what scares me the most. if a cryptolocker were to encrypt the windows server veeam is hosted on, that wouldnt be the end of the world bc the appliance the backups are stored on isnt tied to AD and if MFA was enabled on the console for veeam that would protect the backups at that level.
since veeam isnt providing MFA at this moment, we may look at some other MFA integrated option for admin accounts.
open to ideas as well guys, this has been a big topic, since ive brought it up with the dept.

hyvokar
Expert
Posts: 362
Liked: 23 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by hyvokar » Feb 05, 2019 3:53 pm

v.Eremin wrote:
Dec 05, 2018 10:50 am
Can you elaborate what change you're talking? If you're asking for Insider Protection feature, then, it is available in Veeam Cloud Connect for the Enteprise, correct. Thanks!
Thanks! Exactly what I was looking for.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

hyvokar
Expert
Posts: 362
Liked: 23 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by hyvokar » Feb 12, 2019 1:14 pm

Gostev wrote:
Dec 04, 2018 8:44 pm
Not really a concern, for example we have had Cloud Connect available to regular customers for a few years now > Veeam Cloud Connect for the Enterprise
Ok, called your sales today to find out what is the difference between std, ent and ent+ edition and was told, that cloud connect is only avaiable through vcsp (which I dont want) and suggested to me install wan accelerator :-P
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

hyvokar
Expert
Posts: 362
Liked: 23 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by hyvokar » Feb 12, 2019 2:04 pm

OK, got an email from the sales explaining.

"This add-on purchase requires that the customer has either a Microsoft Enterprise Agreement (EA) or VMware Enterprise License Agreement (ELA). To be eligible, customers are required to provide a valid Microsoft EA number or VMware ELA number."

Why on earth this kind of limitation? Well it seems that Veeam is still missing a solution for small to mid sized businesses ; so, back to my original request - can you implement some kind of insider protection to backup copy job.
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

Rick.Vanover
Veeam Software
Posts: 602
Liked: 130 times
Joined: Nov 30, 2010 3:19 pm
Full Name: Rick Vanover
Location: Columbus, Ohio USA
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Rick.Vanover » Feb 12, 2019 4:27 pm

@hyvokar: Why have that limitation? You have to consider Veeam's business model. We sell through partners and service providers. So much so, that I like to say that "Partnerships are in Veeam's DNA" .

This limitation is in there to protect the business opportunities with Service Providers. And their success with Veeam technologies will amplify the ability to make Veeam products better. So - it's good for business and its good for product innovation also.

Additionally - for a small/midsize business, why go thru all the extra work when a service provider can do it for you?

PS: I like you inner protection idea for a backup copy job however!

hyvokar
Expert
Posts: 362
Liked: 23 times
Joined: Nov 21, 2014 10:05 pm
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by hyvokar » Feb 12, 2019 6:54 pm

Hi Rick, thanks for your input. I was guessing something like that in my post this thread before, but Gostev assured "Not really a concern".
Bed?! Beds for sleepy people! Lets get a kebab and go to a disco!
MS MCSA, MCITP, MCTS, MCP
VMWare VCP5-DCV
Veeam VMCE

Gostev
SVP, Product Management
Posts: 25136
Liked: 3689 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Gostev » Feb 13, 2019 12:43 am 2 people like this post

Except Rick is totally incorrect. I have no idea why is he making such statements, when he was never involved in the corresponding decisions.
hyvokar wrote:
Feb 12, 2019 2:04 pm
Why on earth this kind of limitation?
At this time, there's just too much overhead for us to support VCC-E infrastructure at small customers, so requiring ELA was our way to make sure this technology only goes to big shops, where its complexity is truly needed (by complexity I mean multi-tenancy, gateways, per-tenant quotas, etc. - in other words, everything that is essential to service providers who are serving hundreds of clients).

This may change in future though, as VCC-E matures and results in minimal support load even from inexperienced users. I don't exclude the possibility that it will eventually become an integral part of Enterprise Plus - the edition that has always been about multi-tenant and self-service capabilities. However, we would still need to figure out how to guide our customers to only deploy it in infrastructures where it truly belongs, like massive ROBO environments that need to consolidate backups to HQ. I agree ELA requirement is clumsy, it was nothing but a quick and dirty fix - but it does work, 100% of current VCC-E users are ROBO environments.

For example, deploying VCC-E just to get Insider Protection is for a few backup jobs is definitely an overkill :D there are certainly cheaper ways to get true air-gap than deploying and managing a multi-tenant platform designed for service providers!

Besides, VCC-E can only truly provide Insider Protection when it is installed in a separate data center managed by a totally separate IT team. It will most likely will not help you much if you install it in the dark corner of your single data center, but within the same network and managed by the same IT folks... because in this case, it does not add extra protection comparing to simply using a backup repository that supports native storage snapshots.

mitchellm3
Novice
Posts: 6
Liked: 3 times
Joined: Apr 12, 2016 8:08 pm
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by mitchellm3 » Mar 04, 2019 3:46 am 1 person likes this post

Gostev wrote:
Nov 08, 2018 6:04 pm
Unfortunately, such feature would be utterly useless, because backup files can still be easily deleted using standard OS tools. In fact, this is how it's usually done anyway - most hackers don't even bother starting the backup console, when it's way faster to just run rm /rf or format d:

And, needless to say, cryptolockers don't bother going through the backup console to do their thing either ;) so, with this feature, not only you will still get your backups encrypted and unusable, but you also won't be able to delete them through the backup console to free up disk space for the new ones! :D
As more and more customers are moving away from tape, the desire to find a solution that provides "air-gapped" protection is increasing evermore. No one wants to suffer a ransomeware attack and be left with empty or encrypted appliances. Many solutions have been posited but seem to be a stretch at best. Finding a solution for the majority of your customers will take some time. However, for a minority of your customers, the promised land of an air-gapped solution has just been quickly passed over. I believe implementing MFA or a "delete password" will open up many opportunities for some of us.

Even though you dismissed using MFA or the delete password, your other comment makes me feel better. If cryptolockers don't bother going through the console, then for right now my backups are safe. See, I use HPE StoreOnce and more importantly the catalyst protocol of which requires an API to access the data. That API only exists in the backup application. AFAIK, there is no "catalyst explorer" which allows you to connect to catalyst shares if you know the password. At least there wasn't 2 years ago when I was troubleshooting catalyst issues with VEEAM and asked for a tool like that.

In order for StoreOnce to be close to air-gapped, I'd need to secure the appliance and protect the backup application.

Starting with the appliance, you warned in another email that relying on "immutable" disk arrays is no good if access is gained to factory reset an appliance. I take that to heart and believe it would be quite easy to make the management and ILO networks inaccessible to our primary site. After all, we don't do much management on the StoreOnce appliances. You could require having to RDP to a machine at the remote site of which only that machine would then have access. I could even put the Citrix VDA on it and require MFA for access. Again, that would be quite a hurdle for a cryptolocker or hacker to get over to gain access to the StoreOnce appliances since that machine would not be advertising it's role.

So, if I have locked down my StoreOnce appliances, now my biggest vulnerability is having someone or some program compromising the manager. Through the manager, repositories or backups could be deleted. If we required MFA for delete tasks or a "delete password" I think that would about make the solution pretty close to air-gapped. If you only prompt for manual delete tasks, it won't be too bothersome except for those that have constant space issues. Even then, you could make it a feature that would need to be enabled or have the MFA/delete password be good for 4 hours.

I'm not sure if this helps the DDboost crowd or if Exagrid has its own protocol as well but the API based access for backups is inherently more secure and maybe the market may move more in that direction.

bgalante
Enthusiast
Posts: 43
Liked: 2 times
Joined: Jul 27, 2015 5:14 pm
Full Name: Brian Galante
Contact:

[MERGED] Air Gap Question

Post by bgalante » Apr 18, 2019 1:34 pm

I was wondering if this would work for an Air Gap.

I use CIFS shares for my repos.

If I were to map the same repo cifs share on a Linux server that is not joined to the domain, then rsync the repo to storage that is only attached to the linux box.

Then, in case of emergency, I could share out the storage on the Linux box via CIFS, then within veeam, attach to it? Would that work?

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by foggy » Apr 22, 2019 4:38 pm

Hi Brian, will this secondary repository be offline all the time except when data is being written to it? Otherwise it is not a 100% air-gapped backup.

bgalante
Enthusiast
Posts: 43
Liked: 2 times
Joined: Jul 27, 2015 5:14 pm
Full Name: Brian Galante
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by bgalante » Apr 26, 2019 6:49 pm

Well... When I say offline, the disk area where I copy to would only be accessible via one Linux server and only via ssh, and it would not be bound to AD.

I would have two mounts on my Linux box.

One mount to the Veeam Repo where all my primary backups go to, and another mount to an NFS share that no other server has access to (Except the linux box of course.)
so mounts would look like:

/export/cifs/Veeam_Repo (This is accessible via windows / Veeam Infrastructure and is where all the backups)
/export/nfs/air_gapped_area (This is accessible ONLY from the linux box, the linux box is not bound to AD)

Then, rsync /export/cifs/Veeam_Repo to /export/nfs/air_gapped_area

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by foggy » Jun 13, 2019 5:07 pm

If it is accessible all the time, then there's a chance to compromise it, right? Of course, it is safer than some other approaches, but not a true air-gapped one.

ClarkO
Lurker
Posts: 1
Liked: never
Joined: Apr 07, 2016 8:44 pm
Full Name: Clark Davidson
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by ClarkO » Jun 26, 2019 9:18 pm

Gostev,
You mentioned in your last blog that you had a customer that built a cheap and elegant solution of blending tape right into their existing backup strategy. "Air-gapped" . I am very interested and knowing how this customer was able to do this within Veeam. I have talked to tech support and asked them about an "Air-Gapped" solution and they were a little perplexed and not sure what to do. They wanted me to calculate the backups and make sure the backup I would run won't run during the time of the air-gapped backup. Can Veeam just use the tape feature to do a backup copy of last weeks backups? I could always program the computer that had the tape backup drive in it to shut down and start up when needed. How did this customer of yours do what he was able to do? I am looking for a Air-Gapped solution for our company and I need to present it to the board, but I need to know how to do it so I can get equipment costs and everything to get it done.

Gostev
SVP, Product Management
Posts: 25136
Liked: 3689 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Reply to Gostev, Air-gapped backup

Post by Gostev » Jun 26, 2019 10:25 pm 1 person likes this post

This is easy to do with Backup to Tape job, using primary backup job as the source. Basically, this will let you copy each daily backup to tape, right after the backup is created on disk. If you have some issues settings this up, feel free to create the dedicated a topic in the Tape subforum, as to not to derail this discussion.

Air-gap is normally achieved by physically removing tape from the tape library, and putting it in a safe for some time - until it is old enough and can be reused. This is what the customer is going to do, protecting last 7 days of daily backups like that. I actually explained this in the very same blog, so perhaps you just missed it ;)

Post Reply

Who is online

Users browsing this forum: veremin and 24 guests