Reply to Gostev, Air-gapped backup

[pkelly_sts]

Hmm, is this an enhancement opportunity for Veeam? Create a new kind of repository/agent combination - An agent that can ONLY "pull" backups to itself, and a repository-type that can ONLY be "pulled to" by such pull agents?

Obviously would be dependent on that machine being as utterly locked-down as possible in every other way...

[yasuda]

Has anyone looked into Wasabi immutable cloud storage ?
https://wasabi.com/blog/data-immutability-done-right/

Perhaps some Veeam Managed Cloud Providers will offer a service with immutable storage - maybe using Wasabi on the back end. It would be nice to not have to upload full backups.

[hyvokar]

How about using cloud gateway technology for internal use and add an option to protect the remote backups from deletion? Or would this eat too much cloud providers' markets?

[Gostev]

Not really a concern, for example we have had Cloud Connect available to regular customers for a few years now > Veeam Cloud Connect for the Enterprise

[hyvokar]

Cool! Didnt know that option existed for enterprises. That could possibly make my life a bit easier...
Any change to write protect the backups on the cloud gateway server?

[veremin]

Can you elaborate what change you're talking? If you're asking for Insider Protection feature, then, it is available in Veeam Cloud Connect for the Enteprise, correct. Thanks!

[jihering]

I'm sure this has already been addressed. If so please point me in the right direction.

New VEEAM Backup and Replication user (from an OLD version of Backup Exec tape system). We are a small school district. We recently had new VM Servers, Nimble SAN and QNAP backup storage installed. migrating old VM's to new servers and storage. Running daily VM backups with weekly synthetic fulls. Everything set up and configured by an outside vendor. Been struggling getting through the 1000+ page user manual. I'm a little freaked out by what I've been reading about ransomware locking servers and deleting VEEAM backups. Is it possible to occasionally just copy off the data from a full backup to an external USB storage device for offline/offsite storage? I can see the backup files on the VEEAM server's storage drive (the QNAP).

Again, I apologize if this is a repeat. I couldn't find exactly what I was looking for searching through the forums.

[wishr]

Hi Jihering,

Welcome to Veeam Community Forums and thank you for posting your query.

Sure, you can set up a Backup Copy job to your USB drive, once completed remove the drive from the machine.

Please let us know if you have any additional questions on that. Thank you.

P.S. I've moved your post to an existing thread - definitely recommend taking a look.

[bdufour]

ive been thinking about this, we never had backup appliances tied to AD auth, but obviously veeam is installed on a production server, which is tied to AD and i would like to keep it that way bc of kerberos auth, management, and compliance. i think if veeam could enable MFA on the console that would protect us in the event that a privileged account was compromised, which could log into veeam and delete backups. we have alerting set up whenever an admin account logs into a server, but it may be too late at that point. the biggest thing is, as an admin, i can delete backups from the console - that is what scares me the most. if a cryptolocker were to encrypt the windows server veeam is hosted on, that wouldnt be the end of the world bc the appliance the backups are stored on isnt tied to AD and if MFA was enabled on the console for veeam that would protect the backups at that level.
since veeam isnt providing MFA at this moment, we may look at some other MFA integrated option for admin accounts.
open to ideas as well guys, this has been a big topic, since ive brought it up with the dept.

[hyvokar]

Can you elaborate what change you're talking? If you're asking for Insider Protection feature, then, it is available in Veeam Cloud Connect for the Enteprise, correct. Thanks!

Thanks! Exactly what I was looking for.

[hyvokar]

Not really a concern, for example we have had Cloud Connect available to regular customers for a few years now > Veeam Cloud Connect for the Enterprise

Ok, called your sales today to find out what is the difference between std, ent and ent+ edition and was told, that cloud connect is only avaiable through vcsp (which I dont want) and suggested to me install wan accelerator

[hyvokar]

OK, got an email from the sales explaining.

"This add-on purchase requires that the customer has either a Microsoft Enterprise Agreement (EA) or VMware Enterprise License Agreement (ELA). To be eligible, customers are required to provide a valid Microsoft EA number or VMware ELA number."

Why on earth this kind of limitation? Well it seems that Veeam is still missing a solution for small to mid sized businesses ; so, back to my original request - can you implement some kind of insider protection to backup copy job.

[Rick.Vanover]

@hyvokar: Why have that limitation? You have to consider Veeam's business model. We sell through partners and service providers. So much so, that I like to say that "Partnerships are in Veeam's DNA" .

This limitation is in there to protect the business opportunities with Service Providers. And their success with Veeam technologies will amplify the ability to make Veeam products better. So - it's good for business and its good for product innovation also.

Additionally - for a small/midsize business, why go thru all the extra work when a service provider can do it for you?

PS: I like you inner protection idea for a backup copy job however!

[hyvokar]

Hi Rick, thanks for your input. I was guessing something like that in my post this thread before, but Gostev assured "Not really a concern".

[Gostev]

Except Rick is totally incorrect. I have no idea why is he making such statements, when he was never involved in the corresponding decisions.

Why on earth this kind of limitation?

At this time, there's just too much overhead for us to support VCC-E infrastructure at small customers, so requiring ELA was our way to make sure this technology only goes to big shops, where its complexity is truly needed (by complexity I mean multi-tenancy, gateways, per-tenant quotas, etc. - in other words, everything that is essential to service providers who are serving hundreds of clients).

This may change in future though, as VCC-E matures and results in minimal support load even from inexperienced users. I don't exclude the possibility that it will eventually become an integral part of Enterprise Plus - the edition that has always been about multi-tenant and self-service capabilities. However, we would still need to figure out how to guide our customers to only deploy it in infrastructures where it truly belongs, like massive ROBO environments that need to consolidate backups to HQ. I agree ELA requirement is clumsy, it was nothing but a quick and dirty fix - but it does work, 100% of current VCC-E users are ROBO environments.

For example, deploying VCC-E just to get Insider Protection is for a few backup jobs is definitely an overkill there are certainly cheaper ways to get true air-gap than deploying and managing a multi-tenant platform designed for service providers!

Besides, VCC-E can only truly provide Insider Protection when it is installed in a separate data center managed by a totally separate IT team. It will most likely will not help you much if you install it in the dark corner of your single data center, but within the same network and managed by the same IT folks... because in this case, it does not add extra protection comparing to simply using a backup repository that supports native storage snapshots.

[mitchellm3]

Unfortunately, such feature would be utterly useless, because backup files can still be easily deleted using standard OS tools. In fact, this is how it's usually done anyway - most hackers don't even bother starting the backup console, when it's way faster to just run rm /rf or format d:

And, needless to say, cryptolockers don't bother going through the backup console to do their thing either so, with this feature, not only you will still get your backups encrypted and unusable, but you also won't be able to delete them through the backup console to free up disk space for the new ones!

As more and more customers are moving away from tape, the desire to find a solution that provides "air-gapped" protection is increasing evermore. No one wants to suffer a ransomeware attack and be left with empty or encrypted appliances. Many solutions have been posited but seem to be a stretch at best. Finding a solution for the majority of your customers will take some time. However, for a minority of your customers, the promised land of an air-gapped solution has just been quickly passed over. I believe implementing MFA or a "delete password" will open up many opportunities for some of us.

Even though you dismissed using MFA or the delete password, your other comment makes me feel better. If cryptolockers don't bother going through the console, then for right now my backups are safe. See, I use HPE StoreOnce and more importantly the catalyst protocol of which requires an API to access the data. That API only exists in the backup application. AFAIK, there is no "catalyst explorer" which allows you to connect to catalyst shares if you know the password. At least there wasn't 2 years ago when I was troubleshooting catalyst issues with VEEAM and asked for a tool like that.

In order for StoreOnce to be close to air-gapped, I'd need to secure the appliance and protect the backup application.

Starting with the appliance, you warned in another email that relying on "immutable" disk arrays is no good if access is gained to factory reset an appliance. I take that to heart and believe it would be quite easy to make the management and ILO networks inaccessible to our primary site. After all, we don't do much management on the StoreOnce appliances. You could require having to RDP to a machine at the remote site of which only that machine would then have access. I could even put the Citrix VDA on it and require MFA for access. Again, that would be quite a hurdle for a cryptolocker or hacker to get over to gain access to the StoreOnce appliances since that machine would not be advertising it's role.

So, if I have locked down my StoreOnce appliances, now my biggest vulnerability is having someone or some program compromising the manager. Through the manager, repositories or backups could be deleted. If we required MFA for delete tasks or a "delete password" I think that would about make the solution pretty close to air-gapped. If you only prompt for manual delete tasks, it won't be too bothersome except for those that have constant space issues. Even then, you could make it a feature that would need to be enabled or have the MFA/delete password be good for 4 hours.

I'm not sure if this helps the DDboost crowd or if Exagrid has its own protocol as well but the API based access for backups is inherently more secure and maybe the market may move more in that direction.

[bgalante]

I was wondering if this would work for an Air Gap.

I use CIFS shares for my repos.

If I were to map the same repo cifs share on a Linux server that is not joined to the domain, then rsync the repo to storage that is only attached to the linux box.

Then, in case of emergency, I could share out the storage on the Linux box via CIFS, then within veeam, attach to it? Would that work?

[foggy]

Hi Brian, will this secondary repository be offline all the time except when data is being written to it? Otherwise it is not a 100% air-gapped backup.

[bgalante]

Well... When I say offline, the disk area where I copy to would only be accessible via one Linux server and only via ssh, and it would not be bound to AD.

I would have two mounts on my Linux box.

One mount to the Veeam Repo where all my primary backups go to, and another mount to an NFS share that no other server has access to (Except the linux box of course.)
so mounts would look like:

/export/cifs/Veeam_Repo (This is accessible via windows / Veeam Infrastructure and is where all the backups)
/export/nfs/air_gapped_area (This is accessible ONLY from the linux box, the linux box is not bound to AD)

Then, rsync /export/cifs/Veeam_Repo to /export/nfs/air_gapped_area

[foggy]

If it is accessible all the time, then there's a chance to compromise it, right? Of course, it is safer than some other approaches, but not a true air-gapped one.

[ClarkO]

Gostev,
You mentioned in your last blog that you had a customer that built a cheap and elegant solution of blending tape right into their existing backup strategy. "Air-gapped" . I am very interested and knowing how this customer was able to do this within Veeam. I have talked to tech support and asked them about an "Air-Gapped" solution and they were a little perplexed and not sure what to do. They wanted me to calculate the backups and make sure the backup I would run won't run during the time of the air-gapped backup. Can Veeam just use the tape feature to do a backup copy of last weeks backups? I could always program the computer that had the tape backup drive in it to shut down and start up when needed. How did this customer of yours do what he was able to do? I am looking for a Air-Gapped solution for our company and I need to present it to the board, but I need to know how to do it so I can get equipment costs and everything to get it done.

[Gostev]

This is easy to do with Backup to Tape job, using primary backup job as the source. Basically, this will let you copy each daily backup to tape, right after the backup is created on disk. If you have some issues settings this up, feel free to create the dedicated a topic in the Tape subforum, as to not to derail this discussion.

Air-gap is normally achieved by physically removing tape from the tape library, and putting it in a safe for some time - until it is old enough and can be reused. This is what the customer is going to do, protecting last 7 days of daily backups like that. I actually explained this in the very same blog, so perhaps you just missed it

[sdelacruz]

Gostev if I simply eject the tape from the drive after completing the backup job to tape, will it still be vulnerable to ransomware? We are currently all working from home and only go to the office 2 or 3 times a week. Do we have to take it off the library to be 100% safe?

[wishr]

Hi Sam,

I would say yes because you never know what the next ransomware will be capable of, thus there are some potential risks that should be properly mitigated. Also, you can never be 100% safe, but minimizing the risks is the way to go.

Thanks

[Gostev]

Gostev if I simply eject the tape from the drive after completing the backup job to tape, will it still be vulnerable to ransomware? We are currently all working from home and only go to the office 2 or 3 times a week. Do we have to take it off the library to be 100% safe?

Probably not to ransomware, however many successful attacks involve hackers who penetrated the environment with the help of ransomware, and these hackers may potentially be knowledgeable enough to understand tape, and thus know how to load ejected cartridges back and wipe them.