Offline Airgap for LOTS of data

[electricd7]

Hello-

I am trying to implement an offline airgap plan for a large backup environment. Currently I have several hundred VMs and 50 or so Veeam Agent backups which run nightly to a onsite scale out repository that is about 240TB in size. Following the backup, there is a copy job which runs the copy-from-backup type to an off-site scale out repository to get the data offsite. That repository holds about 140TB of data at any time as the offsite keeps only 7 days of backups. What I would like to institute is a 3rd airgap process where the most recent backup is written to another media which can be physically taken offline and stored. At first I had planned to use another scale out which I could just power down the members following the copy, but it just takes so long to get those copy jobs written when there is this much data. I have looked at cloud-connect as a method, but even so an intelligent hacker can compromise the Veeam server and then delete the backups from CloudConnect just like if they were local. I do know about the insider protection checkbox in CC, and that is an idea, but think I would be limited by bandwidth to get it in cloud connect as we currently enjoy 10GB to backup and 10Gb to offsite. So that brings me to tape and LTO8 auto-loaders. I don't have a lot of recent experience with tape, but it seems logical that we could implement a copy to tape process and take those backups to tape on a schedule and then pull the physical tapes out to protect them. Is that what most are doing? What am I missing? I have read a little about file-to-tape where we could just copy the repo data directly to tape and that might be an option as well, but there are 11 repos here. If we go that route, will it affect our ability to write to those repos while they are writing to tape? We essentially run our backup window over a 12-18hour range, so that doesn't leave much time to go to tape if it will lock the files. Help if you can, I appreciate any advice!

Thanks

[ejenner]

Backing up to a Linux repository is a way of guarding against malware which may have run loose inside your network. It's unusual for malware to be able to exploit the same weaknesses in two kinds of operating system. So if the Windows servers were affected, the Linux ones will likely not be. And vice versa. If it isn't only that which you're worried about then perhaps think about a 'rotated disks' setup. You can enable the option on the repository configuration for rotated hard drives. So some smaller storage devices which can be disconnected could be rotated in and out of use and taken off site.

[nitramd]

Chad,

When I started out in my current position I did not know a lot about Veeam tape backup but I must say that it is incredibly easy to set up and use; just consult the online documentation as needed. With so much data to backup you might consider a tape library that contains 2 or more drives.

I fully support @ejenner's idea of creating Linux based repos; I've repurposed an older server for just this reason with motivation driven by the ransomware outbreak of 2 years ago. The repo works quite well.

[electricd7]

Thanks for the responses. I do like the extra layer of protection of a linux repository, perhaps we can add a layer of complexity that way. I think we are out of the realm of swapping physical hard disks out since we have so much data to sync, thus i am leaning more heavily towards a tape library. If we were to run copy to tape jobs, or file to tape jobs, would we still be able to continue to backup while those are running? We are currently running forward incremental on all jobs, but the merges I would think would still need to access the VBK files which might be locked while they are writing to tape?

[tommyo]

The Linux Repository is a good idea but if your leaning towards tape take a look at Quantum's iScalar3 or iScalar6. We use the iScalar 3 with two drives and the builtin iBlade. All our primary jobs write to disk and then immediately write off another copy to tape. Works like a charm.

[PTide]

Hi,

would we still be able to continue to backup while those are running? We are currently running forward incremental on all jobs, but the merges I would think would still need to access the VBK files which might be locked while they are writing to tape?

There are no merges in forward incremental, you must have meant forever-forward incremental, right? Source job always has a priority, so tape job will be terminated with error. That can be changed via "Prevent this job from being interrupted by source backup jobs" option.

Thanks!

[electricd7]

Correct...forever incremental is where we are currently. I would think once we had the tapes "seeded" we would be OK, just worried about the initial writes to the library. Perhaps we should be using forward-incremental with synthetic fulls once a week to make this tape strategy more appealing? I would like to think we could backup to tape twice a week and rotate magazines so that we always have at least 3 day old data (not great, but not sure how else to manage this amount of data on a daily basis?) Really could use some good advice to get this right.

[18436572]

We are looking to expand our Veeam infrastructure with a Quantum i3, this thread is timely for me!

Question: We leverage Veeam's CBT for forever incremental to disk repositories (and copy jobs too). Does tape leverage CBT?

Thanks in advance!

[PTide]

I would think once we had the tapes "seeded" we would be OK, just worried about the initial writes to the library.

Yes, as long as the length of a backup chain allows to start a tape job two times a week without letting merges to get to the point where increments that haven't been copied yet reside.
I might be missing something important about your design, but why not to let the tape job to copy the latest increment on a daily basis (+ virtual full)?

Thanks!

[PTide]

Hi Devin,

Does tape leverage CBT?

Please elaborate on that. CBT is already leveraged in backup jobs, so there is no point in using CBT in Tape jobs. Normally, blocks in backups that are to be copied are not supposed to change (that's usually called 'corruption') : )

Thanks!

[18436572]

Thanks PTide,

We are a VMWare shop. We run Veeam backup jobs to disk using forever incremental with periodic fulls. CBT is used to create the incrementals. We then copy these via a separate copy job to a second offsite repository with monthly retention for 5 years.

I hope to add a tape changer and add additional copy jobs to these tapes with similar retention (monthly, retain for 5 yrs). My question is simply if the initial copy job will be a full and subsequent jobs incremental? I suppose I didn't make that clear

I am still investigating the best way to produce an air-gap for our backups for ransom-ware needs. I have read the Veeam whitepaper on this and for us a tape changer make the most since. We run about 60TB of production data but have long retention needs (5yrs for some).

[PTide]

My question is simply if the initial copy job will be a full and subsequent jobs incremental? I suppose I didn't make that clear

That depends on how you setup the backup to tape job. You can choose to copy fulls only or fulls+increments, that is controlled by the "Archive incremental backups to tape" checkbox.

Another way of dealing with full backups on tape is to switch the source backup job to forever forward incremental mode (i.e. without fulls) and utilize virtual fulls in the tape job.

Thanks!