Long Term Storage Solution

[nunciate]

Hello, I have been running a few ideas around my head lately thinking about using tape storage vs cloud storage vs maybe immutable storage.

I thought I'd ask the community what you all are doing these days.

First, let me describe my requirements. Our owner requires we keep backups of all data in 2 locations both on disk backups and on tape.
The current policy is to Veeam backup all data daily and keep it on disk storage for 30 days. Also, we backup all of that data daily to tape keeping those for 30 days for Incremental and Weekly full backups, 3 years for the monthly full, and forever for a yearly full tape out.

Now for the OverKill... We also Veeam replicate the entire environment to our DR facility where we have the exact same VMWare/SAN setup and the exact same backup setup. Same as above, we backup everything in DR (all live VM and replicated). We keep that for 30 days on disk and do the same tape retentions.

Tapes in HQ are kept in an On-Site vault and the same in DR.
The original request was that the long-term archival be both offsite and offline. Being that we backup here and in DR we feel we do have offsite backups (though not traditional like Iron Mountain).
As for offline, maybe there is a better way to think about offline.

All said we are talking over 100Tb of data on the SAN before dedupe and compression. I don't have my exact change rate handy right now.
Either way, we are looking at over 100Tb of full backup each week so we are chewing through about 40 LTO7 tapes in HQ each week today. Clearly not getting very good compression on those tapes.
We go through even more LTO6 tapes in DR but we have a ton of those tapes so we don't buy them. We spend maybe $3k a month on tapes for HQ though.

That all said, I guess my question is this. Is there a better way to do this? Given that we keep our tapes onsite at each location.
Should I consider just installing some kind of Immutable storage and just keep my backups there instead of tape? Being Immutable might qualify as offline since it can't be deleted.

We also considered cloud storage but the problem is cost. There is the cost of the initial storage (over 100Tb) and then the growth over time. I can see cloud storage being extremely expensive. Also, there is the likely need for a separate network to backup across so that is even more cost. The benefits of the cloud would be that we only need 1 copy since it could be accessed from either datacenter and it is offsite. It could also be configured to be Immutable.

My only other idea is to just keep using tape. It is reasonably cheap overall but hard to manage since we have over 1500 tapes in storage in each location. My tape library in DR is likely going to have to be replaced in the next year but again, I doubt it will cost more than a SAN and if I go the SAN route I'd need 1 for each location.

If you made it this far, thanks for reading through my long post. Let me know what you would do in my place.

[Chrispyyy]

You could probably do with upgrading to LTO9 from LTO7, as that will give you 3x the capacity per tape. Only disappointing thing with LTO9 is that it didn't give the original 2x quota on native storage. LTO10 is projected to be 36TB native, but no one knows when that will be announced - LTO9 was delayed because of COVID.

Tape is still a viable option for long-term storage IMO.

[david.domask]

Not an official Veeam answer, but I also second make the investment further into tape. You're already familiar with the management concept, LTO9 20 packs go for about $2500-$3000 USD which nets you 900 TB RAW, and while I'm just guessing based on the numbers you're telling for your backup data, I'm guessing that wouldn't be too painful for your situation. The LTO situation is really just getting better and better with each generation, and I think it meets all your criteria now and better than other options.

Regarding your re-thinking on offline, I will posit that no software immutable solution is true immutable. As long as _some_ malicious process can run on the machine, the files are editable, neverminding how easy it is to "punk" a repository, especially if it's a VM. With Veeam Hardened Repositories, we are super open about the threat vector and how it works (just chattr +i basically with all the fanciness of controlled immutable), and that you need to ensure nothing else gets on this server, and Single-Use Credentials means not even attacking the DB will give you the credentials for the repository; that should both give you assurance that you know what you need to protect, and also make you nervous about using anything but a physical server for a Hardened Repo.

https://community.veeam.com/discussion- ... #post24561

I wrote this post awhile back but I stand by my words:

...but put your short-term backups on a proper filesystem like XFS (or ReFS if you want to go the Windows route) and rely on their integrity checks for catching bitrot/repairs. Enable Health Check for your most critical machines to avoid overloading these storages. Tape-out or Offload your backups to a long-term storage as soon as you are feasibly able to; with Tape, use periodic verification of the tapes as they cycle in/out and include a retrieval from your vaults to validate tapes and ship them back out again. For S3, don’t rely on S3 applications bundled with NAS devices for longevity.

Edit: Also, if you're really wanting to go to cloud, consider that AWS and Azure charge for API calls, and that can get expensive over time. I know a lot of our S3 Partners have self-hosted solutions, and I've seen some impressive results for complete infrastructure recoveries from Capacity Tier after ransomware attackers took out the entire environment. The slowest part was downloading from the S3 engine, but with some tweaks I know their vendor was able to improve the situation. Just be sure to really put the S3 vendor through their paces with test restores.

[nunciate]

Ultimately we decided to just stick with tape and will be ordering a new library soon.

[ober72]

Hi,

Also keep in mind that tape is not really air gap in this context. I had a customer point this out to me years ago with our Tenant to tape backups in Cloud Connect. The catch is NIST's definition states that no transfer of data can be automatic " any logical connection is not automated", only manual human transfer. Semantics of course but can be important if dealing with compliance etc. I guess if you are doing file to tape jobs and manually adding the files yourself to the job each time then that would comply.

https://csrc.nist.gov/glossary/term/air_gap

[ober72]

in fact I don't think anyone actually used the term air gap here , I just see it everywhere so assumed. I think you said that you have decided to use tapes definitely cheaper than using cloud providers. The only issue would be scaling if you had a lot of growth. Then there are some nightmares that I faced when tape drives broke down and jobs would get behind.