VBR Disaster Recovery from WORM Tapes after Cyber Attack ...

[ThierryF]

Hello friends,

We are looking at moving from LTO7 Classic mediums to LTO7WORM mediums, as
backup media protection against cyber attack, copying 2.5TB Tape backups a day VBR backups
to Azure/AWS Immuable storage being impossible.

Assuming a scenario with all of my backups to be Tape-Copy'ed on WORM Tapes
(actually avg 10 LTO7WORM Tapes a day, Avg 2.5TB Data daily, synthetic fulls being spread on all days of the week),
my VBR Repositories to be destroyed by Cyber attack, including Veeam Database ...
In other words, VBR env dead ...

Due to WORM Tapes, hacker cannot overwrite my backup medias as read-only once
mediaset is closed and/or tape full.

Assuming also that I have a SQL-based daily full backup of my VeeamDB, stored in a folder on my
Veeam Server (VBR and SQL roles on same physical server) that is protected by a VBR job, with a
daily synthetic full cycle and also being tape-copied to WORM Tapes ...

Assume also that I have a daily hardcopy of all of my backups stored on tape, out of hacker attack ...

What would be the procedure to recover my VBR Env and start VBR Guests (VM/physical servers) Recoveries ?
Reinstalling Servers (OS, SQL, Veeam) on Veeam Server/Proxy and Veeam Repositories,
then Restore SQL DB and let's go ?

But to start restoring SQLDB, tape should be known by new VBR Environment.
Should I just process like loading tapes written on another veeam server, to let new environment
aware of latest WORM tape (the one containing latest SQLBackup), then restore my SQL DB backup files,
then stop and proceed with SQL DB recovery before restarting VBR Env?
Any other missing steps ?

In such a setup (all VBR Backups and SQL DB Backups put to WORM Tapes),
what about need to backup VeeamConfigBackup repository ?
Faster DRP Recovery with VCB backups also on WORM Tapes ?

Any advises are welcome ...

Thierry

[HannesK]

Hello,

Assuming also that I have a SQL-based daily full backup of my VeeamDB

that makes everything complicated when it comes to passwords, because passwords are encrypted with the machine key of the backup server. If the backup server is lost, then you somehow have to manage restoring potentially hundreds or thousands of passwords.

I would go with configuration backup. Copy that somewhere on disk media / object storage / even USB disk might work, with whatever 3rd-party copy tool instead of doing a full restore from tape. It just simplifies everything and saves hours during restore.

Best regards,
Hannes

[ThierryF]

Good advise. VCB Backups being small in size regarding other backups,
it make sense to copy them to Azure/AWS Immuable Storage space for easier restart !

Good idea !
Have a great day !
Thierry

[IanBolton]

Hello
Sorry for resurrecting an old thread, but we have our config backup set to copy to tape, and also to both off-site linux repos with immutability set via a file-copy job. In the event of Bad Things Happening we know that we can get the config backup off tape, but also, if the entire tape setup is nuked, then from the linux repos via a USB key. Even if you dont use a linux repo in the usual setup, you could set one up on a small server with not much storage JUST for your config backup.

[Mildur]

Hi Ian

Thanks for sharing.
Files copied by File Copy Jobs will not be immutable on linux hardened repos. They can be deleted by an attacker.

Thanks
Fabian

[IanBolton]

Oh

Every day is a school day.

[IanBolton]

There's no way around this is there?

My config backup scheduled job writes to a repo - I can't do a backup copy job from that repo to linux repo. because it only deals in actual backup files?

I guess WORM tape is my only hope here...

[Mildur]

You can use an older configuration backup from a week earlier on an air gapped tape to restore the VBR configuration.

Or copy the configuration backup to a managed server which you already have protected by a backup job. Then the config backups gets immutable with this servers backup. It's more a workaround and not really required. The tape solution is the easiest way, if you rotate them.
V12 might bring some additional protection possibilities when it's released.

[IanBolton]

Would a WORM tape for the file to tape not also do the job? Then we could have it constantly appending.

Alternatively yeah we could have 2 tapes in the pool used by the file to tape job which we rotate weekly.

I suppose part of me prefers the WORM tape idea because we would have the latest config backup in case of Bas Things Happening

[Mildur]

WORM Tape will help, as long nothing happens to that tape on a physical level.

[IanBolton]

Very true, very true.
Same risk as we currently have going to a non-worm tape though.

I dunno, part of me thinks it would be so much better to have the most recent config backup. If by some set of circumstances we were left with no VBR, no linux repos, and a config backup airgapped on tape from 1 week ago, that config backup wouldn't have the knowledge of the most recent tapes which contained the most recent backups you are likely to want for forensic analysis and recovery.

I think a WORM tape is sufficient protection, if we have a [manual] schedule of verification jobs and use that tape as part of our DR testing.

[Dima P.]

Hi Ian,

that config backup wouldn't have the knowledge of the most recent tapes which contained the most recent backups you are likely to want for forensic analysis and recovery

Just to make sure you know: Veeam tape backups are completely self-sufficient, even if you have totally fresh Veeam B&R server with a tape drive connected you can import such tapes, catalog the content and start the recovery. Thanks!

[IanBolton]

Yeah but in the event of Bad Things + no config backup, I'd have to catalog 170+ tapes. I have a record of which tapes were used most recently for the fulls in my email, assuming I can get onto my email if everything is cyber-broken.

I am attempting to write my config backups to a WORM tape now, but it appears the drive had a turn and broke the WORM tape. (Case #05545053)

[Dima P.]

Understood, you can definitely use File to tape job to write config backup as a file to tape. As for the drive issue please keep working with our support team (you may also need to open the case with tape drive vendor if that's a hardware issue). Thank you!

[IanBolton]

You can use an older configuration backup from a week earlier on an air gapped tape to restore the VBR configuration.
...
The tape solution is the easiest way, if you rotate them.

Can I just check - if I have two tapes in this pool, and only one is in the Library, it will use whichever is there?

The file to tape job has "export current media set upon job completion" UNTicked.

So

wk1, tape1 will be in the drive and the config backup will be written to it every day.
Wk2, tape1 will be out, but tape2 will be in. The file to tape job will write to tape2.
Wk3, tape2 will be out, tape1 will be back in. The file to tape job will write to tape1, appending to previous.

OR do I need to erase each tape as it's but back into the library?

[Dima P.]

Tape is selected / requested based on restore point you want to restore. If required restore point is on the tape in the library - restore will be performed. If not, say you are restoring incremental restore point for image-level backup or entire folder state from file level backup - you will be prompted for the addition missing tape.

[IanBolton]

Sorry, I meant when writing the File to Tape job to the media pool.

[IanBolton]

just closing the loop, yes, switching the tapes week to week seems to give us some protection. I still want to try another WORM tape though!