Feature Request - Bootstrap to Tape

[ekisner]

I was recently evaluating our backups, particularly against a scenario where malicious actors have infiltrated and compromised all resources, leading to a full site loss. Starting recovery of all resources from scratch.

The one thing a malicious actor can't hit is the air-gapped tapes I keep in a fire resistant safe in a separate building. Even if they have somehow corrupted the backups, if I go back far enough, I've got good tapes. Since I routinely audit the contents of tapes to verify integrity, I am always able to ensure that data loss will always be kept to a minimum.

Formatting the Veeam hardware, re-installing the OS, and re-installing Veeam is easy. Connecting the tape drive is easy. If the repository has been destroyed, however, I cannot recover the config. If the database is destroyed, finding the database on tapes would be a challenge. It's not impossible to recover from, by any stretch, but it's still "more to deal with". And speedy recovery is what separates good from great.

The Veeam database is pretty small, all things considered. The config, even more so. In the context of a tape job backing up a production environment, their size is almost irrelevant.

It would be super cool if, with the check of a box, my GFS tape job could include a bootstrap of everything Veeam needs to get back to that point in time. A copy of the database, a copy of the config, and the functionality to gracefully import and implement both from a simple wizard that lets you hit next a couple times.

At that point, getting my B&R server back up to 100% is as simple as running a wizard, letting me streamline the hardest part of a black start.

Thoughts?

[Mildur]

Hello Erik

I moved your request from the PowerShell forum to the Tape forum.
Have you considered to sent a config backup to a immutable S3 object storage?
One example with Wasabi (There are many other immutable S3 services). Create an immutable S3 bucket and configure your configuration backups with Wasabi as a target.

If you need to recover your entire server, you can download your BCO file from Wasabi and import it to a new backup server. It will contain all information about restore points, tape information and backup server components.

A copy of the database, a copy of the config, and the functionality to gracefully import and implement both

A database copy itself won't be much help. All credentials would be encrypted by the old backup servers machine key. The new backup server cannot use them. Therefore all Credentials have to be reconfigured.
Our configuration backup contains everything you need to recover your backup servers entire configuration. Restoring the configuration from an encrypted configuration backup file will import the original credentials.

Best,
Fabian

[jeronimo]

Hello,

It's interesting that the only response here is "just use S3".

Everything can be cloned to tape, in fact I can just choose to clone an entire repo (whatever is on it) to tape, and I could think that this includes the bootstraps also stored on that repo. But that is not true AFAICS.

OP is right, there would need to be a complete workflow to clone *everything* (including Veeam bootstrap/config itself) to tape and be able to easily restore from it. Not sure why that part was left out.

Best regards

[Mildur]

Hi Jeronimo,

Welcome to the forum.

If I understand correctly, you’re asking whether there’s an option to write the configuration backup directly to tape.

The challenge I see with that approach is the restore workflow: to recover your backup server from a configuration backup file (BCO), you’d first need to deploy a new backup server, configure a tape server, configure your tape device, and then restore the BCO file. Keeping the configuration backup on object storage or a block-based repository is typically much faster, since you can avoid the tape configuration and file-restore steps during recovery.

I’ll add your vote to the request to write BCO files to tape with Backup To Tape jobs, but I can’t promise anything; this request currently does not have a high priority compared to other open requests. File To Tape Jobs can already be used to write BCO files to Tapes.

Best regards,
Fabian

PS:
Starting with v13, we introduced a new capability: if the backup server runs on the Veeam Software Appliance, you can configure High Availability. This replicates the entire configuration to a second appliance. If the primary appliance fails, you can fail over to the secondary appliance and continue operating. Maybe that’s something you’d like to consider as part of a future design update.

[ekisner]

Good morning! I got a notification about the activity on this thread, so I thought I'd weigh in. The config is only part of this, and given all you've said, it's not even the big part because like you say good options already exist.

Consider a situation where you've lost your entire site. Could be a malicious actor, could be a flood, doesn't matter... site's toast and you're starting from zero. What is your order of operations to get back online?

First, you need a B&R server. So you get an OS running on bare metal, install Veeam from ISO, import the config from your S3 storage or your file-to-tape or whatever else. Now you've got Veeam, but you don't have any of your backups - they're there but they're still unknown to the "new" instance. So far there's no big issue.

My next step is to catalogue the tapes so that this new instance can know what backups are on the tapes... my own environment that means 3 LTO8 tapes at the moment. A full day just to catalogue the backups just so that I can start restoring the backups from tape to disk. That of course assumes that this tape was good and not somehow compromised - if it was, we grab the next set and catalogue that, so on and so forth.

This feature request is primarily about avoiding that last step of cataloging the tapes. I can get straight into restoring high priority backups from the tape to the repo, so that I can rapidly start recovering the high priority targets first. I hear you when you say the DB isn't directly usable on a new machine for the reasons you outline. That's mostly an implementation detail. It doesn't even need to be the whole database. All it really needs is a tape job history of sorts.

The ability to rapidly know what data exists and on which of the ~150 LTO8 I have in my vault would dramatically improve tape recovery times. Whether that's a full DB export, or even just a tape catalogue, it would do the trick.