Advice on backup job approach - Hyper-V/proxmox + immutable hardened repo + tape

[crowsprofiles]

Hello!

I hope this is an appropriate place to get some advice and discussion about the best approach to use the different Veeam backup job types. I have searched but couldn't find precisely where such topics go.

We have a mixed environment of a few Hyper-V VMs (fewer and fewer) and Proxmox VMs. We use the Hyper-V agent for those workloads and individual on guest agents for the Proxmox VMs and for physical machines. We use hardened linux repository with immutable storage and a tape server with a library attached.

We would like to have at least a nightly incremental backup for each workload, a weekly full (active due to the immutable storage) and a place the full backups on tape. After all the full backups have been placed on tape, they are to be ejected and moved off-site.

Currently we have a number of jobs, based on host type, grouping multiserver services together, based on priority etc. Each job generally set up in the same way, with an incremental nightly activity and a full active weekly backup. They are mostly scheduled in a few parallel chains. We have a separate weekly backup-to-tape job for picking up the latest full active backups. Incremental jobs are finished quite quickly, but some full active backup jobs can drag on and take more than 24 hours or more.

I am wondering if for example, it would be a better idea to separate out the full active backups into its own job with a copy to tape activity in the job or a separate job immediately after.

Or maybe it would be better to have a copy to tape activity in the existing jobs, but I wouldn't be able to deterministically know which would be the last job to eject the tape.

Ideas? Am i totally wrong here? Thanks!

[soncscy]

Heya crowsprofiles,

Your current set up with just a single job is just fine; the only recommendation I'd ask about it do you really want the active fulls? It's heavy on production, and with XFS/ReFS + fast cloning, you can really get fast synthetic fulls. If remaking the file system isn't feasible right now, is the repo struggling with the IO requirements for Synthetic Operations?

As I get it, your real issue is this:

> but I wouldn't be able to deterministically know which would be the last job to eject the tape.

So you need to add some details; are you writing everything to a single tape with a single job? With a few jobs?

If it's just one job, simply run the tape job once all the source jobs have run and you can set the job to export automatically.

If it's a ton of jobs, it's a simple post-job script to eject a given tape.

Please don't take this as dismissive, but since you even have a concern about knowing when to eject the tape, I'm assuming you aren't using too many tapes in this environment (please correct me if I'm wrong). If that's the case, it's pretty easy to just write up a script that checks the status of the tape jobs and ensures they all wrote data successfully, and if so, eject the tape. Or make it even simpler and just eject the tape regardless once the jobs __should__ have run; let your email reports tell you if you need to re-insert the tape to rerun the job.

In short, keep it as stupidly simple as you can; remember that "no data backed up" is in fact a potential (and desirable) outcome that results in success, and instead of trying to determine the exact point when you should eject the tape, rebuild your workflow to protect the tape once the jobs have run successfully, and use reporting to catch exceptions (e.g., no data backed up on a job). Exceptions are just that, and your workflow should have discrete success/failure definitions, understanding that sometimes 0 bytes backed up is a success definition (because nothing is "wrong" per say, but the conditions set on the job defined no data to backup).

Clarify your tape-out objectives and we can advise further. But in short:

- There's no need for a separate job unless you want it; the result is just extra space/processing/pressure on production without merit
- Define your tape-out needs and success/failure conditions
- Define when exactly you want the tape out of drive

Likely, this is easily done. It's good you're thinking about these things, but you post leads me to think you might be best served defining your current needs a bit more clearly.

[crowsprofiles]

Hey!

Thanks! Lots of good points there, than you.

I'd ask about it do you really want the active fulls? It's heavy on production, and with XFS/ReFS + fast cloning, you can really get fast synthetic fulls.

Active full jobs are definitely heavy on production, but as we understood it, incremental/synthetic fulls don't really work with immutable storage. Reading the documentation again, it seems that might be wrong, synthetic full is compatible with immutable storage. The infrastructure will be happy about that!

You're not wrong about not having a lot of tapes. With proper planning and prioritization, we should be able to fit the off-site critical data on one or two LTO8 tape - I want all the relevant backup jobs placed on tape and then ejected so it can be removed and transferred off-site. Because they're just copy jobs where I want the tape ejected as soon as they are all on the tape, I was thinking about how to achieve that. Scripting is probably the most straight forward - thanks for bringing that up.

- There's no need for a separate job unless you want it; the result is just extra space/processing/pressure on production without merit
- Define your tape-out needs and success/failure conditions
- Define when exactly you want the tape out of drive

Separate jobs - KISS is the way to go, that's clear!

Tape-out success/failure / when eject needs

Walking backwards - I want a tape (i) ejected (exported) once a week containing a (ii) full backup from that (iii) week of (iv) each identified defined jobs.

I think I have a few possible strategies here
1. Script it - have a scheduled task checking for all the full jobs to finish and then initiate a copy to tape job that ejects the tape when finished.
2. Separate scheduled backup job - with synthetic full backups, there shouldn't be a big penalty to rerun them. But I can't do a job for both VM and different agent based computers so not great.
3. Separate scheduled tape copy job - a job that takes the full backups - have to get clarification on how the option "Export current media set upon job completion" and "Schedule - as new backup files appear". When would the job be considered complete (if at all) if the job is scheduled for "As new backups appear"? Especially since Incremental backup is disabled.

I'm not sure I'm totally clear on my strategy yet, but at least confused at a higher level. Thanks!