Linux Backup Repository: How to make sure no job is active

[tmoehle]

Hey,

one of my secondary backup targets for Veeam B&R is a Ubuntu Server with a large zfs pool.
To protect this infrastructure against data loss by accidentally deleting Backups in Veeam or even a ransomware attack, I would like to take a snapshot every day after all backup copy jobs to this target are done.

The simple way would of course be, to trigger the zfs snapshot at a time when the Backup Copy Jobs are usually finished. But sometimes for various reasons a job might still be running and I would like to write a little script, that runs on the repository server and checks, if there are currently any jobs actively running, transfering data or merging backup files. I want to make sure my ZFS snapshots are consistent.

How can I do this? First, I thought, I could just check if there are any running veeamagent processes, but since Backup Copy Jobs are constantly "active" and waiting for new restore points, the veeamagent processes on the repository server are running all the time. Is there any way to check this?

Thanks for your ideas!

[nitramd]

Have you considered using the utility top to monitor the Veeam Agent processes? Just a thought.

[tmoehle]

How does top tell me reliably if a backup job is done?

[foggy]

You can configure backup copy jobs to write some sort of a 'completed" flag as a post-job script and then query these flags with your pre-snapshot script.

[cathym]

We are a service provider seeking to do the same thing for our XFS/ZFS repo.

We have a mix of Backup and Backup Copy jobs, coming in from tenant Agents (workstations and servers) and VBR.

With workstation agents, I don't believe we can run any post-job scripts like you can with VBR/Server agents.

This brings me to two current challenges:

1) How to take consistent ZFS snapshots. Right now ours are "dirty" because we catch various Veeam jobs at various states of writing data.
This also leads to larger snapshots, since we snap frequently (keeps snapshots small, more chances to catch various files in a consistent state in between job runs) and replicate the data in near-real-time to a second standby server for DR purposes.

2) How to keep snapshot size in check. As I said, we keep a lot of snapshots on hand since they're not "clean" and we can't guarantee we've captured all Veeam data in a consistent state in any one snapshot. Also, I wonder if frequent snaps end up capturing a lot of transient data changes causing them to grow unnecessarily.

Would love to know what others are doing, and if there's a better way....

[tsightler]

Hi Cathy. I've not been able to come up with a better way than frequent "dirty" snapshots at this point. At least no way that seems to balance the complexity of implementation and truly improves on it significantly. I've long kicked around the idea of trying to write a script that uses inotifywait to trigger events and then monitor those events and take actions like checking if the associated veeam process for that job is gone or has any other files open and triggering a snapshot if it seems to be done, but it's difficult to know if "all jobs" are done, other than looking for all veeamagent processes and seeing if any files are open. There's just so many corner cases I just wasn't sure this was really better than simply running frequent snapshots as, in busy environments, it might be hours where veeamagent processes are not busy at that leaves some chains unprotected for a long time anyway.

However, now that there are immutable backups in v11, I wonder if it might be easier to implement something like this by monitoring for changes to the immutability flag. It seems like taking a snapshot after a file is flagged immutable might be the perfect time, and be much easier to implement but that's assuming that you plan to use the new immutability feature. My idea is you could just monitor for changes to immutability via inotifywait, and then just limit the minimum frequency of snapshot to something like no more often than every X minutes, so as not to create snapshot storms when things are really busy, but maybe ignore that minimum of no veeamagent processes have open files for writing (i.e. system is truly idle). I believe this would be logically easier to implement, but I haven't actually tried to do it yet.

[jeremybsmith]

Member of Cathy's team here... I've used inotifywait for things in the past but the timing for those things wasn't as critical (e.g. processing a file after it was moved into a directory). For determining whether a Veeam backup/transform/etc job's files are in a consistent state (even assuming Veeam is the only thing that would be changing those files), it seems that there would still be the possibility of race conditions (e.g. another job starts working on a vbk file shortly after it was closed by another job). Maybe that is very unlikely to happen given how the jobs are managed, but it would be great if there were a native Veeam mechanism we could integrate with that could guarantee that at least there are no Veeam jobs working on a particular backup chain's files.

We are also starting to look at using performance extents, which it seems may further complicate any manual attempts to determine that from within the repository itself.

Could something like attempting to (momentarily) exclusively lock all of a backup chain's files work? I'm not sure whether the Veeam agent would always be abiding by those file locks.

I am also curious whether we could leverage the immutable backups functionality for this. I'll have to play around with v11 in testing, but would appreciate any updates you may have.

[tsightler]

I feel like trying to lock files is just introducing complexity to cover a rare case. Sure, is there a possibility that the snap wouldn't be valid, maybe, but if the overall goal is to just improve the accuracy vs simply scheduling frequent snapshots, I feel like it's OK if there's an occasional corner case as, assuming you're like most service providers, there's not a lot of time where "something" isn't running and you probably want to snapshot more frequently than that anyway.

I suppose another option would be to run something on the VBR server that triggers the snapshot based on job status, but I haven't tried this with VCC, which I'm assuming is your use case, correct?

[grufo]

Hi, any progress on this?
Would like to get the same information to safely reboot the system when no jobs are running!
Isn't there an API to query the job status from the B&R?
Did you try something like "https://localhost:9398/api/jobs"?

[grufo]

Tried the API of EM and it works great!

This is my bash script to get the number of running Veeam jobs:

Code: Select all

#!/bin/bash
veeamUsername="veeamuser"
veeamPassword="veeampassword"
veeamAuth=$(echo -ne "$veeamUsername:$veeamPassword" | base64)
veeamRestServer="server-name-or-ip-address"
veeamRestPort="9398"
veeamSessionId=$(curl -X POST "https://$veeamRestServer:$veeamRestPort/api/sessionMngr/?v=latest" -H "Authorization:Basic $veeamAuth" -H "Content-Length: 0" -H "Accept: application/json" -k --silent | awk 'NR==1{sub(/^\xef\xbb\xbf/,"")}1' | jq --raw-output '.SessionId')

veeamXRestSvcSessionId=$(echo -ne "$veeamSessionId" | base64)

veeamEMUrl="https://$veeamRestServer:$veeamRestPort/api/"

veeamRunningJobs=$(curl -k --silent --location --request GET "https://$veeamRestServer:$veeamRestPort/api/reports/summary/overview?format=Entities" --header "X-RestSvcSessionId: $veeamXRestSvcSessionId" | grep -oPm1 "(?<=<RunningJobs>)[^<]+")

echo $veeamRunningJobs

I hope this helps you too!