Feature Request/Improvements for Rotating USB Backups

[JasonM]

Hi All,

I wanted to put forward some feature requests which I believe would improve how Veeam handles USB Rotating Hard Drives. I have customers who want to have their physical backup data in their hands and as a result rotate multiple USB drives on a weekly interval for Full Offsite/DR purposes.

Configuring the job inside Veeam and attempting to meet the customers expectations, I've ran into a few challenges and I feel there could be some small changes made to improve the process.

For reference, customers requirement:
Backup VM's to Backup Server/Local Secondary Storage daily with a set retention period
Copy latest full backup every weekend to Rotating USB Hard Drives
Notifications for backup jobs (Success/Failure)

I won't go into the details around Veeam 9.5 which is where I started as Veeam 10 is the future.

The Veeam job has been configured as a Backup Copy Job:
Immediate copy (mirroring)
Restore points: 2 (Minimum available)
RPO Monitor (Alert me if newly created backup is not copied within): 7 days
Schedule: 6AM Saturday - Monday 6AM

The reason I use a schedule window is I want to minimize/avoid potential corruption when the customer rotates the hard drive (i.e. don't write any data at a time they may be rotating the drives)
the "Immediate Copy" + "RPO Monitor" was needed to avoid "Failure" notifications being generated when there was no backup job run within the 24 hour schedule window that "Periodic copy" seems to use.

The above settings almost get the job done however the additional issues are:

USB Drive Space Issues:
I've found if the USB drive hasn't been rotated for a few weeks and the customer has a large backup job, the subsequent merge jobs Veeam seems to do can cause the USB drive to run out of free space.

To mitigate this, I've had to implement a Powershell script that removes all data under the repository before the job runs, this forces Veeam to copy just the latest backups.

Feature Request #1: I think it would help if there was an option to Erase data from the configured Job Repository before the job runs, erasing data from the repository only (Rather than full drive format) allows additional files to be retained outside the repository folder if you have other recovery media on the drive.

Backup Notification Emails:
Backup e-mail notifications seem to generate for each sub-job/task, i.e. if the Backup Copy Job runs 5 merge sub-jobs it generates 5 notification e-mails. Additionally the e-mail notifications often have minimal information which makes it harder to understand what the notifications are for from the e-mail alone.

Feature Request #2: I would like to see a feature where the e-mail notifications are once per job. I expect Veeam understands the sub-jobs are linked to the main job so why can it not collate all the information into a singular notification E-mail showing the full chain history and whether the overall job was Successful or Failed? The customer has raised this grievance multiple times as they get confused when there are 5 different e-mails generated, they just want peace of mind themselves via a singular e-mail that jobs configured are running successfully.

[PetrM]

Hi Jason,

Thanks a lot for sharing your ideas with us!

1) USB Drive Space issue:

to Erase data from the configured Job Repository before the job runs, erasing data from the repository only

I'm not sure this is a reliable way of handling such cases because it's a bit "dangerous" to erase all data automatically just being based on amount of free space: you will have no restore points if job run is failed by some reason. Moreover, such algorithm might be in conflict with retention policy settings: obsolete data only can be deleted automatically.
Maybe you could take a look at this tool and that simulator to have better estimations of required free space in future?

2) Email notifications:

I would like to see a feature where the e-mail notifications are once per job.

It works as you ask for the periodic copy mode but not for the immediate one. We can think about an implementation of a single notification for the whole job if we have enough similar requests.

Thanks!

[JasonM]

Hi Petr,

I'm not sure this is a reliable way of handling such cases because it's a bit "dangerous" to erase all data automatically just being based on amount of free space: you will have no restore points if job run is failed by some reason. Moreover, such algorithm might be in conflict with retention policy settings: obsolete data only can be deleted automatically.
Maybe you could take a look at this tool and that simulator to have better estimations of required free space in future?

I understand the concern and comments, however the customers feedback and the general understanding is:

You should have a successful backup on the USB drive you have just removed (If not sometimes you run a manual job or leave the drive plugged in for a further week).
The new drive they have plugged in has previous backup data that is deemed old and their understanding is that any old backups will be removed on the drive.
If the backup job fails for whatever reason the understanding is the associated USB drive will not have a healthy/valid backup.

Maybe what the customer is really looking for is "Retention Points" 1 (Where at the moment Veeam forces a minimum 2). Or even just bypassing retention points for this scenario completely. That may allow Veeam to look at retention policies, realize the existing data is expired and then remove it as required?

Some customer comments along the way have been:
They know their full backup is under 5TB and want to use a 5TB external drive as it is a smaller form factor and also does not require external power.
The script we've now put in place at a number of sites was communicated to the customer and they agreed removing all data prior to backup was fine as they just want latest full backup on the external drive.

[soncscy]

Hey Jason,

Just my (probably unwanted) $.02:

>They know their full backup is under 5TB and want to use a 5TB external drive as it is a smaller form factor and also does not require external power.

Your customer is being foolish and, having had this conversation with a lot of my customers, if they're not going to listen to reason, they're better off dropped imo. It's a legal liability for you waiting to happen.

Disconnecting a proper repository is most definitely a smarter/more efficient option than off-the-shelf externals from Target. (I'm being a little facetious obviously) This is a red flag for me for customers, and if I had a long-term customer that was otherwise reliable, I'd force a special SLA for such a situation. Such drives are inherently unreliable, and I'd never trust my backup on such a thing. I get the idea they're going for, but they're just making a terrible decision to save a few (hundred?) bucks. Are you US based? Surely their coronabux haven't run out yet and they can make a real and appropriate backup repo

[PetrM]

That may allow Veeam to look at retention policies, realize the existing data is expired and then remove it as required?

The retention policy check is performed in the end of backup job session, in fact it's the same approach as we use with standard repositories.

they agreed removing all data prior to backup was fine

From my point of view this method is not reliable because it's not clear what will be the action plan if backup is failed by some reason and old restore points are removed by script.

Thanks!

[foggy]

Isn't the ForceDeleteBackupFiles registry value what they are basically looking for?

[dwrandolph]

Could their desire for external USB disks (while definitely not Enterprise grade) be to allow GFS rotations to offsite locations? Disconnecting from a repository does take care of air-gaping, but not physical separation.