[Enhancement Request] Ability to Forget All Missing Backups at repo level

[AlexHeylin]

Case: 05820167
It would have been good either for SOBR sync to have worked out these backups were missing AND to have told / asked to delete from config... OR ability to run Forget All Missing Backups at repo or job level, rather than have to dig through the text logs to find them then have to go to Backups > Disk> and select each VM then run Forget All Missing Backups.

If they're missing (in our case because the disk array the SOBR performance extents live on died) there's nothing we can do about it if they didn't make it to capacity tier.

Thanks

[PetrM]

Hi Alex,

Could you please elaborate on why this option does not solve the problem?

Thanks!

[AlexHeylin]

Hi PetrM,
The option you linked to removes the entire backup - with all the chains and all the RPs from the configuration. That's far more brutal than the required outcome of removing only missing RPs (and any dependent backups) from the config.

In this case the RPs went missing because the performance tier of the SOBR suffered catastrophic failure between the backup creating the RPs and the SOBR offload sending them to Capacity tier. In our case, that was one RP per VM - but depending on the SOBR and backup job settings it could be many more.

Currently the only way to do what we required was to dig through the text logs from the offload job and find the names of the missing RP files, confirm the files were missing, then for each affect VM open Backups > Disk > and select affected Job then each affect VM in the job then run Forget All Missing Backups. This is the approach Support told us to take. That approach does not scale well.

While running Remove From Configuration against every affected Job, then importing them all again would remove the missing RPs it could easily create new problems (backups losing mappings / chains) and could take significant time - during which those jobs cannot be backed up or restored. We've not tested this approach - but would only do so if support told us to, and they recommended the less brutal approach above.

[PetrM]

Hi Alex,

Apologies that I don't catch the idea but as far as I understand you need to remove the specific restore points from configuration and not the entire backup. I guess you're talking about this option? If my understanding is correct, then I have 2 more questions:
1. Why do you need to check text logs regularly?
2. For example, what would you like to add to the wizard of restore points deletion?

Thanks!

[spiritie]

This is what OP is looking for, currently there no way in the UI under "Disk" to do this.

The button in the UI called "Forget all..." only applies to that specific VM in the chain, this is super painful when there is many VM's in the job you want to remove.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[jmm]

Hi Alex and PetrM,
There is a similar discussion about "Feature request: cmdlets or documented methods for Powershell-based interaction similar to "Files" pane.".

I have similar issue (try to forget one restore point and all the backup chain is deleted).

In my organisation, all the backups repository extend are made on top of a dedup storage "VSA hpe storeonce Catalyst".
We have a primary backup respository, quite small but fast.
And using the veeam functionnality "HPE Catalyst Copy ", we copy primary backups to a secondary backup repository largest but slower.

When primary backup repository if full, we delete point directly by HPE Storeonce API on the primary repository.

And VEEAM mark that's point not avaiable.
Now we keep them like that because each time we try to "forget" them the consequences have been the lost of restore point, or the rebuild of the full backup chain in a new backup chain (folder with suffixe "_1" ?)

It will be super cool to have a magic cleaner batch routine that :
- Check integrity of the backup Chain
- Give the possibilité to "forget", deleted point, in a SOBR scenario where primary point are deleted but secondary are avaiable and not deleted by that routine ...

[spiritie]

I've just tried using this command myself, I'm unable to get this working on a backup copy job. Running it what the "-WhatIf flags on the first code says:

What if: Performing the operation "Removing unavailable backup files from configuration database... on target "My_Backup_Copy_Name".

But this running it without the "-WhatIf" and the command just finishes nearly instant and nothing happened when checking under "Disk".

Can anyone from Veeam confirm whenever this command only works on backup jobs?

I've tried 2 methods:

Code: Select all

$backup = Get-VBRBackup -Name "<NAME OF THE BACKUP>"

Remove-VBRBackupFile -Backup $backup

Code: Select all

Get-VBRBackup -Name "<NAME OF THE BACKUP>" | Get-VBRBackupFile | ?{$_.IsAvailable -eq $false} | %{
    Remove-VBRBackupFile -backupFile $_ -Confirm:$false
}

[AlexHeylin]

Hi PetrM

I understand you need to remove the specific restore points from configuration and not the entire backup.

I needed to remove all missing restore points from all backups which are causing messages in the offload jobs about missing points (which doesn't say which points / which jobs).

I guess you're talking about this option?

That's the option we had to use - but it's suboptimal for this case.

1. Why do you need to check text logs regularly?

We don't regularly - however when the offload jobs generate messages about missing points / unable to upload, the only way to find out what jobs and points is to read the text logs - or in our case open a support case and get them to read the logs and tell us.

2. For example, what would you like to add to the wizard of restore points deletion?

I think you've misunderstood - we don't want to use that per-VM-in-basckup-job wizard. We want a simple, obvious, and scalable way to have a SOBR (possibly non-SOBR repo too) forget all missing RPs. Once RPs are missing from disk it's VERY unlikely that they'll ever come back. We, possibly stupidly, expected that rescanning the SOBR would sort all this sort of thing out - not completely ignore it and not tell anyone, and leave it to mysterious messages in the offload jobs to give us a clue - then we have to dig through the text logs, then run this option? for each VM in each job. Give us something that does this in 3-5 clicks to resolve for the whole SOBR.

Thanks

Alex

[tgx]

+1

Anytime you have an admin mucking around in log files, you are doing something wrong or ineffectively.
It's a burden on the admin, and a burden on Veeam support to have to assist with its analysis.

[PetrM]

Hello,

@AlexHeylin The described scenario looks like a typical technical issue that must be investigated and resolved by our support team. You don't need to analyze the debug logs or text files: the best way is to let our engineers analyze debug logs in case of any errors with offload or any other jobs. As far as I understand, it happened just once in your environment so I'm not sure that we should build something to resolve an infrastructural issue that rarely occurs. However, we can think about implementation in our future releases as long as we have enough requests for this functionality.

@spiritie I'll ask my colleagues to check your question.

@jmm Would you be so kind to clarify why you use HPE StoreOnce API to delete restore points? We have a special retention policy mechanism that must be configured for every job, it removes the obsolete restore points. Manual deletion can work but is not supported.

Thanks!

[oleg.feoktistov]

Hi @spiritie,

Remove-VBRBackupFile should also work for backups made with backup copy jobs. Just tested it in v11a, all went fine.
Does this code return any unavailable backup files in your case?

Code: Select all

Get-VBRBackup -Name "<NAME OF THE BACKUP>" | Get-VBRBackupFile | where {$_.IsAvailable -eq $false}

Best regards,
Oleg

[colohost]

I've run into this issue after changing the repository a large backup job used, and having to cancel the move operation Veeam 12 had begun because it was going to take weeks to complete and I couldn't leave the backup job disabled for that long.

Due to my cancelling the move operation, after giving it a day and realizing it would take weeks, I ended up with a bunch of orphaned backups in an unknown state, where both those orphaned ones, and successfully moved + newly taken backups, both exist under the same job name.

I now need to delete or forget all unavailable restore points from those orphaned backups, but the Get-VBRBackupFile command fails because the argument to the -Backup parameter matches what are now duplicate job names. Normally you would:

Get-VBRBackup to get the job name
GetVBRBackupFile -Backup <job name>

but you can't do that with duplicate names, and that command won't accept a GUID you retrieve via "Get-VBRBackup | Select name, Id", so you're stuck with no way to obtain the id's you need to delete the missing files via power shell.

I can do it fine via GUI, but this job has 112 VM's in it, so clicking into each one, one at a time, and then deleting unavailable backups, is going to take forever. I should be able to just Ctrl-A the entire list of VM's and delete all unavailable restore points with one click.