Can't deleted backups that were partially offloaded to a now removed capacity tier

[BrianBuchanan]

Hello,

I've opened a couple of tickets (Case #07167944 and #07237136) on this issue for two different B&R servers where we had to delete the capacity tier of a SOBR.

Both servers had SOBR with capacity tiers, but we started having problems with the offload and we ended up deleting the capacity tiers. The root cause was a weird ISP issue that was making uploads from Windows really slow (running Google Internet Speed Test on a windows computer gave results around 3-Mbps but on a linux desktop it gave 800+. Speedtest.net was always 800+ Mbps regardless of the O/S) and it took a year to finally be resolved.

However I'm now left with outdated backups of VMs that no longer exist that can't be deleted. Any attempt, manually or when the backup job's "delete outdated backup" step runs, results in "sequence contains no elements". I think this message is relating to references to the old capacity tier.

I found one workaround on the first server; because our upload issues were resolved we added a new azure capacity tier (didn't reuse the old one) and the offload ran fine, including offloading these old outdated backups. We had also enabled immutability and when the immutability period finally ended the backups were deleted just fine during the "delete outdated backups" step.

On the second server I'd rather not have to add a capacity tier to the SOBR and then wait out the retention period just to delete these old backups. The VMs are long gone, beyond our retention period already but it seems like adding a capacity tier and letting it offload is the only straightforward fix? I don't want to delete the entire backup, or forget it from the configuration, just the VMs that are long gone.

Hmm, I might have just found another option. I moved the VM's backup to another job, which both failed and succeeded. The "Move Backup" job failed overall with the same "Sequence contains no elements", but the disk backup actually did get moved and I was able to delete it afterwards. I'll add this to the ticket.

[david.domask]

Hi Brian,

Thanks for sharing the previous case numbers and your intended workarounds; such workarounds should not be necessary, and I'd actually ask you open a new support case and share the case number here.

Sequence contains no elements errors basically means that we tried to retrieve something from the Veeam Configuration Database and did not get what we expected, and usually this requires a review of the Veeam Configuration database. I cannot tell right now why moving the backups seemed to help (despite the failures) but it should be discernible from the logs.

I don't think your workaround is dangerous, but it's a workaround you shouldn't need, so let's get one more case open to review this behavior more closely.