I just found out capacity tier offload of one of my backup jobs has been failing for several days now. The SOBR has a one performance and one capacity tier (backblaze) and has been working fine for a long time. A month ago I changed the capacity tier to use a bucket with object lock when B2 acquired immutability support. Now this backup chain seems to have a reached a stage where it has to delete some files in the object storage which errors with a "DeleteMultipleObjects" failed message (see below).
Immutability is set to 30 days in the B&R object repository and backups are copied to capacity tier as they are created. "keep only the last version" is enabled on the bucket at backblaze end as was the recommended practice prior to immutability support in B2. Has that changed? The error (below) refers to some invalid version id.
Wonder why no email notification is received for this error. In the B&R console selecting "Jobs" and "HISTORY" shows no error (all jobs successful) but selecting "Storage Management" shows the offload errors, so I failed to notice this for several days.
Error seen in the statistics window:
Code: Select all
Starting performance tier offload
0 backups will be moved to the capacity tier
12 backups will be copied to the capacity tier
Processing Alt Backup VMs Error: DeleteMultipleObjects request failed to delete object [Veeam/Archive/Veeam/2e..elided...9778ff/00000000-0000-0000-0000-000000000000/blocks/5d0b...elided...1a4c/24928....elided...9054c.00000000000000000000000000000000.blk] and [23] others, error: NoSuchVersion, message: 'Invalid version id specified' 01:19
Object storage repository cleanup 00:12
Job finished with error at 12/20/2020 8:10:43 PM