About half a year ago, I set up a couple dozen Windows Server machines backing up to Wasabi S3, using agents (v6.0.2) managed by VBR (12.0.0.1420 P20230718). A few days ago, one of them started failing with the following error:
Code: Select all
29/11/2023 03:16:48 :: Error: REST API error: 'S3 error: The specified version does not exist.
Code: NoSuchVersion', error code: 404
Other: Key: 'Veeam/Backup/backups/Clients/{ed994b08-2f62-4e90-9543-71bbd7f9b951}/86a834ca-b9a7-4d8a-841e-9ff8c2099c03/CloudStg/Meta/Blocks/Checkpoints/metastore_{5d7cf0da-d68e-43d3-863b-18b90a6185f4}.{cd090eda-bd6e-48c7-992d-d9531ff97fdb}', VersionId: '001698817825431586741-IcZvLEZQ2v', HostId: 'pBlSTMm4a7WX/NGvy+vdbheUxvtMsTutHWXw4Hn0l9rIGswNQ7znWtgzdt71wpYggUKWx8VTOrAV'
Storage chain is corrupted. Some objects are missing.
Failed to load object. Object path: [/Meta/Blocks/Checkpoints/metastore_{5d7cf0da-d68e-43d3-863b-18b90a6185f4}.{cd090eda-bd6e-48c7-992d-d9531ff97fdb}], version: [001698817825431586741-IcZvLEZQ2v].
Agent failed to process method {Cloud.DeleteCheckpointsNoSession}.Rescanning the repository has done nothing. Ran a health check on the backup, and it failed with the same error, but did not actually mark any backups as invalid. Tried an active full before and after the health check - keep getting the same error. Other machines in the same job, and in other jobs managed by the same server, located on the same network, and backing up to the same bucket, are backing up as normal.