S3 errors: identification and troubleshooting

[evgenich]

Hi everyone,
Using Minio S3 as backup repo
When running backup jobs, 5 VMs, 3 disks on each.
Total backup size 1,4 TB


Everything seems to be working ok, but when trying stress tests receive such kind of errors:
1. For several VMs - runs ok, for others: (rerun helps)
Error: No cloud backup session was found for backup ID 'ec7c3046-fe01-43af-b507-a9a75d178cb1'Agent failed to process method {Cloud.StopBackupSession}.

2. Failed to delete backup Error: S3 delete multiple objects request failed to delete object [Veeam/Backup/backup/Clients/{8fd5a398-4a4d-4ad8-9e00-2c44187d6059}/94b8f54f-5208-45c8-b6bc-b7273c203fbf/CloudStg/Meta.Copy.0/Blocks/Checkpoints/metastore_{c26ee82b-dc5c-404d-904d-5feac570efbd}.{b21f1fa9-d809-4995-980d-010499f8fdc8}] and [0] others, error: IncompleteBody, message: 'You did not provide the number of bytes specified by the Content-Length HTTP header.'
Agent failed to process method {Cloud.DeleteBackup}.

3. Error [Bench_5VM_256KB_new - dummy_vm_06] Failed to delete backup Error: HTTP exception: WinHttpReceiveResponse: 12002: The operation timed out

4. Any way to monitor open concurrent tasks to repo?

Didnt find proper docu for such errors. Any suggestions how to identify reasons for that?
Thank you

[david.domask]

Hi Evgenii,

For items 1-3, these are technical issues best reviewed by Veeam Support. 3 looks a bit like just connectivity issue, but best to let the Support Team take a look at the logs.

Please open a case and include logs from the Veeam Backup and Replication Server (select the 1st radio option and select the affected jobs for exporting logs).

For #4, would the solution from this thread work for you? It's interactive, but it will be nice fast look at active tasks on the Veeam server.

[evgenich]

Ok, thank you. Will open case for that

[sfirmes]

Evgenii, If you post the support case# or PM me with it, I would like to look into the root cause since I work closely with MinIO.

Thanks.

Steve

[dabrigo]

Hi all,

i'm getting "You did not provide the number of bytes specified by the Content-Length HTTP header" errors too sometimes while offloading to MinIO, can you please share the findings after your analysis?

Thanks,
Davide

[david.domask]

Hi Davide,

It's best to open a Support Case to check on the issues. The topic creator did not share a case number so cannot check what was the result on theirs, but for such issues a Support Case is always a best first start.