IBM object Storage 503 errors

[evandrosp]

Hi there,

Can anyone help me with? Case #07392627

Today, our main problems are the number of delete requests Veeam sends to object storage during backup jobs.
Veeam is sending too many delete requests to object storage buckets, some days it hits millions. The error code for these delete requests is 503.

Additionally, we have many machines that take too long to start the backups, and the backup process takes too much time on the "Storage initialized" step.
In the last backup window, we had VMs that stayed in the "Storage initialized" process for 2 to 6 hours just to start the backup. As we have a backup window to run the backups, many backups failed because there wasn't enough time to complete them, as a single VM can take up to 6 hours just to start moving the backup to the bucket.

The backups are sent directly to a bucket on IBM object storage.
We have a bucket for each job on VBR. The amount of data is large, ranging from 10TB to 50TB per day.

Thanks,

[Gostev]

Hi, sounds like your chosen object storage platform just can't keep up with the load. You should contact the vendor for recommendations, perhaps adding extra nodes to the cluster will help if bottleneck is compute resources? Otherwise my only other recommendation would be to consider some less problematic alternative. Thanks

[sfirmes]

Thanks for posting the support case so that we can look into it.

Are you using VBR v12.2 or v12.3? In this post S3-integrated Storage - Block Size the benefits that those versions of VBR provide, especially with object storage deletion processing are explained.

Hope this helps.

[evandrosp]

Hi there,

@gostev, yes, we are working with IBM Cloud to identify the bottlenecks and follow their recommendations.

@sfirmes, the VBR version is 12.3.0.310 , i will take a look into the post.

I am just working around and trying to find a solution.

Thanks

[sfirmes]

Evandro,

I see from the support case notes that you are working with our support to resolve this issue. I will continue to monitor the progress and hopefully a resolution is found soon. Thanks for reaching out.