Backup to S3 - ListObjects/DeleteObjects operations

[allenwliu]

Hi,
I am using Veeam 11 to backup to an S3 cloud. It seems that when Veeam expires backups on cloud, there are many ListObjects/DeleteObjects operations. I am just wonder how Veeam do the jobs when expires backups on cloud and if there are ways to reduce these operations or there are any controls.
Thanks,
Allen Liu

[HannesK]

Hello,
and welcome to the forums.

The first step would be to use V12, because we optimized things there. Second I refer to the sticky forum FAQ

Reducing API calls would increase storage costs... you can try, but I would go for a storage provider without API call costs.

Best regards,
Hannes

[allenwliu]

Hi Hannes,
Thank you for quick response. We are a cloud provider without API call charges. Some customers complains about that too many API calls result in performance impact. I went through forum FAQ as you pointed out. Could you tell a little more specific what method to use to reduce API calls?
Thanks,
Allen Liu

[Gostev]

Increasing block size reduces API calls proportionally, however it also increases the storage usage. For example, using 4MB block size instead of the default 1MB will reduce API calls by 4x but increase storage usage by 2x. And because of the latter, it almost never make sense to implement this change. The default 1MB block size is really the most optimal.

Perhaps you should clarify, what actual issue are you trying to solve through reducing the number of calls? Since storage devices' only job is to handle I/O requests, it's not something you should worry about normally?

[orb]

Thanks for the info,

So, is this overhead a consistent 50%, tied directly to the object size?
I'm curious, does this overhead space store metadata just for Veeam, or S3, or maybe both?

I'm guessing this counts towards space usage, not the actual backup size.

Oli

[Gostev]

Correct, it is consistent whether you use block, file or object storage.

This has nothing to deal with the storage of metadata though, and rather with the physics of block-based, image-level incremental backup: the bigger the block size is, the more unchanged data will have to be captured and stored in an incremental backup.