AWS S3 Infrequent Access: how to reduce the number of API calls

[bit_inspector]

Support ID = 04345915

Hello,

we are using AWS S3 Infrequent Access with the immutable feature to copy/offload backups from a Scale Out Repository.

Unfortunately we did not calculate the costs for PUT, COPY, POST, LIST when estimating the price for AWS S3 and our budget is heavily overdrawn.

We initially started at the end of July and as of today the costs for requests for roughly 35 TB Backups accumulated to:

July:
37,613,594 Requests (Amazon Simple Storage Service EUC1-Requests-SIA-Tier1) = 376,14 USD

August:
79,901,319 Requests (Amazon Simple Storage Service EUC1-Requests-SIA-Tier1) = 799,01 USD
84,534,867 Requests (Amazon Simple Storage Service EUC1-Requests-Tier1) = 456,49 USD

- The immutable feature is set to 30 days
- The backups are encrypted
- We run synthetic fulls once a week on ReFS
- Backups older than 8 days are offloaded
- When the repository reaches 80% capacity backups are offload by "override"
- Retention is set to 30 days
- Weekly health checks
- The jobs have the setting "LAN target"

My questions:

- Why are we at 160.000.000 Requests in August, when the Rule of thumb is 1 TB Backups = 1.000.000 Requests? Shouldn't it be more in the 35 to 40 Million region (35 TB of Backups)?
- Why are there 84 Million Request on S3 Standard when the Bucket is configured as IA via Veeam?

- How can we significantly reduce the number of request made by Veeam to cut costs? We need to reduce the requests as much as possible.
- Would it help to switch to incremental forever instead of synthetic fulls once a week?
- Would it help to disable encryption in Veeam?
- Would it help to do less frequent health checks, let's say every 2 or 4 weeks instead of weekly?
- Would it help to lower the retention period from 4 to 2 weeks?

Thank you
Thomas

[HannesK]

Hello,
and welcome to the forums.

There are multiple things to take into account.

1) you doubled the API costs by going away from the default settings. Please see FAQ about you LAN target setting. post338749.html#p338749
2) Immutability costs around one full backup in API costs every 10 days

If you really want to reduce costs, then I recommend going to a cloud provider, that has a user friendly pricing model

Question 2-4: no, that does not change anything from a cost perspective

Question 5: only for disk space. Keep in mind, to also reduce immutability time. (well, if you stay at infrequent access, then there are extra charges for early deletion)

Best regards,
Hannes

[bit_inspector]

Hi Hannes,

thank you for your reply.

So "Local Target" should be set instead of "LAN Target", do I understand that right?

But I still don't understand why Standard S3 API costs are charged, when the bucket is set to S3 IA via Veeam?

And how would you explain the total number of requests, 160 million. Even if I set "Local Target", it still would be 80 million request instead of the expected 30 to 40 million (1 TB = 1 million requests)?

Q5: OK, so switching to Standard S3 would cut the API cost bei 50% per the AWS price list. The storage costs will be higher, but in sum we would save money. Regarding the immutability time, what is a recommended time frame by Veeam? I think one must consider, that attacks are sophisticated today, and therefor a couple of weeks are not a bad idea (plus deploying a good IDS/IPS).

Thank you
Thomas

[HannesK]

Hello,
actually I have no idea about the Standard S3 API costs for S3 IA. I don't run that setup.

Yes, one MByte source data = 1 PUT request with Local target (see FAQ). Going to larger blocks would reduce API costs, but increase disk space costs. The defaults are fine form a cost perspective.

Don't forget about the API costs for a full backup every 10 days for immutability. Whether you configure 3 days or 30 days... every 10 days we need to make sure, that all relevant blocks are still immutable. This operation is already cost optimized. The alternative would be to use more disk space (=more costs). If you have 30 days retention, then I would protect around half of it, maybe 14 days. But that's just personal preference.

Keep in mind, that we protect the configured days + 10 to avoid API costs (the user guide explains it some kind of, but I would not bother too much about the details).

Best regards,
Hannes

[Gostev]

Please note that S3 IA is specifically recommended for offloading GFS fulls only (in other words, using the Move policy with a large window). There's a note about this right in the UI under the corresponding check box. Using S3 IA with the Copy policy, which processes every single incremental backup, and with immutability on top will definitely be a huge money drain due to much higher API costs for this S3 tier.

As you can see from your numbers, even with the default block size setting of 1MB (twice larger than your current one), your API costs will still exceed storage costs.

The only way to make S3 IA API costs somewhat reasonable with the Copy policy and immutability enabled is to use the largest 4MB block size in Veeam. However, it will increase your storage consumption due to much larger increments, and also make granular restores directly from object storage slower and more expensive.

All in all, using regular S3 and the default block size usually provides the best balance for most use cases.

Please keep in mind that switching the block size requires an active full backup to be performed.

[MrSpock]

Hi,

The major part of our AWS S3 costs does also come from API calls. Take a look at our usage for the last month. The purple parts are API calls. These consist of 31 % PutObject operations and 69 % WriteObjectLockRetentionInfo operations. The amount of PutObject is about the same every day, but WriteObjectLockRetentionInfo is peaking on a few days as you can see in the chart.

I have 3 restore points that are COPIED to AWS S3 with standard storage class and immutability for 5 days.

Is there a way to decrease the amount of WriteObjectLockRetentionInfo operations during the peaking days or is there a way to perform these operations less often?



Thanks,

Johan

[HannesK]

Hello,
yes SOBRArchiveImmutabilityGenerationDays (0-30 is possible. 10 is default which I mentioned earlier) does that. But that will increase storage costs. So I'm not sure, whether you really want to change that value...

Best regards,
Hannes

[MrSpock]

HannesK,

Does that mean that retention information will be changed on all objects each 0-30 days instead of each 10 days? Maybe I should experiment a bit with that value as API costs are much higher than storage costs today.

I found no information about SOBRArchiveImmutabilityGenerationDays in the forums or anywhere else. Is it just a DWORD value in the "Veeam Backup and Replication" registry section on the backup server?

Thanks,

Johan

[Gostev]

Yes, all registry values go there.

[MrSpock]

Hello,

Has SOBRArchiveImmutabilityGenerationDays been removed in VBR 11? The value seems to be ignored after the upgrade and WriteObjectLockRetentionInfo is now occurring every 10 days again. The registry value is set to 30.

Regards,

Johan

[sfirmes]

In Veeam Backup & Replication v11 the registry key was renamed to be SOBRCapacityImmutabilityGenerationDays

Thanks
Steve

[gummett]

- Why are there 84 Million Request on S3 Standard when the Bucket is configured as IA via Veeam?

@bit_inspector We have very recently had the following confirmation from AWS about this behaviour:

When using the APIs directly for Object Lock operations like PutObjectLockRetention (reported as WriteObjectLockRetentionInfo in AWS Cost Explorer) and PutObjectLegalHold those are charged at Tier1 rates regardless of the storage class of the underlying object

This is excellent news as it keeps the cost of using immutability with IA to a minimum

[stewsie]

HI

Not using S3 IA but S3 Standard to offload backups for immutability

I had configured the Veeam jobs to use LAN and not Local storage. I have since changed this back after reading veeam-backup-replication-f2/read-this-f ... ml#p338749

I am in contact with an AWS consultant who is surprised the Veeam jobs are generating so many Put requests

Is this expected? The Put requests are generating the highest charges

The PUT requests last week were 39,315,673.000 Requests $208.37 whereas the GB price was $39.13

This at the time would have been for around 9TB worth of data split over 4 backup jobs with multiple VMs.

As yet I haven't contacted Veeam support but can do

[Gostev]

This is due to offloading backups which used a small block size. Small blocks size means many more objects for the same data size, and each object needs an API call to be written.

S3 IA is a very tricky storage class due to high costs for just about everything else offsetting slightly lower storage costs. So for anyone determined to use S3 IA, which honestly rarely makes sense for anything other than oldest GFS backups offload (which in turn are better off sent to Archive Tier anyway), you should consider using large block size for your backups.

For S3 Standard, the default block size usually provides the best balance between API and storage costs. Using large blocks may both increase and reduce your overall bill, this depends solely on the type of protected data. Because some workload types may results in very large incremental backups when using a large block size (if they create large number of small changes across the entire disk).

Just remember that changing block size requires an active full backup and then re-uploading all data again. In your case I would just start with object storage from scratch (new bucket), otherwise you will keep wasting API calls on retention processing of those many small blocks you have already created.

Overall, S3 API call charges is a sad concept and is the reason why more and more customers are switching to modern cloud object storage providers like Wasabi who don't charge for S3 API calls at all. Because conceptually it's the same as if on-prem block storage vendors were to additionally charge for SCSI commands or I/O operations issued, thus adding a whole new layer of complexity into the storage planning. This makes no sense to me and on-prem object storage vendors are not doing it either.

[stewsie]

Thanks Gostev for the excellent response. I wasn't aware that we were able to use other providers where immutable backups are a requirement. Is Wasabi using S3 Compatible storage?

[veremin]

Correct, Wasabi is fully compatible AWS S3 storage (that besides everything else does support object lock functionality). Thanks!

[Gostev]

There are other vendors like that too. There's the dedicated "Compatible + Immutability" section for them in the sticky topic on this forum.

[sarfaraz_k]

The number of requests you’re seeing in August (160 million) does seem high given the rule of thumb that suggests 1 TB of backups equals around 1 million requests. This discrepancy could arise from several factors, including Veeam’s configuration and how it interacts with S3.

To reduce requests and cut costs, consider switching to an incremental forever backup strategy, which minimizes full backup requests. Disabling encryption in Veeam could also help, although it’s essential to weigh the security implications.

Reducing the frequency of health checks to every 2 or 4 weeks instead of weekly, and lowering the retention period from 4 weeks to 2, can further decrease the number of requests.

[Gostev]

That is actually not correct to the most part. It seems AI have been used to generate this post? Because SOBR offload to object storage is always incremental forever, there are no alternatives. While leveraging encryption has no impact on the number of API calls.