Discussions related to using object storage as a backup target.
Post Reply
bit_inspector
Novice
Posts: 5
Liked: never
Joined: Mar 26, 2020 12:40 pm
Full Name: Thomas
Contact:

AWS S3 Infrequent Access: how to reduce the number of API calls

Post by bit_inspector »

Support ID = 04345915

Hello,

we are using AWS S3 Infrequent Access with the immutable feature to copy/offload backups from a Scale Out Repository.

Unfortunately we did not calculate the costs for PUT, COPY, POST, LIST when estimating the price for AWS S3 and our budget is heavily overdrawn.

We initially started at the end of July and as of today the costs for requests for roughly 35 TB Backups accumulated to:

July:
37,613,594 Requests (Amazon Simple Storage Service EUC1-Requests-SIA-Tier1) = 376,14 USD

August:
79,901,319 Requests (Amazon Simple Storage Service EUC1-Requests-SIA-Tier1) = 799,01 USD
84,534,867 Requests (Amazon Simple Storage Service EUC1-Requests-Tier1) = 456,49 USD

- The immutable feature is set to 30 days
- The backups are encrypted
- We run synthetic fulls once a week on ReFS
- Backups older than 8 days are offloaded
- When the repository reaches 80% capacity backups are offload by "override"
- Retention is set to 30 days
- Weekly health checks
- The jobs have the setting "LAN target"

My questions:

- Why are we at 160.000.000 Requests in August, when the Rule of thumb is 1 TB Backups = 1.000.000 Requests? Shouldn't it be more in the 35 to 40 Million region (35 TB of Backups)?
- Why are there 84 Million Request on S3 Standard when the Bucket is configured as IA via Veeam?

- How can we significantly reduce the number of request made by Veeam to cut costs? We need to reduce the requests as much as possible.
- Would it help to switch to incremental forever instead of synthetic fulls once a week?
- Would it help to disable encryption in Veeam?
- Would it help to do less frequent health checks, let's say every 2 or 4 weeks instead of weekly?
- Would it help to lower the retention period from 4 to 2 weeks?

Thank you
Thomas
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: AWS S3: how to reduce the number of API calls

Post by HannesK »

Hello,
and welcome to the forums.

There are multiple things to take into account.

1) you doubled the API costs by going away from the default settings. Please see FAQ about you LAN target setting. post338749.html#p338749
2) Immutability costs around one full backup in API costs every 10 days

If you really want to reduce costs, then I recommend going to a cloud provider, that has a user friendly pricing model :-)

Question 2-4: no, that does not change anything from a cost perspective

Question 5: only for disk space. Keep in mind, to also reduce immutability time. (well, if you stay at infrequent access, then there are extra charges for early deletion)

Best regards,
Hannes
bit_inspector
Novice
Posts: 5
Liked: never
Joined: Mar 26, 2020 12:40 pm
Full Name: Thomas
Contact:

Re: AWS S3: how to reduce the number of API calls

Post by bit_inspector »

Hi Hannes,

thank you for your reply.

So "Local Target" should be set instead of "LAN Target", do I understand that right?

But I still don't understand why Standard S3 API costs are charged, when the bucket is set to S3 IA via Veeam?

And how would you explain the total number of requests, 160 million. Even if I set "Local Target", it still would be 80 million request instead of the expected 30 to 40 million (1 TB = 1 million requests)?

Q5: OK, so switching to Standard S3 would cut the API cost bei 50% per the AWS price list. The storage costs will be higher, but in sum we would save money. Regarding the immutability time, what is a recommended time frame by Veeam? I think one must consider, that attacks are sophisticated today, and therefor a couple of weeks are not a bad idea (plus deploying a good IDS/IPS).

Thank you
Thomas
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: AWS S3: how to reduce the number of API calls

Post by HannesK »

Hello,
actually I have no idea about the Standard S3 API costs for S3 IA. I don't run that setup.

Yes, one MByte source data = 1 PUT request with Local target (see FAQ). Going to larger blocks would reduce API costs, but increase disk space costs. The defaults are fine form a cost perspective.

Don't forget about the API costs for a full backup every 10 days for immutability. Whether you configure 3 days or 30 days... every 10 days we need to make sure, that all relevant blocks are still immutable. This operation is already cost optimized. The alternative would be to use more disk space (=more costs). If you have 30 days retention, then I would protect around half of it, maybe 14 days. But that's just personal preference.

Keep in mind, that we protect the configured days + 10 to avoid API costs (the user guide explains it some kind of, but I would not bother too much about the details).

Best regards,
Hannes
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: AWS S3: how to reduce the number of API calls

Post by Gostev » 2 people like this post

Please note that S3 IA is specifically recommended for offloading GFS fulls only (in other words, using the Move policy with a large window). There's a note about this right in the UI under the corresponding check box. Using S3 IA with the Copy policy, which processes every single incremental backup, and with immutability on top will definitely be a huge money drain due to much higher API costs for this S3 tier.

As you can see from your numbers, even with the default block size setting of 1MB (twice larger than your current one), your API costs will still exceed storage costs.

The only way to make S3 IA API costs somewhat reasonable with the Copy policy and immutability enabled is to use the largest 4MB block size in Veeam. However, it will increase your storage consumption due to much larger increments, and also make granular restores directly from object storage slower and more expensive.

All in all, using regular S3 and the default block size usually provides the best balance for most use cases.

Please keep in mind that switching the block size requires an active full backup to be performed.
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: AWS S3: how to reduce the number of API calls

Post by MrSpock »

Hi,

The major part of our AWS S3 costs does also come from API calls. Take a look at our usage for the last month. The purple parts are API calls. These consist of 31 % PutObject operations and 69 % WriteObjectLockRetentionInfo operations. The amount of PutObject is about the same every day, but WriteObjectLockRetentionInfo is peaking on a few days as you can see in the chart.

I have 3 restore points that are COPIED to AWS S3 with standard storage class and immutability for 5 days.

Is there a way to decrease the amount of WriteObjectLockRetentionInfo operations during the peaking days or is there a way to perform these operations less often?

Image

Thanks,

Johan
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by HannesK »

Hello,
yes SOBRArchiveImmutabilityGenerationDays (0-30 is possible. 10 is default which I mentioned earlier) does that. But that will increase storage costs. So I'm not sure, whether you really want to change that value...

Best regards,
Hannes
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by MrSpock »

HannesK,

Does that mean that retention information will be changed on all objects each 0-30 days instead of each 10 days? Maybe I should experiment a bit with that value as API costs are much higher than storage costs today.

I found no information about SOBRArchiveImmutabilityGenerationDays in the forums or anywhere else. Is it just a DWORD value in the "Veeam Backup and Replication" registry section on the backup server?

Thanks,

Johan
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by Gostev » 1 person likes this post

Yes, all registry values go there.
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by MrSpock »

Hello,

Has SOBRArchiveImmutabilityGenerationDays been removed in VBR 11? The value seems to be ignored after the upgrade and WriteObjectLockRetentionInfo is now occurring every 10 days again. The registry value is set to 30.

Regards,

Johan
sfirmes
Veeam Software
Posts: 224
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by sfirmes » 3 people like this post

In Veeam Backup & Replication v11 the registry key was renamed to be SOBRCapacityImmutabilityGenerationDays

Thanks
Steve
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
gummett
Veteran
Posts: 404
Liked: 106 times
Joined: Jan 30, 2017 9:23 am
Full Name: Ed Gummett
Location: Manchester, United Kingdom
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by gummett » 1 person likes this post

bit_inspector wrote: Aug 20, 2020 7:14 pm - Why are there 84 Million Request on S3 Standard when the Bucket is configured as IA via Veeam?
@bit_inspector We have very recently had the following confirmation from AWS about this behaviour:
When using the APIs directly for Object Lock operations like PutObjectLockRetention (reported as WriteObjectLockRetentionInfo in AWS Cost Explorer) and PutObjectLegalHold those are charged at Tier1 rates regardless of the storage class of the underlying object
This is excellent news as it keeps the cost of using immutability with IA to a minimum
Ed Gummett (VMCA)
Senior Specialist Solutions Architect, Storage Technologies, AWS
(Senior Systems Engineer, Veeam Software, 2018-2021)
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by stewsie »

HI

Not using S3 IA but S3 Standard to offload backups for immutability

I had configured the Veeam jobs to use LAN and not Local storage. I have since changed this back after reading veeam-backup-replication-f2/read-this-f ... ml#p338749

I am in contact with an AWS consultant who is surprised the Veeam jobs are generating so many Put requests

Is this expected? The Put requests are generating the highest charges

The PUT requests last week were 39,315,673.000 Requests $208.37 whereas the GB price was $39.13

This at the time would have been for around 9TB worth of data split over 4 backup jobs with multiple VMs.

As yet I haven't contacted Veeam support but can do
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by Gostev » 3 people like this post

This is due to offloading backups which used a small block size. Small blocks size means many more objects for the same data size, and each object needs an API call to be written.

S3 IA is a very tricky storage class due to high costs for just about everything else offsetting slightly lower storage costs. So for anyone determined to use S3 IA, which honestly rarely makes sense for anything other than oldest GFS backups offload (which in turn are better off sent to Archive Tier anyway), you should consider using large block size for your backups.

For S3 Standard, the default block size usually provides the best balance between API and storage costs. Using large blocks may both increase and reduce your overall bill, this depends solely on the type of protected data. Because some workload types may results in very large incremental backups when using a large block size (if they create large number of small changes across the entire disk).

Just remember that changing block size requires an active full backup and then re-uploading all data again. In your case I would just start with object storage from scratch (new bucket), otherwise you will keep wasting API calls on retention processing of those many small blocks you have already created.

Overall, S3 API call charges is a sad concept and is the reason why more and more customers are switching to modern cloud object storage providers like Wasabi who don't charge for S3 API calls at all. Because conceptually it's the same as if on-prem block storage vendors were to additionally charge for SCSI commands or I/O operations issued, thus adding a whole new layer of complexity into the storage planning. This makes no sense to me and on-prem object storage vendors are not doing it either.
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by stewsie »

Thanks Gostev for the excellent response. I wasn't aware that we were able to use other providers where immutable backups are a requirement. Is Wasabi using S3 Compatible storage?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by veremin » 1 person likes this post

Correct, Wasabi is fully compatible AWS S3 storage (that besides everything else does support object lock functionality). Thanks!
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: AWS S3 Infrequent Access: how to reduce the number of API calls

Post by Gostev » 1 person likes this post

There are other vendors like that too. There's the dedicated "Compatible + Immutability" section for them in the sticky topic on this forum.
Post Reply

Who is online

Users browsing this forum: No registered users and 11 guests