Regarding block generation/api charges

[theviking84]

So 3 times per month, we have the "full backup worth of api calls" due to immutability in aws s3. I'm trying to get a handle on the expectation of cost.

When talking about a full backup worth of apis, I'm assuming this is only regarding the short term chain and NOT any gfs points, as only the short term chain is immutable in aws s3. Correct? So meaning if my original data was say 10TB worth for fulls, but I'm keeping 3 weeks straight on short term chain, and so my size in the bucket is 14tb now.... The every 10 day api is only based on 10TB (10,000,000 api calls) right? So I could expect something like 30,000,000 api calls per month with immutability?

To get more detailed, are these 30,000,000 calls all of the category PUT,COPY,POST,LIST? (the most expensive of calls) Or are there also GET, SELECT and if so how many of those? What about "data scanned by "s3 select" api?

So far for the month of March I'm at 1/3 of the monthly cost I budgeted for which is ok as we are 1/3 through the month. But aws has the "forecast" thing where they are guessing what your monthly cost will be and it is over my budget. I am not sure how they can possibly be guessing anywhere near accurate as they don't know about your 3 times per month calls. Overall I'm using s3 standard but it seems like the API cost is almost more than the storage. it isn't, but somewhat up there.

I am just feeling like I can't get a handle on the actual pricing yet and wonder if I'm missing something Veeam is doing. If I have 14TB that is 14*$23 per tb= $322 and just grows per tb added, not bad. But api is harder to get right.

[Gostev]

Since we need to extend immutability on the existing block which are re-used in the following restore points, write operation is involved - so it must be PUT.

Size of the source backups does not matter. The number of API calls can be estimated as the number of 1MB blocks in the source (protected) data minus 20% - as with an average daily change rate, a quarter of the blocks from the previous week are not reused in the next week's backups. The bigger the daily change rate, the less objects will need to renewed, but more new object will have to be stored.

[theviking84]

Im not grasping your "size of source data minus 20%" quite yet. So if I was protecting 30TB of server vmdks but the backup took it down to 15TB, you are saying the backup size didn't matter and its 30TB minus 20% = 24TB worth so 24,000,000 calls times 3 per month so 72,000,000 calls in a single month?? Or please give an example so I can understand better.

In my case it is about 10TB was the intial full worth in s3 bucket, and I'm up to 14.4TB now after a few weeks worth of retention. My immutability is set at 4 which = plus 10 for block gen so 14 days.

[Gostev]

Yes, what your wrote above is the correct estimation. Backups size don't matter because eventually, we operate on objects which represent source disk blocks, so the total number of those blocks is all that matters. Of course, this estimation is a "worst case" scenario as there are some optimizations, for example zeroed source blocks can be discarded from estimations.

[theviking84]

Thank you! At least I have a more full grasp now. A little disheartening as AWS really charges a lot for this many api calls every month. In my example and using the AWS price calculator tool right on AWS website, the api costs are only $110 less per month than the storage if I just were to put in 20TB and 72,000,000 api calls.

So one more point on this... At the very least, I'm expecting since the immutability period is only 4+10=14 days, I am not going to expect the api calls to dramatically keep increasing as longer term retention comes into play? Like still only the short term simple retention chain in object storage will be used here right? Or would weekly/monthly/yearly gfs as they are all added, end up still making those 72,000,000 calls go way up higher and higher?

[Gostev]

No. The only thing matters for the Capacity Tier immutability processing costs is the size of the source data.