Discussions related to using object storage as a backup target.
Post Reply
Sebastian Kulis
Novice
Posts: 3
Liked: never
Joined: Jan 31, 2019 10:00 am
Full Name: Sebastian Kuliś
Contact:

S3 requests calculator

Post by Sebastian Kulis »

Many S3 calcs and pricings include, beside of storage price a PUT/COPY/POST/LIST Requests and GET/SELECT and Other Requests. I am assuming that these are not major costs but I want to predict all costs.
Whats these values are or how to calculate them per 1 job i.e.: backup job or restore vbk from cloud to on-premise.
Gostev
Chief Product Officer
Posts: 31606
Liked: 7097 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3 requests calculator

Post by Gostev »

It does became largely unnoticeable if you do long term retention in object storage. However, if you are only storing data for a few months, it can in fact become a very noticeable part of your bill.

Number of API calls can roughly be estimated as the number of non-zeroed 1MB blocks in source images. For example, if your VMs take up 1TB of disk space total, then initial offload will require 1 million API calls. From that point on, it's incremental forever so is dependent on the change rate.
Sebastian Kulis
Novice
Posts: 3
Liked: never
Joined: Jan 31, 2019 10:00 am
Full Name: Sebastian Kuliś
Contact:

Re: S3 requests calculator

Post by Sebastian Kulis »

In my scenario plan is BCJ with GFS (monthly, yearly) and only Full BCJ offload (increments only locally). Initial and so monthly 16TB of data, so I should calculate 16 million API calls per month?
Gostev
Chief Product Officer
Posts: 31606
Liked: 7097 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3 requests calculator

Post by Gostev »

Correct for the first month, and much less in the following months, because the following GFS fulls will be offloaded incrementally (only delta with the previously offloaded fulls).
poulpreben
Certified Trainer
Posts: 1024
Liked: 448 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: S3 requests calculator

Post by poulpreben »

Gostev wrote: Apr 15, 2019 1:08 pmNumber of API calls can roughly be estimated as the number of non-zeroed 1MB blocks in source images.
It is much more than that. I literally just looked at our load balancer, and the average block size is average 300-400 KB. To me, it looks like the VBK/VIB file is simply "exploded" directly into object storage without any optimizations. We have worked with 3-4 object storage vendors who are claiming that Veeam is offloading only 1 MB blocks, but this is a false statement.

We only use Veeam in "Copy Mode". Does it work differently than for "Move Mode" in terms of block size?
gummett
Veteran
Posts: 405
Liked: 106 times
Joined: Jan 30, 2017 9:23 am
Full Name: Ed Gummett
Location: Manchester, United Kingdom
Contact:

Re: S3 requests calculator

Post by gummett » 2 people like this post

Hi @poulpreben, the 1 MB (default) is size of source blocks, so before compression. 2.5-3.3x compression would result in the sizes you’re seeing.
Copy and Move work in the same way, the difference is just what recovery points are uploaded.
Object storage utilisation can be calculated at https://vse.veeambp.com/ and for background on the logic please see https://www.veeam.com/wp-designing-budg ... -tier.html
Ed Gummett (VMCA)
Senior Specialist Solutions Architect, Storage Technologies, AWS
(Senior Systems Engineer, Veeam Software, 2018-2021)
Gostev
Chief Product Officer
Posts: 31606
Liked: 7097 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3 requests calculator

Post by Gostev »

Right... he even quoted that himself :D
poulpreben
Certified Trainer
Posts: 1024
Liked: 448 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: S3 requests calculator

Post by poulpreben »

Oh, I see. I misunderstood “non-zeroed” blocks. Since it was in the context of source storage, I thought you were referring to thin provisioning.

The VSE link is 404. Is there a calculator there to “roughly” estimate the amount of blocks?
gummett
Veteran
Posts: 405
Liked: 106 times
Joined: Jan 30, 2017 9:23 am
Full Name: Ed Gummett
Location: Manchester, United Kingdom
Contact:

Re: S3 requests calculator

Post by gummett »

Sorry, proofreading fail! :oops:
https://vse.veeambp.com/

transaction costs are shown, but counts can be downloaded if you log in - DM me if you want login details
Ed Gummett (VMCA)
Senior Specialist Solutions Architect, Storage Technologies, AWS
(Senior Systems Engineer, Veeam Software, 2018-2021)
Gostev
Chief Product Officer
Posts: 31606
Liked: 7097 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3 requests calculator

Post by Gostev » 1 person likes this post

poulpreben wrote: Sep 22, 2020 4:30 amIs there a calculator there to “roughly” estimate the amount of blocks?
Probably no need for a calculator for rough estimation: 1TB disk = 1 million blocks = 1 million API calls. You may have less API calls due to zeroed and other repeating blocks, but never more.
Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests