Measuring number of PUT requests made to object storage

[codylittle]

Hey All,

Is there currently any way of measuring the number of PUTs made each time a job runs?
We're currently calculating this by the amount of data transferred / block size which is only giving a rough estimate not taking the compression into account.

Regards,
Cody

[Mildur]

Hi Cody

I'm not aware of such statistics directly in VBR.
But you can use reporting capabilities on the object storage itself to measure the used PUT's and compare that to the offloaded data in VBR for the same time period.
Do you see huge differences between your estimation and the reported PUT operations?

Thanks
Fabian

[codylittle]

Hey Fabian,

Cheers for the reply, currently we have a number of backups that run approximately at the same time and would like to determine the how many of the "total" requests are coming from each job.
Our approximations were anywhere from 1.3x to 3x off when adding up for each job vs the total from our object storage provider.

Would it be possible to just sanity check our math.

(Data Transferred * Compression rate) / block size

Data Transferred * Compression rate - To get the total amount of data copied
() / block size - To get the total number of blocks

This is based off our knowledge of that the blocks are formed and then compression is added from there, is our knowledge of this wrong?
Or are the blocks "formed' after compression is applied? so splitting the already compressed data into blocks of size X?

Regards,
Cody

[jorgedlcruz]

Hello,
I have a few great resources that might help with this task, some are more proactive, like a Whitepaper or an estimator, and another is more reactive, so you need monitoring tools, etc. Here we go:

• One of my favorite reads, which includes formulas, S3 classes, and much more. Please save this document and read it often. I have it on my bookmarks: https://www.veeam.com/wp-designing-budgeting-aws-object-storage-cloud-tier.html?wpty
• Time to Veeam Sizing Estimator, aka VSE, I have used this tool since its inception. Built by one of the brightest Architects, using well-known formulas from the rps and others, it would give you without a doubt a great estimation of Object storage calculation, all theoretical. You can use an RVTools export in case you want to be precise with the estimations.
• As said, you can always monitor puts, and other Object Storage API calls, either on Azure or AWS, or others; they usually expose this information. Here is an example on how to use Grafana, and connect it directly to Azure to obtain with all the required details, the numbers you are looking for. It would require perhaps for you to have different buckets, storage accounts, etc. If you use folders on a bucket will be a bit difficult to monitor these.

I hope all this information will be helpful for you; please let us know.

[codylittle]

Thanks for the update Jorge,
I've actually read that budgeting for AWS WP, you're right it is a very valuable resource.
One thing I was wondering about it, it does mention that the number of PUT requests for the metadata should be negligible but just wondering what the approximate size of the metadata and how that scales with backup size.

I wasn't aware of that backup sizing calculator, from having a quick look at it, it seems pretty awesome so far.
In terms of monitoring the number of PUTs we're currently experimenting with Minio and from what we've found so far, the metrics gathering for number of requests can only be done at a top level of overall number of reqs. (If this is wrong feel free to correct me (: )

I'll give these a shot and see if I can get our estimates closer to what we're observing.

Regards,
Cody

[codylittle]

For those playing along at home, we ended up continuing our tests on a minio installation.
To measure the total number of requests, we setup bucket event notifications to a PostgreSQL db.

From there some powerbi magic gives us the needed stats.

[jorgedlcruz]

Amazing workaround Cody,
I was just googling about this, and there it seems to be a really verbose log, which perhaps passing it to Loki, or elastic you can do something, but seeing how simple is using your method, documented here, I would stick to this. If you ending doing something really atractive, please share some blog, image, wiki, etc.

Thank you so much!