Sudden storage increase in AWS S3 storage

[rezen]

Hi guys,

We have been uploading our backup data directly to S3 for the last seven months or so and haven' t had any issues until recently where our bucket size increased from around 40TB to 65TB within the last few weeks.
We have checked the data uploaded to S3 for the last month or so and it has been pretty consistent with no massive change.
We then looked at what was stored in the bucket and found some old data from early June which had much longer retention periods as usual (we only retain the last 5 days of data for the backup copy) .
I understand how the block generation works, which usually adds more days to the retention period and our S3 data usually expire in two weeks or so but recently upload data has more than a month before it expires.
I opened a case with support (#07336603) and they sent me this article https://helpcenter.veeam.com/docs/backu ... ml?ver=120, which shows the default generation period for Amazon S3 is 30 days. I don't think it was always like that.
I tried changing the S3 generation interval from 30 to 10 days by following this https://www.veeam.com/kb4470 but am not sure if it's actually working.

The only change we have recently made was updating the Veeam B&R to 12.1.2.172 on 7th June.
Has anyone reported a similar issue?

[haslund]

Are you using AWS S3 or IBM Cloud?
How long a period do you have defined for immutability?

[rezen]

Hi haslund,

We're using S3 and the immutability is set for 5 days for the repository.

[tyler.jurgens]

Veeam changed the default block generation for Amazon S3 to 30 days in one of the Veeam 12 releases (I think it was one of the most recent patches). They did this in order to cut down on API calls that are generated to Amazon S3 buckets, as API calls can often cost you more than just keeping the data around longer. So yes, your size may have increased but your API calls should have decreased. It would be prudent to check how this affected your bill.

You can see the v11 version of the applicable document only lists 10 days for block generation.
https://helpcenter.veeam.com/archive/ba ... k_gen.html

[Gostev]

Right, the default block generation was increased for AWS S3 to reduce fully loaded costs (storage + API calls). It was not increased for any other cloud object storage providers because they either don't charge for API calls at all (like Wasabi) or at least don't charge for immutability extension API calls specifically (like Azure Blob Storage).

[rezen]

Hi guys,

Thanks for your responses and confirmation.
Our forecasted cost for current month has increased by almost 40% compared to the last month's bill.
I've reduced the generation period to 10 days with the registry key in the KB4470 and will see how it goes.

[Gostev]

I would expect that with the new 30 days setting you will only start seeing total cost benefits after some time. As for the current month you will be paying for both the increasing storage consumption and for API calls of previous more frequent immutability extensions.

[tyler.jurgens]

Additionally, now you have the increased storage, and increased API calls since you switched block generation down to 10 days again. That additional size will need to work its way out of the system (waiting for the initial block generation time of 30 days for those specific objects) to work its way through. Your call on how to proceed, but if Veeam is correct on keeping API calls down to a minimum will decrease your bill over time that initial 40% cost increase would pay for itself over the next few months.

[rezen]

Thanks again, guys.
We will see how we go with the 10 days retention and are also looking into the Azure Blob storage as Gostev mentioned.
When we looked at it a couple of years ago, it didn't offer immutability back then but it does now and interesting to see how it compares with S3.

[mcz]

Guys, I also saw a huge increase in storage consumption on my wasabi-bucket the last weeks. The thing is that you can't really figure out why and what has caused this increase. Summing up the restore points doesn't help as you never know how much deduplication due to identical blocks/objects there might be.

Is it possible that the REAL storage consumption will be shown in future releases? Just like adding a new column to the details and then you could do the math an would understand, why you suddenly needed 10 TB on top the last weeks. Just some weeks ago I had an open case where I asked why no space was reclaimed from the object storage and even with the tier 2 engineer it was very hard to impossible to figure out, what consumed how much.

So I'd say a bit more transparency is needed here... Thanks!

[Gostev]

As explained above Wasabi (or any other object storage for that matter) didn't receive the discussed changes so please don't derail the topic. You can open a support case to troubleshoot your increase. Thanks

[mcz]

Sorry, I didn't wannt to derail the topic. It's just a pain that you don't have the real figures what a vm really does consume on your object storage. Because if you have an increase, what I wannt to answer is "which vm did cause that increase"? Has anybody changed/created a lot of data on that vm or do we have a change in retention? It's hard to impossible to answer that questions and support never had the tools and answers to these questions. So please consider it as feature request...

[gummett]

... or at least don't charge for immutability extension API calls specifically (like Azure Blob Storage).

Hi Anton, I just wanted to clarify that Azure do charge for immutability extension calls, under the 'Other operations' category, per https://learn.microsoft.com/en-us/rest/ ... id#billing

Amazon S3 charges an S3 Standard PUT operation for extending retention (regardless of the storage class the object is in).

[JeroenL]

Sorry, I didn't wannt to derail the topic. It's just a pain that you don't have the real figures what a vm really does consume on your object storage. Because if you have an increase, what I wannt to answer is "which vm did cause that increase"? Has anybody changed/created a lot of data on that vm or do we have a change in retention? It's hard to impossible to answer that questions and support never had the tools and answers to these questions. So please consider it as feature request...

Using VeeamOne you can see per VM what the daily change rate has been.

[Gostev]

Hi Anton, I just wanted to clarify that Azure do charge for immutability extension calls, under the 'Other operations' category, per https://learn.microsoft.com/en-us/rest/ ... id#billing

On paper yes, but not actually... they are effectively free which is why we kept the value at 10 days for Azure.

Also, I believe you're employed with their competitor (AWS)? Let's have Microsoft employees issue corrections wrt. to Azure Blob Storage if any are needed please.

[UPDATE] Looks like the reason could be that in Azure, these operations are 10x cheaper than PUTs... which is why their cost has been at the noise level on the overall bill for us.