AWS S3 standard pricing expectations

[theviking84]

I've got around 17TB of backup data on premise. I am going to be creating a SOBR and extending with Amazon S3 standard object storage. I understand it will be incremental forever and I'm going to move longer term stuff to the s3 with move mode in the sobr. I'm going to estimate 22TB so I have a little growth and hope that covers for at least a while.

My question is, using the Amazon s3 price calculator tool here is what I am putting in. Is this correct for how veeam might work with this?

22 tb per month. 22,000,000 PUT COPY POST LIST requests to s3. ? That comes out to roughly $630 per month. The vast majority of time we would never restore from this s3, and only a file here or there at most unless we have a disaster.

I'm just trying to understand if I'm missing some huge thing that veeam will do with api calls or I'm way wrong on how I am calculating this. Our growth rates seem to be like 150gb per day, so I could expect that much going to s3 daily then? That's like 4.5 TB per month of growth but the incremental nature of the sobr copy/offload and retention of the jobs to only keep say 4 weekly, 12 monthly etc should prevent it from really growing a ton I'd think? (the main retention will only be like a week, so that would always roll out, and then some vms will have the weekly/monthly etc)

I'm just afraid of like some kind of exponential huge growth and cost on this. So anyway 17T of data with some longer retention refs style in s3, isn't going to go to 34tb in a month etc etc right? My price sounds in line from using aws calculator?

Last question, when it comes to immutable in s3. So if I have a job with some vms in it, and they have 4 weekly, 12 monthly, etc. I would still want to only set the immutable option to 2 or 3 days? Like much less than all the data that is stored? I usually see it recommended to keep far less immutable than you store, but that means all your longer term backups are not protected.

[Gostev]

Since offload to object storage is forever-incremental, you only offload those 17TB worth of backup data once. After that, it is changes only (new and changed data). So, you don't need those API costs "per month" but only once, and a few times less each following month.

You're correct, it's ReFS-style storage architecture with Veeam in object storage, so GFS full backups don't take any extra physical disk space except for metadata.

The Capacity Tier immutability is designed to protect most recent backups against ransomware, so normally you should use the immutability period of no more than a few weeks. For immutable GFS backups, you will want to wait for the Archive Tier feature of the upcoming v11, which offloads aging GFS restore points further to Glacier (or Glacier Deep Archive) for long term storage. There, they can be optionally immutable for their entire lifetime.

[theviking84]

Thank you Gostev! Some of this is great news.

So to clarify, when we get to v11 and use the glacier archive tier... Will this mean another 17TB must go to glacier "once" and then it will still be able to do incremental forever offloads to archive tier for each monthly/yearly copy? Or will it have to be separate fulls each time it goes into glacier?

So I'd have 17tb in s3 and then 17tb in glacier, and then ongoing just changes to both despite keeping monthly and yearlies in glacier?

[Gostev]

Yes, always changes only.

Conceptually, the Archive Tier is largely identical. There are however some noticeable differences due to Glacier design peculiarities to make the economics work, for example we use much larger objects due to very high API costs. For the same reason, while offload to the Archive Tier is also incremental-forever, each new restore point "dedupes" itself only with the immediate previous restore point - and not across all restore points, like in the Capacity Tier. But the end result is largely the same - just with a bit less storage savings, although storage savings are much less important considering how cheap Glacier is (and especially Deep Archive).

We do actually also provide an option to store standalone fulls in Glacier, as some people don't like the idea of forever-incremental archiving when it comes to storing the data for very many years, however it is not a default behavior.

[theviking84]

I'm assuming your own opinion is that the incremental forever nature for long term is still fine and the non default behavior is only to comfort those people? I can understand their logic but I believe there is a lot of redundancy of the data and metadata to where it is still most likely going to be ok? I guess obviously or you wouldn't have this as default behavior therefore catching most customers.

I think glacier is 5 times cheaper than S3 so yeah even with that lesser dedupe it still should even out.

Thanks again!

[Gostev]

We know that regular S3 is extremely reliable: due to inline integrity checks and multiple copies of each object, it provides by an order of magnitude more resilient storage comparing to your typical on-prem backup storage. So, we're very confident with doing forever incremental there.

We don't have any experience with Glacier yet, which is why we provide an option. The default is from the assumption that Glacier is no different from regular S3 in terms of reliability, but since its implementation details are best kept secret, and given how extremely cheap the Deep Archive tier is, one could assume there's reduced redundancy (number of copies) at least in case of the Deep Archive. So, some customers may prefer independent fulls at least with the latter.

[jmmarton]

I could see compliance teams also wanting the option for standalone fulls in Glacier. Since we're talking potential long-term archiving, by using standalone fulls it can be guaranteed that when a restore point is deleted all associated objects for that point in time are removed. But with the forever-incremental that's not the case as objects may still be referenced by another restore point and thus still remain.

Joe

[Gostev]

Well, this could only happen if that part of the virtual disk did not change between restore points (for example, between weekly GFS). So basically, in either case you will still have the same exact blob in your backup repository: either only one copy shared between all restore points where it is present, or multiple copies of the same exact blob. In light of this, I don't see how it can make any difference to compliance teams.

[theviking84]

Gostev,

Regarding this thread.. object-storage-f52/aws-s3-how-to-reduce ... 68858.html

It mentions that every 10 days a full backup worth of api calls is needed. You said above first month api is large but future months are a few times less.

But this seems to be a lot more expensive than first thought if every 10 days another full 17tb worth of api is needed to keep any short term immutability?

[Gostev]

Yes, enabling the immutability feature will cost you some additional money with cloud object storage vendors who charge for API calls. But this is still many times cheaper than what you would pay if using container-based immutability, since the latter cannot support forever-incremental backups - and storage is so much more expensive than API calls.

But if this is still an issue, keep in mind there are other cloud object storage providers who don't charge for API!