Veeam and Azure Blob Storage API calls

[wa15]

We are considering long-term archiving to cloud instead of tape. Looking at Azure Blob pricing, they charge let's say $0.010 per 10,000 write operations and 0.050 per 10,000 list/create operations for the cold-tier.

How does one go about knowing how many write operations or list/create operations Veeam does with SOBR? Trying to get an estimate of how much our monthly cost with Azure would be which requires having an idea of the number of read/write operations. How does one go about estimating this?

[HannesK]

Hello,
the "educated guess" is approximately 1 PUT per 1MB of source data size for each full or incremental backup that is offloaded. Not counting deduplication between multiple full backups. That represents 99% of the API costs. Other PUTs and GETs are not really relevant. This is only for backup. Restore is a different story of course.

Best regards,
Hannes

[wa15]

Thanks Hannes. That helps.

Let's say our single full backup is 15TB, and we want to backup twice a month. Very first backup would be a full 15TB. Then the second backup would also be a full backup, but would be deduplicated with the first one. Is there extra extra API calls for that deduplication? Asking because you mentioned "Not counting deduplication between multiple full backups"

Also, any idea for read API calls in case of restore?

[HannesK]

Hi,

Then the second backup would also be a full backup, but would be deduplicated with the first one

yes, correct.

Is there extra extra API calls for that deduplication?

not really. It's not classic deduplication where blocks are moved. it's similar to ReFS block cloning.

Best regards,
Hannes

[wa15]

Thanks. To follow up, any ideas on the number of API calls in case of restores? Is there a ballpark figure per MB of data restored?

[HannesK]

Hello,
it is not really possible to estimate restore calls. Example: if parts of the required data are still available in a local repository, we we use that.

Best regards,
Hannes

[Gostev]

Worst case scenario is the number of non-zeroed 1MB blocks in the restored machine's disks. Usually less since we get matching blocks from local SOBR extents, as Hannes noted. Thanks!

[wa15]

Thanks guys!

[wa15]

Bringing this thread back to life again for one more question:

We offloaded about 30TB of VBKs to Azure for an initial offload, resulting in about 24 million PutBlob operations. Do we expect this same number of Put operations for the next VBK offload? Or will that number go down since Veeam only offloads new data?

[wa15]

In addition to the question above, what type of Azure API call is triggered when old backups are removed out of Veeam? And how many operations approximately per lets say 1TB of data?

[HannesK]

Hello,
no, there will be less data transfer / put operations for the next VBK. Veeam will only offload the required data (incremental) to object storage. The concept of "real full backups" does not exist for object storage.

Imagine it like a REFS repository. The amount of data transferred is like an forever incremental chain.

For deletion: same like for upload. If you go into the storage account, you can see all the files. The opposite of "put" is "delete blob".

I just did the math on your 30 TB / 24 Mio puts and you have less than one put per MByte (did you change block size?)... in general it's about more or less one operation per MByte

Best regards,
Hannes

[wa15]

Thanks Hannes.

Regarding put operations for our 30TB offload, I think it is closer to 30 million as I didn't have the graph scaled correctly (and our offload is really closer to 29TB but I was rounding up) so the 1 put per MB is about right.

[Ralph.A]

Morning,

I've tested the backup copy of V10 and i have a question for all of you who tested it.

Block Blob Storage / Standard / General StorageV2 / Hot tier / LRS

I did a couple of test to see how its works and i was stagered by the WRITE OPERATION Cost

1.6Tb Full Backup + 25Gb incremental for 5 day = 13 500 000 Write operations (+-100$ca ) and thats only week 1...

Is there a way to calculate the Write operation Cost ?

Also any advise on how to manage the write operations ?

I was thinking about creating a second job with a forever incremental, not sure how to approach this

[dalbertson]

This was discussed in another post.

object-storage-f52/veeam-and-azure-blob ... 57973.html

[Gostev]

I was thinking about creating a second job with a forever incremental, not sure how to approach this

There's no need to do that. Offload to object storage is forever-incremental regardless of the source backup chain type.

What does make difference is changing Storage optimization setting in the advanced job setting to Local target (large blocks). This makes Veeam use 4MB block, which will reduce the number of blocks (and so API costs) 4 times. However, this comes at the cost of about 2x large incremental backups (which adds to storage costs), and also slower (and more expensive) granular restores directly from cloud backups. Please note that changing block size requires an active full backup.

If you are not married to Azure and it's API costs is a problem, then another option would be to consider object storage providers which do not charge for API calls, for example Wasabi.

[bennytalent]

hi,
i'm planning an amazon s3 cloud tier implementation and i am struggeling with the price calculation.
does anyone have expirience with the aws calculation values "PUT, COPY, POST, LIST requests to S3 Standard - Infrequent Access" and "GET, SELECT, and all other requests from S3 Standard - Infrequent Access" in conjunction with veeam cloud tier?

thx in advance guys

p.s. stay safe

[HannesK]

Hello,
please see the answers above. If you multiply that with your costs, then you can do an estimation.

Or just ignore it, because it's not the main cost factor

Best regards,
Hannes

[UniMatrix]

Morning,

I've tested the backup copy of V10 and i have a question for all of you who tested it.

Block Blob Storage / Standard / General StorageV2 / Hot tier / LRS

I did a couple of test to see how its works and i was stagered by the WRITE OPERATION Cost

1.6Tb Full Backup + 25Gb incremental for 5 day = 13 500 000 Write operations (+-100$ca ) and thats only week 1...

Is there a way to calculate the Write operation Cost ?

Also any advise on how to manage the write operations ?

I was thinking about creating a second job with a forever incremental, not sure how to approach this

We are experiencing similar unexpected costs, write operations are THREE TIMES the 'data stored' cost. About 100 euro's a week, which we expected to be the monthly costs...

[dalbertson]

@UniMatrix what version are you on?

[dalbertson]

Can you open a support case please? We should be able to track it down and see what’s causing the issue.

[UniMatrix]

@dalbertson v10 and the update that came out recently 10.0.0.4461 P1

[UniMatrix]

Can you open a support case please? We should be able to track it down and see what’s causing the issue.

Thanks, will do!

[dalbertson]

Then if you could post the support Id in here it will be helpful.

[Gostev]

What's your source data size? With the default settings, a simple estimate is 1 write per 1MB of source data. So 1TB of source data = 1 million write operations. This is true for both the full and increments, except after the initial full it will be incrementals forever, and you should be looking at the changed source data size instead, which is fairly small in your case.

Also, check if you are possibly using the default "Local" block size under storage optimization in the advanced backup job settings. Because using small block size will obviously inflate the number of write operations accordingly. The default setting is 1MB, but WAN Target setting uses 256KB, which means 4 times more write operations.

[UniMatrix]

I (finally: had to be made case administrator...) managed to open a case #04210999

Costs seem to have settled down a bit after the initial copy but the 'write operations' part is still larger than expected.

Not sure if it is possible to change block size for the SOBR tiering / offload jobs alone (read something about different block sizes not being possible) and if changing it 'globally' leads to more space being used on our local repositories or having to do full's again.

[HannesK]

the block size is a "per-job" setting. It should not be changed without good reason. Saving a little disk space is not a good reason

https://helpcenter.veeam.com/docs/backu ... ml?ver=100

[UniMatrix]

What's your source data size? With the default settings, a simple estimate is 1 write per 1MB of source data. So 1TB of source data = 1 million write operations. This is true for both the full and increments, except after the initial full it will be incrementals forever, and you should be looking at the changed source data size instead, which is fairly small in your case.

Also, check if you are possibly using the default "Local" block size under storage optimization in the advanced backup job settings. Because using small block size will obviously inflate the number of write operations accordingly. The default setting is 1MB, but WAN Target setting uses 256KB, which means 4 times more write operations.

There is now about 6TB in the storage account, according to the Azure pricing calculator 6 million write operations should cost about 50 euro's, however, we see about 200 euro's in write operations.

All jobs are configured with the 'Local target' storage optimization.

We also have a case with Microsoft, as we are billed 'grs' everywhere while the storage account is configred with lrs... (which also drives the costs up).

[Gostev]

You may have missed this in my post, but what matters for the number of operations is solely the source data size, and not the amount of data in the storage account.

[gummett]

I should probably plug this white paper here, not least so its found by people searching for this subject
For Azure just replace 'S3 standard' with 'Hot', and 'Infrequent Access' with Cool (also 1-zone = LRS)
https://www.veeam.com/wp-designing-budg ... -tier.html

Hopefully it's helpful

[ilovecats]

I have set up a scale-out repo with Azure blob storage as the capacity tier in "copy all backups" mode, essentially using Azure as offsite storage. It seems to work just fine.

However, I noticed while observing the insights and metrics on the Azure side that Veeam backups generate a fair amount of transactions, mostly in the form of PutBlob write operations.

As an example, a test incremental backup of only around 2GB resulted in about 9K PutBlob operations.

As we know Azure does charge for these operations. With this backup size to transaction ratio it can actually add up to some significant numbers over time.

I wonder if Veeam has been optimized for this, and if there are any settings I can adjust to reduce this?

My understanding is that for this type of scale-out repo setup, Veeam would first backup normally to performance tier local storage (a NAS in my setup), then simply copy the backup from the NAS to the Azure storage account. However, I noticed in my Azure blob container that the files Veeams uploaded are not full .VIB files etc, but rather in its own format of folders and many small parts. Could this process have been too wasteful in terms of write operations?