Using object storage as a backup target
wa15
Expert
Posts: 282
Liked: 22 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Veeam and Azure Blob Storage API calls

Post by wa15 » 1 person likes this post

We are considering long-term archiving to cloud instead of tape. Looking at Azure Blob pricing, they charge let's say $0.010 per 10,000 write operations and 0.050 per 10,000 list/create operations for the cold-tier.

How does one go about knowing how many write operations or list/create operations Veeam does with SOBR? Trying to get an estimate of how much our monthly cost with Azure would be which requires having an idea of the number of read/write operations. How does one go about estimating this?

HannesK
Veeam Software
Posts: 6119
Liked: 899 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by HannesK » 2 people like this post

Hello,
the "educated guess" is approximately 1 PUT per 1MB of source data size for each full or incremental backup that is offloaded. Not counting deduplication between multiple full backups. That represents 99% of the API costs. Other PUTs and GETs are not really relevant. This is only for backup. Restore is a different story of course.

Best regards,
Hannes

wa15
Expert
Posts: 282
Liked: 22 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by wa15 »

Thanks Hannes. That helps.

Let's say our single full backup is 15TB, and we want to backup twice a month. Very first backup would be a full 15TB. Then the second backup would also be a full backup, but would be deduplicated with the first one. Is there extra extra API calls for that deduplication? Asking because you mentioned "Not counting deduplication between multiple full backups"

Also, any idea for read API calls in case of restore?

HannesK
Veeam Software
Posts: 6119
Liked: 899 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by HannesK » 1 person likes this post

Hi,
Then the second backup would also be a full backup, but would be deduplicated with the first one
yes, correct.
Is there extra extra API calls for that deduplication?
not really. It's not classic deduplication where blocks are moved. it's similar to ReFS block cloning.

Best regards,
Hannes

wa15
Expert
Posts: 282
Liked: 22 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by wa15 »

Thanks. To follow up, any ideas on the number of API calls in case of restores? Is there a ballpark figure per MB of data restored?

HannesK
Veeam Software
Posts: 6119
Liked: 899 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by HannesK »

Hello,
it is not really possible to estimate restore calls. Example: if parts of the required data are still available in a local repository, we we use that.

Best regards,
Hannes

Gostev
SVP, Product Management
Posts: 26706
Liked: 4277 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by Gostev »

Worst case scenario is the number of non-zeroed 1MB blocks in the restored machine's disks. Usually less since we get matching blocks from local SOBR extents, as Hannes noted. Thanks!

wa15
Expert
Posts: 282
Liked: 22 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by wa15 »

Thanks guys!

wa15
Expert
Posts: 282
Liked: 22 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by wa15 »

Bringing this thread back to life again for one more question:

We offloaded about 30TB of VBKs to Azure for an initial offload, resulting in about 24 million PutBlob operations. Do we expect this same number of Put operations for the next VBK offload? Or will that number go down since Veeam only offloads new data?

wa15
Expert
Posts: 282
Liked: 22 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by wa15 »

In addition to the question above, what type of Azure API call is triggered when old backups are removed out of Veeam? And how many operations approximately per lets say 1TB of data?

HannesK
Veeam Software
Posts: 6119
Liked: 899 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by HannesK » 1 person likes this post

Hello,
no, there will be less data transfer / put operations for the next VBK. Veeam will only offload the required data (incremental) to object storage. The concept of "real full backups" does not exist for object storage.

Imagine it like a REFS repository. The amount of data transferred is like an forever incremental chain.

For deletion: same like for upload. If you go into the storage account, you can see all the files. The opposite of "put" is "delete blob".

I just did the math on your 30 TB / 24 Mio puts and you have less than one put per MByte (did you change block size?)... in general it's about more or less one operation per MByte

Best regards,
Hannes

wa15
Expert
Posts: 282
Liked: 22 times
Joined: Jan 02, 2014 4:45 pm
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by wa15 »

Thanks Hannes.

Regarding put operations for our 30TB offload, I think it is closer to 30 million as I didn't have the graph scaled correctly (and our offload is really closer to 29TB but I was rounding up) so the 1 put per MB is about right.

Ralph.A
Lurker
Posts: 1
Liked: 1 time
Joined: Feb 24, 2020 5:47 pm
Contact:

[MERGED] Veeam Backup to Azure Blob - Write Operation Cost

Post by Ralph.A » 1 person likes this post

Morning,

I've tested the backup copy of V10 and i have a question for all of you who tested it.

Block Blob Storage / Standard / General StorageV2 / Hot tier / LRS

I did a couple of test to see how its works and i was stagered by the WRITE OPERATION Cost

1.6Tb Full Backup + 25Gb incremental for 5 day = 13 500 000 Write operations (+-100$ca :shock: ) and thats only week 1...

Is there a way to calculate the Write operation Cost ?

Also any advise on how to manage the write operations ?

I was thinking about creating a second job with a forever incremental, not sure how to approach this

dalbertson
Veeam Software
Posts: 280
Liked: 70 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Veeam Backup to Azure Blob - Write Operation Cost

Post by dalbertson » 1 person likes this post

This was discussed in another post.

object-storage-f52/veeam-and-azure-blob ... 57973.html

Gostev
SVP, Product Management
Posts: 26706
Liked: 4277 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [MERGED] Veeam Backup to Azure Blob - Write Operation Cost

Post by Gostev » 1 person likes this post

Ralph.A wrote: Feb 24, 2020 6:06 pmI was thinking about creating a second job with a forever incremental, not sure how to approach this
There's no need to do that. Offload to object storage is forever-incremental regardless of the source backup chain type.

What does make difference is changing Storage optimization setting in the advanced job setting to Local target (large blocks). This makes Veeam use 4MB block, which will reduce the number of blocks (and so API costs) 4 times. However, this comes at the cost of about 2x large incremental backups (which adds to storage costs), and also slower (and more expensive) granular restores directly from cloud backups. Please note that changing block size requires an active full backup.

If you are not married to Azure and it's API costs is a problem, then another option would be to consider object storage providers which do not charge for API calls, for example Wasabi.

bennytalent
Novice
Posts: 3
Liked: never
Joined: Sep 17, 2013 11:34 am
Full Name: Benjamin Hornung
Contact:

[MERGED] AWS S3 calculation for api requests with cloud tier

Post by bennytalent »

hi,
i'm planning an amazon s3 cloud tier implementation and i am struggeling with the price calculation.
does anyone have expirience with the aws calculation values "PUT, COPY, POST, LIST requests to S3 Standard - Infrequent Access" and "GET, SELECT, and all other requests from S3 Standard - Infrequent Access" in conjunction with veeam cloud tier?

thx in advance guys :)

p.s. stay safe

HannesK
Veeam Software
Posts: 6119
Liked: 899 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by HannesK »

Hello,
please see the answers above. If you multiply that with your costs, then you can do an estimation.

Or just ignore it, because it's not the main cost factor :-)

Best regards,
Hannes

UniMatrix
Novice
Posts: 6
Liked: never
Joined: Dec 04, 2015 9:18 am
Full Name: Erik van der Neut
Contact:

Re: [MERGED] Veeam Backup to Azure Blob - Write Operation Cost

Post by UniMatrix »

Ralph.A wrote: Feb 24, 2020 6:06 pm Morning,

I've tested the backup copy of V10 and i have a question for all of you who tested it.

Block Blob Storage / Standard / General StorageV2 / Hot tier / LRS

I did a couple of test to see how its works and i was stagered by the WRITE OPERATION Cost

1.6Tb Full Backup + 25Gb incremental for 5 day = 13 500 000 Write operations (+-100$ca :shock: ) and thats only week 1...

Is there a way to calculate the Write operation Cost ?

Also any advise on how to manage the write operations ?

I was thinking about creating a second job with a forever incremental, not sure how to approach this
We are experiencing similar unexpected costs, write operations are THREE TIMES the 'data stored' cost. About 100 euro's a week, which we expected to be the monthly costs...

dalbertson
Veeam Software
Posts: 280
Liked: 70 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by dalbertson »

@UniMatrix what version are you on?

dalbertson
Veeam Software
Posts: 280
Liked: 70 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by dalbertson »

Can you open a support case please? We should be able to track it down and see what’s causing the issue.

UniMatrix
Novice
Posts: 6
Liked: never
Joined: Dec 04, 2015 9:18 am
Full Name: Erik van der Neut
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by UniMatrix »

@dalbertson v10 and the update that came out recently 10.0.0.4461 P1

UniMatrix
Novice
Posts: 6
Liked: never
Joined: Dec 04, 2015 9:18 am
Full Name: Erik van der Neut
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by UniMatrix »

dalbertson wrote: May 27, 2020 11:48 am Can you open a support case please? We should be able to track it down and see what’s causing the issue.
Thanks, will do!

dalbertson
Veeam Software
Posts: 280
Liked: 70 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by dalbertson »

Then if you could post the support Id in here it will be helpful.

Gostev
SVP, Product Management
Posts: 26706
Liked: 4277 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by Gostev »

What's your source data size? With the default settings, a simple estimate is 1 write per 1MB of source data. So 1TB of source data = 1 million write operations. This is true for both the full and increments, except after the initial full it will be incrementals forever, and you should be looking at the changed source data size instead, which is fairly small in your case.

Also, check if you are possibly using the default "Local" block size under storage optimization in the advanced backup job settings. Because using small block size will obviously inflate the number of write operations accordingly. The default setting is 1MB, but WAN Target setting uses 256KB, which means 4 times more write operations.

UniMatrix
Novice
Posts: 6
Liked: never
Joined: Dec 04, 2015 9:18 am
Full Name: Erik van der Neut
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by UniMatrix »

I (finally: had to be made case administrator...) managed to open a case #04210999

Costs seem to have settled down a bit after the initial copy but the 'write operations' part is still larger than expected.

Not sure if it is possible to change block size for the SOBR tiering / offload jobs alone (read something about different block sizes not being possible) and if changing it 'globally' leads to more space being used on our local repositories or having to do full's again.

HannesK
Veeam Software
Posts: 6119
Liked: 899 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by HannesK » 1 person likes this post

the block size is a "per-job" setting. It should not be changed without good reason. Saving a little disk space is not a good reason ;-)

https://helpcenter.veeam.com/docs/backu ... ml?ver=100

UniMatrix
Novice
Posts: 6
Liked: never
Joined: Dec 04, 2015 9:18 am
Full Name: Erik van der Neut
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by UniMatrix »

Gostev wrote: May 27, 2020 7:58 pm What's your source data size? With the default settings, a simple estimate is 1 write per 1MB of source data. So 1TB of source data = 1 million write operations. This is true for both the full and increments, except after the initial full it will be incrementals forever, and you should be looking at the changed source data size instead, which is fairly small in your case.

Also, check if you are possibly using the default "Local" block size under storage optimization in the advanced backup job settings. Because using small block size will obviously inflate the number of write operations accordingly. The default setting is 1MB, but WAN Target setting uses 256KB, which means 4 times more write operations.
There is now about 6TB in the storage account, according to the Azure pricing calculator 6 million write operations should cost about 50 euro's, however, we see about 200 euro's in write operations.

All jobs are configured with the 'Local target' storage optimization.

We also have a case with Microsoft, as we are billed 'grs' everywhere while the storage account is configred with lrs... :roll: (which also drives the costs up).

Gostev
SVP, Product Management
Posts: 26706
Liked: 4277 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by Gostev »

You may have missed this in my post, but what matters for the number of operations is solely the source data size, and not the amount of data in the storage account.

gummett
Veeam Software
Posts: 187
Liked: 39 times
Joined: Jan 30, 2017 9:23 am
Full Name: Ed Gummett
Location: Manchester, United Kingdom
Contact:

Re: Veeam and Azure Blob Storage API calls

Post by gummett » 1 person likes this post

I should probably plug this white paper here, not least so its found by people searching for this subject :)
For Azure just replace 'S3 standard' with 'Hot', and 'Infrequent Access' with Cool (also 1-zone = LRS)
https://www.veeam.com/wp-designing-budg ... -tier.html

Hopefully it's helpful
Ed Gummett (VMCA)
Senior Systems Engineer UK & Ireland
Veeam Software

ilovecats
Novice
Posts: 4
Liked: never
Joined: Sep 09, 2015 7:16 pm
Contact:

Azure object storage and transaction (write) PubBlob operation counts

Post by ilovecats »

I have set up a scale-out repo with Azure blob storage as the capacity tier in "copy all backups" mode, essentially using Azure as offsite storage. It seems to work just fine.

However, I noticed while observing the insights and metrics on the Azure side that Veeam backups generate a fair amount of transactions, mostly in the form of PutBlob write operations.

As an example, a test incremental backup of only around 2GB resulted in about 9K PutBlob operations.

As we know Azure does charge for these operations. With this backup size to transaction ratio it can actually add up to some significant numbers over time.

I wonder if Veeam has been optimized for this, and if there are any settings I can adjust to reduce this?

My understanding is that for this type of scale-out repo setup, Veeam would first backup normally to performance tier local storage (a NAS in my setup), then simply copy the backup from the NAS to the Azure storage account. However, I noticed in my Azure blob container that the files Veeams uploaded are not full .VIB files etc, but rather in its own format of folders and many small parts. Could this process have been too wasteful in terms of write operations?

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests