Discussions specific to object storage
veremin
Product Manager
Posts: 16880
Liked: 1430 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by veremin » Feb 25, 2019 6:36 pm

Correct, hot and cold tiers are supported, archival - not. Thanks!

Gostev
SVP, Product Management
Posts: 24787
Liked: 3519 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by Gostev » Feb 25, 2019 8:05 pm

dimaslan wrote:
Feb 25, 2019 9:50 am
This of course changes pricing by x5. Is a VLT on Azure the only solution for this for now then? Even with 30 TB, the difference is significant.
Actually, as usual in this life, it's really not as straightforward and easy as just looking at the price... archive storage has its own peculiarities which very well balance its lower cost.

For example, with regular object storage today, we're able to offload backups in forever-incremental manner, effectively providing source-side dedupe between multiple full backups. Also, we're able to perform block-level retention to immediately and granularly remove any unused blocks. This is possible because we're able to update metadata sitting in object storage in real time.

None of this will be possible with archive storage due to its nature, so we'd have to resort to classic incremental chains with periodic fulls instead of forever-incremental to make this "special" object storage work while keeping retention processing possible. Much like you will have to do it with your VTL workaround, anyway. However, as soon as you have created just a few periodic fulls, you suddenly find yourself consuming 5x more object storage than Capacity Tier would, which obviously kills all those pricing benefits.

And note that we did not even get to the time and the cost of performing restores from archive storage, which is even a sadder story.

andriktr
Enthusiast
Posts: 41
Liked: 1 time
Joined: Mar 02, 2015 11:53 am
Full Name: Andrej
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by andriktr » Feb 28, 2019 7:45 pm

Do I understand correctly that with U4 update we can put backup copy jobs directly to the Azure Blobs ?
Thank you.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3519 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by Gostev » Feb 28, 2019 8:32 pm

No, your understanding is not correct: backing up (or copying backups) directly to object storage is not supported.

dimaslan
Service Provider
Posts: 50
Liked: 7 times
Joined: Jul 01, 2017 8:02 pm
Full Name: Dimitris Aslanidis
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by dimaslan » Mar 01, 2019 7:56 pm

Gostev wrote:
Feb 25, 2019 8:05 pm
Actually, as usual in this life, it's really not as straightforward and easy as just looking at the price... archive storage has its own peculiarities which very well balance its lower cost.

For example, with regular object storage today, we're able to offload backups in forever-incremental manner, effectively providing source-side dedupe between multiple full backups. Also, we're able to perform block-level retention to immediately and granularly remove any unused blocks. This is possible because we're able to update metadata sitting in object storage in real time.

None of this will be possible with archive storage due to its nature, so we'd have to resort to classic incremental chains with periodic fulls instead of forever-incremental to make this "special" object storage work while keeping retention processing possible. Much like you will have to do it with your VTL workaround, anyway. However, as soon as you have created just a few periodic fulls, you suddenly find yourself consuming 5x more object storage than Capacity Tier would, which obviously kills all those pricing benefits.

And note that we did not even get to the time and the cost of performing restores from archive storage, which is even a sadder story.
Gostev, I understand the benefit you are describing as well as the drawbacks of Archive tier. It is, however, ideal for situations where you need to store monthly and yearly full backup files which will only in a case of audit or emergency need to be read. Especially as an MSP, even if you need to do this for 3-4 customers, it might mean adding 10+ TBs per month new storage space to accomodate the new monthlies. With 0.0019 $ / GB, even 50 TBs amount to roughly $ 100 / month for the space which is great, considering the fact it saves the fuss of managing + maintaining storage and the headache of expanding the volumes.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3519 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by Gostev » Mar 01, 2019 11:02 pm

Dimitris, perhaps you misunderstood my point then.

What I am saying is that in the scenario when you need to retain multiple monthly and yearly full backups, Capacity Tier on cool blobs will be cheaper than storing standalone fulls on archive blobs. While archive blobs are indeed 5 times cheaper, you will need 10 times more storage to store 12 monthly and a few yearly standalone full backups there. So overall, when using archive blobs you will pay more - not less.

dimaslan
Service Provider
Posts: 50
Liked: 7 times
Joined: Jul 01, 2017 8:02 pm
Full Name: Dimitris Aslanidis
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by dimaslan » Mar 04, 2019 12:31 pm

Gostev,

I will need to read a little bit more about how the tiers work, then.

Thank you.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3519 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by Gostev » Mar 04, 2019 7:04 pm

Here's the relevant snippet from the What's New document:
Space efficiency: In addition to source-side compression, backup files
are offloaded to object storage in the forever incremental manner, thus
preventing duplication between multiple full backups, which effectively
provides source-side deduplication

PGoldi
Lurker
Posts: 1
Liked: never
Joined: Mar 07, 2019 7:57 am
Contact:

[MERGED] Veeam Backup & Replication Console 9.5 Update 4 Azure Blob Storage

Post by PGoldi » Mar 07, 2019 10:02 am

Hello,
is it now possible to select the Azure Blob Storage as target for a backup copy job?

best wishes
Patrick

wishr
Veeam Software
Posts: 1148
Liked: 117 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by wishr » Mar 07, 2019 10:37 am

Hi Patrick,

Welcome to Veeam Community Forums and thanks for posting your question.

I've moved your post to an existing discussion around Capacity Tier capabilities introduced with B&R 9.5 U4 release worth checking.

Answering your question, no it is not possible to target blob storage using copy jobs. Object Storage integration is a part of Capacity Tier of Scale-Out Backup Repositories. There is a post above by Gostev explaining the purpose of this feature.

Thanks and let us know if any additional questions arise,
Fedor

brekus
Lurker
Posts: 1
Liked: never
Joined: May 24, 2011 3:18 pm
Full Name: Brian Rekus

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by brekus » Mar 16, 2019 1:41 am

Hello all,

Quick question about the capacity tier going out to Azure. When a chain is closed, and that full and incrementals are sent out to azure, if there is already a full out there is Veeam/Azure smart enough to know if some blocks are already out and haven't changed? Out bandwidth out isn't the greatest so I am hoping that it's smart enough to know whats already out there.

Thanks!

veremin
Product Manager
Posts: 16880
Liked: 1430 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by veremin » Mar 19, 2019 6:37 pm

Correct, identical blocks won't be sent second time. Thanks!

Manuji
Novice
Posts: 9
Liked: never
Joined: Apr 04, 2019 3:40 pm
Full Name: Girish
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by Manuji » Apr 04, 2019 3:56 pm

Hello Team,
New to Veeam. I have a quick question on Veeam B&R and azure blob storage. If I create a copy on prem and want to duplicate it to the azure blob storage (D2D2C) all from one job. is it possible?
Secondly is Veeam B&R compatible with hot,cool & archival tier in azure?

veremin
Product Manager
Posts: 16880
Liked: 1430 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by veremin » Apr 04, 2019 6:34 pm 1 person likes this post

If I create a copy on prem and want to duplicate it to the azure blob storage (D2D2C) all from one job. is it possible?
Copy policy is not included in Capacity Tier today, but we're working on it already. For now you can specify a certain period of time (operational restore window), after which old backups belonging to sealed backup chain will be moved to object storage.
Secondly is Veeam B&R compatible with hot,cool & archival tier in azure?
Only hot and cool access tiers are supported.

Thanks!

Manuji
Novice
Posts: 9
Liked: never
Joined: Apr 04, 2019 3:40 pm
Full Name: Girish
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by Manuji » Apr 04, 2019 8:15 pm

So if I understand this if I create a full b/up and I am part SOBR I cannot create copy job of this full b/up to azure blob storage BUT as of now I can only specify when this full b/u is say 20 day old move it to Azure storage. Correct?

Gostev
SVP, Product Management
Posts: 24787
Liked: 3519 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by Gostev » Apr 04, 2019 8:33 pm 1 person likes this post

Correct. Think about it as a way to extend capacity of the particular repository infinitely. Capacity Tier does not give you a different repository, but rather merely extends capacity of an existing one.

vchris
Novice
Posts: 6
Liked: never
Joined: Feb 01, 2013 12:56 pm
Full Name: vChris
Location: Suisse romande
Contact:

Re: 9.5 Update 4 - Azure Blob Storage

Post by vchris » May 06, 2019 2:21 pm

Gostev wrote:
Jan 25, 2019 4:23 pm
It is unlikely that immutable storage will ever be supported for Capacity Tier due to its WORM nature. There are just too many issues around "online storage" use case with most basic stuff that you don't immediately realize, like inability to cleanup incomplete or failed writes even.

However, we will certainly consider immutability for possible future Archive Tier, which is a different use case from Capacity Tier that will likely have vastly different integration approach - due to Glacier-type storage peculiarities and those issues enabling immutability brings. Which also means that will lose most of the benefits of Capacity Tier, and it will no longer be "beautiful" ;) limiting you to true archival scenarios (essentially making it an off-site tape archive replacement).
Hello Gostev, I was trying to use the immutability, time retention not legal hold, mechanisms to avoid the "IT went rogue" on a short interval (beside of physical tapes that stay the better protection into a safe not handled by IT).

It’s a scenario I prepare for a Veeam On Tour.

The setup/objective was to use it on Backup Copy - GFS with 5-7 points, points older than 1 week go to “Capacity Tier”, that way the "recursive" chain stay on the "Perf tier" and the weekly (5 weekly setup) go to the "Capacity Tier" and stay protected from deletion for 4 weeks with immutability “Time retention”.

It does work but we have sometime a cleanup warning (because of immutability immediately activated, I wish we could delay the lock for like one hour after the last modification as some other solutions offer).

Does that usage make any sense to you ?

Of course the futur “Archive Tier” you speak about make more sense, to replace tape out and use WORM like policy, but here the objective is to protect even short term retention outside the Archive Tier.

(On the side I’ve setup StarWinds VTL with Cloud Replication and immutability, and of course it does work well).

Gostev
SVP, Product Management
Posts: 24787
Liked: 3519 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by Gostev » May 06, 2019 3:47 pm

Unfortunately, I am not well aware of technical details Azure implementation of immutability. But I know it's not that simple and straight-forward with Amazon, and none of the above will provide the desired mechanisms to avoid the "IT went rogue". So, it needs to be researched much deeper.

For example, with Amazon object-level immutability is more of a marketing term, in reality what they sell behind this term is regular object versioning. This in turn means, for example, that you still CAN delete immutable objects through S3 API (resulting in an object ending up in a special state with the deleted flag). More importantly though, hackers or malicious IT folks (or ransomware) can still freely overwrite immutable blocks with some rubbish content. And while again, the immutable version of the block will still be preserved "somewhere", restores will fail because they will be using the latest version of the block (with rubbish content).

The good news is that contrary to my earlier response you're quoting, we did seem to find a way to potentially use the object-level immutability feature (aka PUT Object retention) with the Capacity Tier. We're still largely in the design phase though, so it's a bit too early to talk about details. Also, it's currently supported by Amazon S3 only (Azure blob storage only supports container-level immutability that is not fully compatible with our Capacity Tier architecture, as per the warnings you're seeing).

vchris
Novice
Posts: 6
Liked: never
Joined: Feb 01, 2013 12:56 pm
Full Name: vChris
Location: Suisse romande
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by vchris » May 07, 2019 2:13 pm

Thank you for the updates :-D

mcz
Expert
Posts: 296
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by mcz » May 15, 2019 9:09 am

Gostev wrote:
Feb 25, 2019 8:05 pm
For example, with regular object storage today, we're able to offload backups in forever-incremental manner, effectively providing source-side dedupe between multiple full backups. Also, we're able to perform block-level retention to immediately and granularly remove any unused blocks. This is possible because we're able to update metadata sitting in object storage in real time.
Anton, simple question regarding deduplication on object storage: Is deduplication only performed between the same chain/vm or is it performed across the whole repository? Let's assume that we've got 5 vm's, backed up by 2 different tasks and multiple fulls.

Now I guess that veeam is performing deduplication for at least the same vm in the same chain. But what about other vm's from a different jobs / different chains? Does deduplication across the whole object storage take place or will it only be limited to the same vm/chain?

Thanks for the clarification!

Gostev
SVP, Product Management
Posts: 24787
Liked: 3519 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by Gostev » May 17, 2019 8:05 am

@mcz deduplication is within the same chain, so the answer depends whether you're using single or per-VM backup files with your primary jobs. Thanks!

mcz
Expert
Posts: 296
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by mcz » May 17, 2019 9:10 am

Thanks for the answer, Anton. Yes, we're using per-VM backup files, so we're not benefiting that much from deduplication in this case. I assume that things will change when the next update will bring us the functionality to do direct backups & copy jobs to object storage. There I would expect to have the option to switch to "all-in-one" as we can currently using local repos.

Am I right?

Gostev
SVP, Product Management
Posts: 24787
Liked: 3519 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by Gostev » May 17, 2019 9:25 am

Typically over 90% of deduplication benefits come from dedupe between multiple fulls (periodic and GFS), whereas deduplication across VMs is minimal in the bigger picture. I mean, you can make the numbers looks really good in a POC with freshly deployed VMs, but after just a few month of OS updates VMs will start to deviate so much that you will see very little benefits in the long run. Because of that, I personally would stick to per-VM chains.

WasabiDan
Lurker
Posts: 1
Liked: 5 times
Joined: May 22, 2019 6:49 pm
Full Name: Dan Keldsen
Contact:

Re: 9.5 Update 4 - Azure Blob Storage

Post by WasabiDan » May 22, 2019 6:56 pm 5 people like this post

Gostev wrote:
Jan 31, 2019 4:02 pm
It's not really there, look at its size ;) what is left on NAS is a VBK stub containing metadata only - this serves as a local cache, for performance considerations and to reduce the number of API calls which object storage providers often charge extra for.
FYI - Dan from Wasabi.com here - as of this week at VeeamOn we are freshly minted in the "Veeam Ready Program for Object Storage" - and for those who don't know Wasabi yet, not only do we not charge for API calls, we don't charge for egress either, just a flat price per gigabyte of S3 API-compatible storage. We're now 2 years into our publicly available cloud storage, and we heard early on in the market that microcharges were the devil. (or words to that effect ;) )

From an engineering elegance perspective, I love the Veeam approach of a VBK stub as a local cache (and we've found many software solutions that are incredibly... lazy/inefficient in spewing API requests into the cloud). But for cost purposes with cloud storage, no microcharges to be found with Wasabi.

More details on our joint offering at:
https://wasabi.com/veeam-partner-solution/

yasuda
Enthusiast
Posts: 63
Liked: 10 times
Joined: May 15, 2014 3:29 pm
Full Name: Peter Yasuda
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by yasuda » May 29, 2019 5:15 pm

Hi Dan, do you have any comment on the previous discussion of immutable storage? Is Wasabi's immutable storage diferent from "...Amazon object-level immutability is more of a marketing term, in reality what they sell behind this term is regular object versioning..." ?

dw432948jkk
Novice
Posts: 8
Liked: 1 time
Joined: Sep 05, 2019 8:26 am
Full Name: Peter Müller
Contact:

[MERGED] When will veeam offer "real" cloud-backup-integration?

Post by dw432948jkk » Sep 05, 2019 8:31 am

We are currently deciding on which back-up solution we will use. Although we like veeam better in on-premise backup, there is unfortunately no integrated cloud back-up solution yet. However, for a few months now, this connection has been offered by Altaro, which we also like, with the direct integration with Azure blobs as offsite backup.

There is always the request that veeam should offer this integrated. A few months ago, it was said that veeam is working on it and that it should be released in the fall.

Will there be this direct connection at short notice and if so which providers are supported (Azure, AWS an GCS would be great)?

Otherwise, we would probably soon decide for Altaro and would then be away as a potential customer.

wishr
Veeam Software
Posts: 1148
Liked: 117 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: When will veeam offer "real" cloud-backup-integration?

Post by wishr » Sep 05, 2019 10:29 am

Hi Peter,

Welcome to Veeam Community Forums and thanks for posting!

Please take a look at this and this posts as they should answer some of your questions. I've merged your post with the corresponding topic.

We already have a private beta of the next B&R version so you may jump in and give this functionality a try - please reach out to your local Veeam sales representatives to get access to it.

Regarding compatible providers, please check that thread.

Thanks

dw432948jkk
Novice
Posts: 8
Liked: 1 time
Joined: Sep 05, 2019 8:26 am
Full Name: Peter Müller
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by dw432948jkk » Sep 09, 2019 11:48 am

Ah, ok, that sounds good. I'll take a look at that.

Will existing customers with instance licenses also get a "free" upgrade to version 10?

wishr
Veeam Software
Posts: 1148
Liked: 117 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by wishr » Sep 09, 2019 12:24 pm

Hi Peter,

Sure, as long as your subscription is still valid when v10 is released.

Thanks

dw432948jkk
Novice
Posts: 8
Liked: 1 time
Joined: Sep 05, 2019 8:26 am
Full Name: Peter Müller
Contact:

Re: 9.5 Update 4 and Microsoft Azure Blob Storage

Post by dw432948jkk » Sep 09, 2019 1:36 pm

How long will it take for version 10 to be released?

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest