Immutable Backup on Azure Blob (immutable blob storage)

[awatkins@akdmc.com]

We are currently using enterprise to backup a number of physical and Virtual servers to a NAS. These backups are not protected from ransomware, so I've been reading up about immutable backups. I have seen some posts regarding AWS, but does anyone know if it is available for Azure yet? I know Azure immutable blob storage is available, but can't find anything on using it with Veeam. Been waiting for a definitive reply from Veeam, but getting tired of waiting. Thanks

[Gostev]

This functionality is not available with Azure, because blob storage does not support object-level locking yet. Thanks!

[awatkins@akdmc.com]

Thanks Gostev. Since Azure blob storage is not supported. Is there any other way of taking a backup to Azure that is protected from Ransomware? (Other than backing off to WORM).

[Gostev]

The only option I see is using Backup to Tape job to a VTL backed by an Azure blob storage container with time-based retention lock policy enabled.

[helpdesk@gmb.eu]

Isn't this what we are looking for, or doesn't Microsoft Azure support integration with Veeam B&R ?
https://github.com/MicrosoftDocs/azure- ... storage.md

It was one off the most desired features I was looking for in the presentation yesterday with the V10 release

[dalbertson]

That is container (bucket) level and not blob (object) level.

[Gostev]

Container-level immutability is designed as an offline tape replacement for "VTL to Cloud" use case. Where what you do is classic incremental backup with periodic fulls (because this is how tape backup works), and thus are able to "seal" the entire backup file once it is copied over - as it will never be changed or re-used again.

Veeam's object storage integration is far more advanced than that. Our design goal from the start was to AVOID the need for periodic fulls to cloud. Not only it gets super expensive quickly from cloud storage perspective to store multiple fulls, but also for the vast majority of our customers, periodic fulls to cloud are simply impossible based on their environment size and available bandwidth.

This is exactly the reason why our object storage offload engine is forever-incremental, or in other words we do source-side deduplication. When it's time to offload a restore point contained in a periodic full backup, we still offload only blocks that are unique to this restore point, while the rest of blocks are simply "referenced" to the existing blocks already in object storage.

Now, for obvious reasons, implementing immutability in such circumstances requires controlling locking on individual object level, because the same object can be a part of different restore points, and has to be re-locked periodically when it's time to remove the oldest restore points that it is included in.

Hope it makes sense! And it's just a question of time - I'm sure Azure will eventually catch up and offer object-level locking as well. Amazon just had a good head start

[mcz]

Anton, now you've made a good prediction that Azure would catch up soon regarding object-level locking. Any idea if the same would or could happen to wasabi in the near future?

[dalbertson]

@mcz we could ask them for an answer directly.

@vbelagodu @wasabi-jim Can you elaborate on if and when you will support immutability.

[vbelagodu]

@mcz, we are actively working on our object lock support. We are anticipating this feature would be supported before end of Q2 2020. We have existing support for immutability which works a bit different compared to object lock, we have a significant customer base who are using Wasabi's existing design. We are working with our product & engineering team to ensure Veeam's implementation of object lock and Wasabi's immutability feature can co-live as stand-alone features, though the features overlap in its ability.

With existing Wasabi's immutability support - you can achieve similar effect of your backups, you could write Veeam backups to an immutable Wasabi bucket - You may have to work the period you need the objects locked in a bucket. We'd be happy to work with you to come up with a solution that would work for your storage needs. Please reach out to support@wasabi.com for further dialog on this matter.

[dalbertson]

@vbelagodu Thanks for that. I will work with Vilas and the Wasabi team on testing their object lock feature when it is ready so that we can list it as such on the Veeam Ready list.

[mcz]

Greate job guys, thanks for the quick answer and you're really doing a very very good job here! As Anton Gostev has pointed out during v10 launch, object lock could and will be the best alternative compared to tapes - which is something I was always waiting for. So I assume that there will be some kind of newsletter or any other notification when the functionality is ready for production...

Thanks!

[dalbertson]

It will be announced via normal marketing channels. I’m sure the wasabi team will make some noise as well. FYI if you are going to VeeamON make sure to stop and see the wasabi team there

[vrykodee]

https://github.com/MicrosoftDocs/azure- ... -manage.md

Immutable storage is available on Azure and this above link is how to implement/manage it?

Hope this helps ...
David

[dalbertson]

@vrykodee Thats still talking about container level immutability and not blob level immutability

[mcz]

@vbelagodu

Hi Vilas,

I'm curious and can't wait for immutability... Is it still planned to get released in Q2, are we close to a release or will it last a little bit longer?

Thanks!

[mcz]

...just saw that my message to Vilas isn't highlighted in red, which probably is the reason why he doesn't get a notification. How to achive that, can please someone give me some advice?

Thanks!

[oleg.feoktistov]

Just insert @ and write his username. It should become available from the list of all usernames and then enclosed in mention tag as soon as you click on it.

[mcz]

Thanks Oleg for helping! When I insert @ and type (or even insert) the username, I don't get any list to click on. Can you please try that for me or did I miss something? Thanks!

[veremin]

You won't see this in quick post editor, to access it click "edit & preview" button.

In quick editor use forum codes to mention particular individual:

Code: Select all

[mention]nickname[/mention]

Thanks!

[mcz]

Ah, thanks! Now I get the error message:

You cannot use certain BBCodes: mention. Looks like that it's limited to veeam-stuff? Can you please post the mention-code for me?

[veremin]

Done!

[Gostev]

Please note that mentions are enabled for use by green nicknames only.

[Slowrider5]

Does anybody know if the Azure Soft Delete option is a feature that could be used to mitigate the lack of immutability for the time being? I get it's not truly immutable because it doesn't protect against the storage account or entire container being deleted. But it's not clear if all the blobs in a container can be recovered by restoring a snapshot all at once, or if you can only recover blob by blob (which would be impractical for the number of blobs that Veeam is storing).

[bwoodard]

We considered Azure Soft Delete as an option. Something to be aware of is if the account either has the ability to delete the entire storage account or container, or has the ability to disable soft delete, it has the ability to permanently delete the data. You can do this by disabling soft delete, recovering the data (bringing it out of soft delete status), and then deleting the data. See MS docs here:

Permanently deleting soft deleted backup items
Backup data in soft deleted state prior disabling this feature, will remain in soft deleted state. If you wish to permanently delete these immediately, then undelete and delete them again to get permanently deleted.
(https://docs.microsoft.com/en-us/azure/ ... ture-cloud)

Provided that you are able to secure these permissions satisfactorily behind protected accounts (including Veeam svc account) this could be relatively viable, but it will at least 2x your retention costs since soft deleted data still consumes storage space and is billed at the same rate as standard data.

[jim.lowry]

I'm not sure about the specific Azure Soft Delete option itself, but it sounds like you are looking for some sort of permanent deletion protection. You can use the insider protection feature on the VBR and VCC repos. See here: https://helpcenter.veeam.com/docs/backu ... ml?ver=100.

While it doesn't protect from all ransomware, the files can still be encrypted if ransomware gets to it, it does protect against someone deleting something permanently. It will retain files in a "recycle bin" for the amount of time you specify. So less sophisticated ransomware, that doesn't look for a recycle bin is protected.

Another good option is air gapped tape arrays. Turn off the tape (or VTL) after the job completes or at least remove it from the network. It's old school but reliable.

[JZTOR]

Hello Veeam experts.
Please let me know if you are aware of any limitations or considerations while using the Immutable Azure object storage for the Veeam Scale-out repository.
Also, please advise if there is difference from Veeam perspective with using Immutable Azure vs AWS object storage as a backup target.
Thank you

[jim.lowry]

@JZTOR

Currently using immutable object storage in AZURE is not supported. The API set for BLOB does not do object level immutability, but only container. We do not support container based immutability as we require the more granular control of object level immutability. When Microsoft adds the object level API, we will test, validate, and update the documentation and various forum posts.

AWS is fully supported because the S3 API supports object based immutability. Technically, any object storage/vendor using the S3 API set that includes object level immutability is fully supported (not all S3 deployments use the specific API set needed). To see a complete list of currently support object vendors and what vendors have verified with us immutability support, see here: object-storage-f52/unoffizial-compatibi ... 56956.html

[ortec-RW]

Sorry to keep bothering everyone with my Immutability quest.

So, AWS is the only one supporting the S3 Object locking API with Veeam. Azure and some others only support it at the container level. Which made me start thinking......

I can set up a container in Azure with immutability, and not control it via Veeam using the API. I fill it up using Veeam with 1-full+forever-incrementals, just like it's a normal cloud repository. The chain of backups in this container is safe using immutability and for as long as I set it to be. Azure controls it, not Veeam.

It's only in my head now. Purely hypothetical. Will this work ?

[ortec-RW]

I think my brain already farted.

Tried this just now, but the first Offload to Cloud on the scale-out repository to Azure already fails because it does detect that the Blob storage container is set to immutable. So you cannot fool Veeam into controlling the immutability on the container level.

Kinda makes sense I guess.

So back to AWS we go.