Official word on S3 object lock (immutable) repositories

[HangTen]

Hi!

I know that according to the documentation, "Veeam Backup for Microsoft 365 does not support object lock for the S3 Compatible bucket" but I set up a test using Wasabi S3 object lock for a repository and the jobs that are targeting it seem to be working. Is this a case of it possibly working but not supported by Veeam (i.e. one can use it but at one's own risk)?

[Mildur]

Hi Hin

Just because it works, doesn't mean it is supported.
If our product needs to remove or change an object which is immutable, and it can't, it can lead to many issues. Our support will not be able to help you in that case.

Official immutable support is on our road map for a future release, but we cannot comment on any ETA yet.

Thanks
Fabian

[Mike Resseler]

Hi Hang,

The problem is on how we write away the data and update the metadata. At a certain point in time, VB365 will report that it cannot make modifications (because of the modifications are not allowed) As Mildur said, we are working on the secondary repository (backup copy repository) support for immutability

[HangTen]

Thank you Fabian and Mike. This is the answer I need. It might "work" but is not officially supported.

[Regnor]

Hi Hang,

The problem is on how we write away the data and update the metadata. At a certain point in time, VB365 will report that it cannot make modifications (because of the modifications are not allowed) As Mildur said, we are working on the secondary repository (backup copy repository) support for immutability

Sorry for hijacking this topic.
With object locking, the bucket also needs to support versioning. So shouldn't it be possible to create new versions of the meta data objects, while not touching the backup objects?
Or even if you need to modify those, the original backup stays in the bucket as version 1.
Just an idea with my rather simple knowledge of S3 object storage

[Mike Resseler]

Regnor,

The thing is that VB365 needs to be aware of that, otherwise the DBs that knows of the objects will go crazy . We will have it in the next version

[Regnor]

@Mike: I know that it won't work currently, this is just how I would try to solve that
(as an alternative of using backup copy)

[dhayes16]

Hello All...Until some form of immutabilty is available is it still recommended to load VBO into a VM and backup the VM via VBR and send to something like a Linux Hardened Repo or via a SOBR to Object? Also, if VBR backs up the VBO VM in this way could there be a problem if the VBO backup is running and VBR comes along and wants to back it up?

[Mildur]

Hi Dave

If you want to have immutability, then that‘s the way for now.

Also, if VBR backs up the VBO VM in this way could there be a problem if the VBO backup is running and VBR comes along and wants to back it up?

No, we use vss to have a consistent state of the vb365 jet databases (adb files). Also the configuration database will be backed up.

You can see the vss writer with vssadmin.
Veeam Backup for Microsoft 365 Writer

Thanks
Fabian

[dhayes16]

Once again.. Thank you Mildur!! . Very much appreciate your response. Honestly immutability is high on my list these days so we will continue this method. Thanks.

[crun]

Hi

Is there any ETA for VBO365 release which would support immutability?
Do you plan the object locking implementation for all s3-compatible storages or only some selected flavours? Can we expect it to support Azure blobs too or this would come later?
Can we expect it on backup & copy jobs? (Mike mentioned only the backup copy)

[Mildur]

Hi Crun

It will be added in V7, which is planned for early Q1 2023.

Do you plan the object locking implementation for all s3-compatible storages or only some selected flavours?

Yes. All S3 object lock enabled object storages should be supported. Please check our compatibility lists later when we have v7 released.

Can we expect it to support Azure blobs too or this would come later?

Yes, we plan to support immutability for Azure blobs in V7.

Can we expect it on backup & copy jobs? (Mike mentioned only the backup copy)

Only backup copy jobs will be immutable.

Best,
Fabian

[crun]

Thanks Fabian. Three more questions:

1) Do you have backup job immutability on the roadmap?
With object storage we need s3-compatible (or blob) in both backup & copy. Only copy would be immutable
If we stick to VBR backing up VBO365 basically we can have both copies immutable

2) Less related, but is there any plan to have a better performing migration tool of VBO365 data into object storage? Or one that can concurrently continue to take increments while migrating. This topic seems full of horror stories
veeam-backup-for-microsoft-365-f47/migr ... 6ce9da8953

3) Any chance (probably not) to schedule backup copies from a non-extended standard repository?

[Mildur]

1) Do you have backup job immutability on the roadmap?

Nothing which we can share today. But we always consider such improvements for a future version of VB365.

2) Less related, but is there any plan to have a better performing migration tool of VBO365 data into object storage?

I suggest continuing this question in that other topic. So we can keep this topic about immutability.

3) Any chance (probably not) to schedule backup copies from a non-extended standard repository?

Chances for that request are rather low.

Best,
Fabian

[cerberus]

I can't wrap my head around why S3 object lock will only be available for backup copy jobs, wouldn't it make sense to have this on the main backup job itself?

Trying to understand what the driver is behind supporting such a critical functionality (in terms of ransomware protection) on the backup copy job only, I am sure there is a logical explanation for this - do most folks that use VBO run backup AND backup copy jobs?

Purely from a cost perspective, enabling this on backup copy job adds additional cost for S3 storage, you are essentially having to double up the space to use this feature.

[Gostev]

From what I heard the reasons are purely technical. Primary backup format design will need to be changed to support immutability, which will require some time to be implemented. Immutable backup copies are easier because copy jobs work on an already known set of data, which is why they are the first step.

Purely from a cost perspective, enabling this on backup copy job adds additional cost for S3 storage, you are essentially having to double up the space to use this feature.

But it's not like you can avoid having a second offsite copy of your backups regardless? Otherwise you won't be compliant with the 3-2-1 backup rule, which is the biggest sin there is for a backup admin.

[cerberus]

Agreed, there definitely needs to be another job to meet the 3-2-1 backup rule.

Was going to store a copy locally (local repo on the local server storage) in addition to offloading to S3. We will need to re-think this part a bit.

[Mike Resseler]

@cerberus Unless I misunderstand it, but from a Jet DB repository doing a copy to S3 will not work. We support backup copy job from any object storage to any object storage, but not from Jet DB repository

[SnakeSK]

Mike you support backup copies only to commercial providers. Those of us who have onprem OSR or offsite OSR even with immutability will be left out even if you implement the feature.

[Mildur]

Hi Snake

You can use any Object Storage in V7 as a target for backup copies. Just see the discussion above and my comment. It doesn‘t have to be a commercial provider or archive object storage.

Thanks
Fabian

[HangTen]

Hi!

I know that according to the documentation, "Veeam Backup for Microsoft 365 does not support object lock for the S3 Compatible bucket" but I set up a test using Wasabi S3 object lock for a repository and the jobs that are targeting it seem to be working. Is this a case of it possibly working but not supported by Veeam (i.e. one can use it but at one's own risk)?

Here's an interesting follow-up to this test. I upgraded the server that housed this backup on the unsupported repository to v7 and recently wanted to see if I could migrate all the backups on there to a new v7 server. I thought it would be a simple procedure to just stop the proxy service, copy the metadata folder to the new server, and create the repository on the new server pointing to the same buckets and once the organization is configured, all the historical data would be there. Turns out v7 has locked down this unsupported repository and it can't be added. "Select a bucket with disabled blob versioning". So the workaround was to install v6, create the repository pointing to the copied metadata folder, and then upgrade to v7. All of that worked and historical data is visible.

To clarify, this was JUST A TEST and not being used in production.

[Polina]

Hi Hin,

In v7, we blocked adding new non-immutable repositories with enabled versioning to prevent potential problems with backup data consistency and retention. The already existing repositories continue to work, however, we strongly recommend disabling versioning there. Migrating backups between versioned and non-versioned buckets may cause data corruption and it is not supported.

In the next few days, we're going to publish a KB explaining how to handle such cases.