Test restore from Azure object storage

[Joel1298]

Hello, I have a VBR server with a backup job targeting local storage and a backup copy job targeting Azure blob storage. How can I set up a secondary VBR server for testing restores from the Azure blob container?
I attempted to add the repository to another server but stopped at the message indicating that the repository is controlled by another VBR server. Another post seemed to suggest putting the repository in maintenance mode but the documentation suggests this is only applicable to scale-out backup repositories.
What is the correct method to test restores without damaging current backup data?

[Ivan239]

If at the time of changing the owner of the repository there are no jobs running on it, the current data will not be damaged.
All subsequent job runs from the first VBR server will simply fail immediately after launch.
Putting the repository in maintenance mode before changing ownership guarantees that there will be no running jobs on it.

Disabling all jobs targeted at this repository should be an acceptable alternative to maintenance mode for this case.

[Joel1298]

When I try to add the repository to a secondary VBR server to test a disaster recovery scenario I get this message:
"Selected object storage repository is already managed by another backup server. If you continue, all jobs currently using this repository will fail."
Are you saying that if I disable the jobs on the production VBR server I will be able to add the repository to the test VBR server without messing anything up? Then perform my test restore and resume the jobs on the production server?
I don't necessarily want to change the owner, I just want to perform a test restore and then let the production server continue its backups as usual.

[Ivan239]

Changing the repository owner is a mechanism designed to protect against concurrent data modifications, which can lead to data corruption. For any interaction with the repository from the second VBR, it is necessary to take ownership of it. Once the tests are completed, it will be necessary to go through the add/change repository wizard again on the side of the first VBR to return ownership to it. As long as only one VBR is working with the repository at any given time, the data on it will be fine (as long as it’s not being deleted, of course )

[Joel1298]

What is the mechanism for changing the owner? Do I just ignore the above warning and continue adding the repository to change the owner? Then I re-add the repository to the production server?

[Ivan239]

"What is the mechanism for changing the owner? Do I just ignore the above warning and continue adding the repository to change the owner?"

Yes, this is what changes the owner of the repository. I also want to emphasize that I am sharing information here because I know how direct backup to object storage works. But this does not mean that the described scenario is formally supported. In case of any doubt, it is best to contact support for verified instructions

[apolloxm]

I just checked the my azure backup repositories, there is no maintenance mode as we are just used the simple repository instead of the scale-out repositories. We have configured the backup copy jobs directly to upload the data to azure immutable storage. Should I disable those jobs before I changes the owner of the repository? so the process would be as following:
1.disabled the related the jobs which upload to azure or put the repository into maintenance mode (if have the maintenance mode)
2.changes the owner of the repository into new server
3.Testing on the new server
4.after finishing the testing, changes the owner of the repository into old server
5.enable the jobs on the old server

[Mildur]

Hi all

Our user guide has a new guideline on how to test "direct to object storage" on a second backup server.
You must use credentials with read only permissions to connect a second backup server to your object storage repository.
This will allow the connection without interfering with the owner status or objects on the repository.
--> This is a supported scenario.

Requirements:
- object storage credentials with read-only permissions
- direct to object storage repository

1.) Create a second user/access keys with ready only policy
2.) Connect your object storage bucket/azure storage container to the second backup server

Best,
Fabian
____________________

Userguide:
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

You can add an object storage repository to a second backup server using credentials with the read-only access permissions that allows you to perform data recovery options. If you use credentials with full-access permissions, it will lead to unpredictable behavior and data loss.

IMPORTANT - Consider the following:

This option works for object storage repositories only if they meet the following requirements:
Capacity/Archive Tier:

• You plan to add these object storage repositories as a capacity or archive extent of a scale-out backup repository.

• The object storage repositories do not have data encryption enabled. If encryption is enabled on these repositories, you will not be able to add object storage repositories using credentials with read-only permissions.

Direct To Object Storage:

• You can use this option for direct backup object storage repositories added either as a standalone repository or a performance extent of a scale-out backup repository.

[apolloxm]

Hello Fabian,

Thanks for your reply!Can you help to confirm that Veeam only support access key to login to azure? We had opened a ticket Microsoft and got information from Microsoft said that Access Key doesn't support ready only permission.(access key had the full permission), if that, how we can test restore from azure object storage without any data loss?

[Mildur]

Hi Apollo

I answered in the other topic (please don‘t do cross posting your question )

V12.1 supports now „ Microsoft Azure Storage Accounts (Entra ID)“ as well.

https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Try „Storage Blob Data Reader“ role for the account instead of „Storage Blob Data Owner“.

Best,
Fabian

[apolloxm]

please don‘t do cross posting your question
sorry for that. I will not do that in the future

Try „Storage Blob Data Reader“ role for the account instead of „Storage Blob Data Owner“.
do you means that create a new storage account?

access key only had the full permission as following article
https://learn.microsoft.com/en-us/azure ... ccess-keys

Can I still use the old process?
1.disabled the related the jobs which upload to azure or put the repository into maintenance mode (if have the maintenance mode)
2.changes the owner of the repository into new server
3.Testing on the new server
4.after finishing the testing, changes the owner of the repository into old server
5.enable the jobs on the old server

[Mildur]

Yes, you can still use the old process. But it won't be officially supported.

put the repository into maintenance mode (if have the maintenance mode)

Maintenance mode only exists for capacity and archive tier.
Please be aware, capacity/archive tier with encryption enabled (encryption that is set on the Capacity Tier step of the SOBR wizard) may lead to backup job issues on the primary backup server. With encryption enabled, each backup server has to write their own encryption keys to the object storage. We had support cases in v12 because customers tested encrypted capacity tier on a second backup server and backup jobs stopped working.

do you means that create a new storage account?

Correct. This should work.

1.) Register a new azure application with a security certificate for authentication and assign only the read only role:
- Storage Blob Data Reader

2.) On the secondary backup server add a new <Microsoft Azure Entra ID storage account> to the configuration. Choose <Use the existing account> and provide your tenant and application ID (from the manually registered application):
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

3.) Connect Azure Blob to the second VBR and specify your <Microsoft Azure Entra ID storage account>.
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Best,
Fabian

[apolloxm]

Hi Fabian,

Thanks for your reply! our veeam server in our production was V12. Since the Microsoft Azure Entra ID was supported started from V12.1. I need to install Veeam V12.1 on the secondary backup server. Is that OK as the production Veeam server and secondary backup server are in difference version?

[Mildur]

A backup server with a higher build number should work, because nothing changes on the repository with read-only permission.
But to lower the risk in case of a permission misconfigurations, I recommend to keep both server on the same build level.

Best,
Fabian

[Joel1298]

Is this process actually documented somewhere? There seems to be a lot of guess work above and I'm looking for a more official or polished steps to follow to complete a test disaster recovery restore.
Here is where I'm at so far. I begin by adding the repository to a new VBR server. I begin adding a new Azure object storage repository and choose the Entra ID option for the credentials and choose create a new account and complete the device login. Then before moving forward with adding the repository I go in to my Azure portal and remove the owner and data contributor roles for the newly added Veeam app registration and add reader and blob storage reader roles instead. Then I continue in Veeam and I am able to select the container and folder but when I attempt to click next I get an authorization error. "Failed to get Azure container immutability config. [AuthorizationPermissionMismatch]."
What permissions do I need to add to allow my testing VBR server to add the repository with read only access? I don't see this documented anywhere.

[apolloxm]

I would like to ask Veeam to release a kb article for those process. I opened a ticket to Veeam about this issue, the support team also don't know what is real process..........07170632..............

[Mildur]

It works for S3 with read only IAM policies in our lab (tested with AWS S3). Let me run a test with Azure as well.

Best,
Fabian

[apolloxm]

Hello Fabian,

Thanks for your information! Is it work with Azure?

[Mildur]

Hi all

Our QA team will do a test if we can support Azure read only accounts or not.

Best,
Fabian

[apolloxm]

Hello Fabian,

Please post the result if you have any update from your QA team.Thanks!

[Mildur]

Hi all

Our QA team finished their testing.
Unfortunately it is not possible with Azure to create a read only policy. All three permissions are required to do the initial connection.

I will ask our user guide team to update the user guide with this limitation:
- Read-Only is possible with AWS S3 or S3 compatible, but not with Azure Blob.

And we added it as a requirement for one of the next versions to have it possible for Azure as well.

Therefore please use the method provided by @Ivan239 for now:

Changing the repository owner is a mechanism designed to protect against concurrent data modifications, which can lead to data corruption. For any interaction with the repository from the second VBR, it is necessary to take ownership of it. Once the tests are completed, it will be necessary to go through the add/change repository wizard again on the side of the first VBR to return ownership to it. As long as only one VBR is working with the repository at any given time, the data on it will be fine (as long as it’s not being deleted, of course )

Best,
Fabian

[apolloxm]

Hello Fabian,

Would you please also let your guide team update the guide how to do this disaster recovery with azure to avoid any misunderstanding. Thanks!

[Mildur]

Yes, I will do so.

Best,
Fabian

[Joel1298]

I would like to express my dissatisfaction with this proposed resolution. It is imperative that we are able to fully test our recovery environments without impacting our production environment. I don't think we should have to take down our production backup system to run our testing.

[Mildur]

For direct to object storage, you don‘t have to „take down your production backup system“.

1) Make sure no job is running against this Azure Blob repository
2) connect your Azure blob to the second backup server
3) go back to the production backup server and click through the repository properties to get back ownership
4) backup jobs can be started again if required
5) Start testing on the secondary backup server

All this can be done within 5 minutes and shouldn‘t impact your normal job schedule.

It’s different for Capacity Tier or Archive Tier. Here we always recommended to put on the maintenance mode first while testing.

We have object storage testing with read only credentials listed on our roadmap for one of the next versions. For both scenarios, direct to and as part of a capacity tier or archive tier.

Best,
Fabian

[apolloxm]

I will ask our user guide team to update the user guide with this limitation:
- Read-Only is possible with AWS S3 or S3 compatible, but not with Azure Blob.

it looks like your guide team didn't update the user guide with this limitation. and they didn't tell user how to restore the data from Azure object storage.

[Mildur]

Hi Apollo

We will update the user guide with the limitation „read-only doesn‘t work for Azure“.
As discussed here and confirmed by our QA team, Azure cannot use read-only accounts. And connecting a repository from multiple backup server with write permission is not supported by us. We cannot document unsupported methods in our user guide.

Please use the unsupported steps Ivan posted earlier in this topic. It won‘t make it to the user guide, but it has proven itself workable for many customers.

Best,
Fabian

[apolloxm]

Hi Fabian,

if this is unsupported steps. what is the supported steps? We would like to have an official supported steps to guide customer how to restore from Azure object storage.

This is a normal requesting as a customer need to test the disaster recovery process from azure immutable.

[Mildur]

Your request is noted. We know the importance of this request.
But it requires first development on the product side. At the moment I cannot provide an ETA when we can deliver this update.

Thank you,
Fabian

[Mildur]

Hi all

The user guide is now updated with the limitation for Azure:
https://helpcenter.veeam.com/docs/backu ... ml?ver=120

Best,
Fabian