Veeam 12 SOBR with VMware and Wasabi - restore testing issues and questions

[dcm-ebb]

For today's fun, we are visiting the axiom that "for every expert, there is an equal and opposite expert" which has led me to my current operational mode of documenting and then testing out every step of the documentation before claiming I am any where near being an expert.

For my scenario, I have a vCenter version 6.5 appliance, and two ESXi 6.0 hosts. Due to the very slow nature of getting upgrade equipment installed at this remote site, that is what I am faced with supporting for now. Backups for about six months went from Veeam 11 to a local file on a backup repository on a RAID array on the same ESXi 6.0 server as the Veeam and file server VM, and then via backup copy job to a datastore on a USB hard drive that is attached to the host and mounted as an actual disk drive. Not the most stable or even a recommended configuration, but what was available to get started.

Our upgrade version was configured in that we upgraded to Veeam 12 and then created a new SOBR with that similar local RAID array drive as the primary backup location (performance tier) and an immutable bucket on Wasabi as the capacity tier. A secondary backup then copied the job to the same local USB datastore as the first scenario just to have an extra local emergency copy.

The problem arose when I gave a consultant express instructions to help us document what we should do if we lost that site. How could we recover our file server data if we lost the key used to create the SOBR and also lost the original backup files. We proceeded to do a successful restore by creating a new key/password pair on the Wasabi bucket and a new SOBR, scanning, then doing a restore. During the process the server took control somehow of the bucket, but this broke the source server's access.

A second expert with a different team explained that my mistake was doing this restore technique, while my source server was still in use and intended to remain in use. He suggested that I do future tests (after fixing the source server) by first making sure no backups are running (makes sense since this is just a once a month activity) then putting the source Veeam SOBR in maintenance mode. Then on the destination server where I am testing the restore, I should then create the new SOBR with the SAME key/password pair as used to set up the source SOBR. Once I scan the repository, import the backups, decrypt them, and test them, I should then DETACH the SOBR on the destination. Then I can safely take the source SOBR out of maintenance mode.

A third expert, on the same team as the second saw my instructions and points out that maybe even my revised procedure may still break the source. She suggests that the safest thing to do is to restore test using the same server as the source SOBR, and the VM just gets restored with a different name on the source cluster. This too makes sense, and if I have enough space I will try this out once we have it fixed up again.

My question, is there some good section of the documentation that could be referenced in reply here that will explain what the safe things to do for restore from Wasabi are? If there is nothing specific to that, could it be added? If you have any questions or if you don't think I explained enough, let me know. My goal is to have a quick reference for SOBR restore testing that I and everyone can use to avoid messing up the source SOBR in the future.

Case #06067125 is the one where we are trying to fix the source SOBR. This is where the restore site took over the Wasabi bucket so the source can no longer write to it, and what I am trying to understand correctly so I don't repeat this in the future.

Thanks.

[Mildur]

Hi Edward

then putting the source Veeam SOBR in maintenance mode.

That's important. The source server must not interact with the bucket while you connect it to a second server.

We have it documented in our user guide --> https://helpcenter.veeam.com/docs/backu ... ml?ver=120

IMPORTANT

Only one backup server must interact with an object storage repository. If you want to perform disaster recovery testing, you can add the object storage repository to another backup server. However, on the initial backup server, you must switch the object storage repository to the Maintenance mode. Otherwise, metadata on the object storage can get corrupted or out of sync, and you will not be able to restore data.

Additionally make sure, that the source and restore server have the exact same patch level of Veeam Backup & Replication installed.

Then on the destination server where I am testing the restore, I should then create the new SOBR with the SAME key/password pair as used to set up the source SOBR. Once I scan the repository, import the backups, decrypt them, and test them, I should then DETACH the SOBR on the destination. Then I can safely take the source SOBR out of maintenance mode.

A SOBR is not required on the second server to test the restores. You only have to connect the bucket as object storage repository and run an import.

You can follow those steps:
1. Switch the Wasabi bucket on your primary backup server to the maintenance mode
2. Connect Wasabi bucket to a new backup server --> https://helpcenter.veeam.com/docs/backu ... ml?ver=120
3. Import the backups --> https://helpcenter.veeam.com/docs/backu ... ml?ver=120
4. run restore tests
5. remove Wasabi bucket from new backup server
6. Disable maintenance mode from the Wasabi bucket on the primary backup server

Best,
Fabian

[Regnor]

I can confirm that Fabian's process is the way to go. This is what we're doing for DR tests and haven't experienced any issues so far.

[dcm-ebb]

Appreciate the feedback, especially the links and instructions. Once we get the database fixed up, I will be trying this out and reporting on how it worked out.