Safely testing recoveries from SOBR cloud tier

[CofIAlan]

I've wanted to ship our backups offsite for several years, and with v10 SOBR and cheaper S3-compatible storage providers I'm now in the acceptance testing phase of that project. Setup and about a month's worth of offloads accomplished the first part of testing where the actual backup process appears to work quite well (as usual with Veeam, it just works). I'm now ready to test restoring some of our VMs and data to ensure that this new SOBR cloud tier is actually storing accurate copies of our backups.

What I'm wondering is if anybody has tips, tricks, or gotchas to conducting this part of the test. Ideally I'd like to use a read-only IAM configuration so that there's no chance of messing up our production backups. While trying to find information about restore methods I ran across this post: object-storage-f52/recovering-from-a-di ... 58727.html This is essentially what we'd do after an actual disaster because we'd have no production systems left to mess up, but it doesn't seem entirely safe for testing purposes. First, there's the fact that data access would be read/write AND need to talk with some sort of non-existent testing infrastructure (we're quite small and do not have test storage or server systems). Second, it would be a second, restored Veeam instance attempting to interact with the same cloud storage as our actual Veeam instance. It doesn't seem like those would play nicely, but perhaps that's invalid.

If we were to experience a disaster, my playbook based on self-contained VBK and VIB files from local or Backup Copy Jobs is to spin up a Veeam server, then vCenter, then domain controllers. Since cloud-based SOBRs don't store data in individual files like this I'm not sure of the best way to safely simulate that recovery plan. Is there a way to just attach to a cloud provider using a Veeam browser of some sort and extract VBK/VIB files (or VM config and disk files)? Is there a way to get Veeam to use a SOBR in a read-only mode so that no data is touched in the cloud?

Or am I thinking about this the wrong way? Should I be approaching this test in some other fashion?

Thanks!
Alan

PS - If a Veeam browser (Explorer) doesn't exist for SOBR cloud tiers that would be pretty cool. Enter an endpoint and API key, then select what to restore and where to put it with no risk to the cloud data. Bonus points if it could integrate with other Veeam explorers to allow fast item recovery (though my use case would be thrilled with just bringing back entire VM files).

[soncscy]

Hey Alan,

I've done this for a ton of clients, and you can do it very simply -- just attach your object storage repo to any Veeam Server (punch in the keys and such), right-click, and select "import backups". I think it even automatically tries this when you add it.

From there, you can initiate any restore operations as if it were an on premises backup, and coolest part is, as best I can tell, you don't have to repatriate the data to test.

I'd run down the array of restores you do:

1. Do a file level restore
2. Test an instant-recovery alongside production (I recommend this because you can start the instant recovery with the NIC detached, and they can coincide without harming each other)
3. Test a restore to your EC2 or Azure if you have the space/time/bezosbucks to test

I'd recommend an application restore too, but we're having weird issues doing this directly when we try this with a few clients. Start a file level restore and then doing an application restore seems to work, but dunno what's happening with the rest. Trying to convince the clients to open tickets.

But, that aside, the rest should be very seamless, and honestly for me it's been impressive to see it "just work". It doesn't touch the cloud data except to read it, and you don't have to pull a full backup down to do the operations, just start the restore. If you want to download the backup for more testing locally, then it's as easy as select "Download" on that backup.

[oleg.feoktistov]

Hi Alan,


Adding to Harvey's point, if you have no spare resources to build a separate lab on, you could just put performance extents of your SOBR to maintenance mode and thus force direct restore from capacity tier every time you choose a restore type. This scenario simulates unavailability of your performance extents and is well described on one of these schemes.

Thanks,
Oleg

[CofIAlan]

Thank you both. I didn't want to jump into this test and then accidentally blow up my previous work. I set up a random desktop computer with v10 Community Edition. I was able to add an object storage backup repository and import our backups. This was done using a read-only API user as well, so that appears to "just work" when run against an existing cloud repository. The restore is a little slow (based on other threads that seems common) but otherwise is currently plugging along. I'll be hand-transferring the VM files to our environment to spin them up, isolated from everything else, and ensure they actually boot.

The only issue I ran into is one of the reasons you should always test your backups:
When Veeam asked for our encryption key before importing the backups it didn't work. It said the key was wrong. Turns out that I had the wrong iteration of the key saved. I set a new key on our production server, brought that over to the test box, and voila: backups started importing.

[Gostev]

I'll be hand-transferring the VM files to our environment to spin them up, isolated from everything else, and ensure they actually boot.

Keep in mind you can do instant recovery into your environment directly from backups in object storage. It works reasonably well for the purpose of ensuring the VM actually boots. Definitely much faster than transferring the entire VM files

[CofIAlan]

Absolutely. In this case it's a totally isolated system that's designed just to test the overall files and cloud provider. In an actual recovery I'd be shipping them right into our VMware cluster so I could spin them up as soon as they were restored.

So far, so good!