Perform a full DR from Remote site with replica's

[GraPghaST]

We are trying to make a Disaster Recovery Plan (DRP) in case the main office burns down.

At the main office we have 3 ESX hosts in one virtual vCenter, with a physical Backup server and lots of local storage and a Dell Equallogic for central storage. At the Remote Site, we have one ESXi host in a different virtual vCenter. Veeam B&R backups all the production VM's to a local backup server. Veeam also replicates the production VM's to the Remote site ESXi host. I created a separate Veeam VM at the remote site. Every day, SQL backups are made from the Veeam production Database, and copied to the remote site Veeam installation. When a DR has to be performed we uninstall Veeam at the Remote site, restore the Veeam Database from production, and reinstall Veeam pointing to the new database. Veeam can start at the remote site, and everything is visible. OK!

Our main goal is to pick up that remote site ESXi host in case of a disaster, drop it at a temporary office site and power up all VM's. B&R of that data will be set up after this. First things first, the VM's have to get running asap. When we start Disaster Recovery, first thing you want to do is start up the Replica's stored at the remote site. We have two possible scenario's, each does not work (well enough):

1) Perform a failover on the replica. Won't work, because the target ESX host (The one in the production office) isn't available anymore. The main office burnt down, remember?
2) Manual startup through the remote site vCenter. Is possible, but a restore point can't be chosen: We are not starting the VM up through Veeam and all its data.

So, at the remote site we have perfect replica's of all VM's including restore points, a current database, a Veeam installation, a vCenter and one ESXi host to power up from. But still we can't get our environment up and running at the remote site without doing concessions.

Questions: Are replica's suiteable for Disaster Recovery from a remote site? if not, what is by Veeam themselves recommended as a fool-proof Disaster recovery setup in case of disaster on the main office? Can Backups be of any help? As in: perform a backup to the remote site, and in case of a disaster perform an Instant Recovery of the backed-up VM to get it running, and perform a Storage vMotion when new hardware is added to the vCenter cluster?

Many thanks for your time and answers!

[Vitaliy S.]

1) Perform a failover on the replica. Won't work, because the target ESX host (The one in the production office) isn't available anymore. The main office burnt down, remember?

I might have missed something from your post, but it is not necessary to have a source host accessible for a backup server to perform a failover. Assuming that you've replicated Veeam Backup configuration database to a remote site and attached that database to a new Veeam backup server, then you can perform a failover to any available restore point.

Are replica's suiteable for Disaster Recovery from a remote site?

Yes, of course.

Can Backups be of any help? As in: perform a backup to the remote site, and in case of a disaster perform an Instant Recovery of the backed-up VM to get it running, and perform a Storage vMotion when new hardware is added to the vCenter cluster?

That would also work, but generally we recommend doing backups locally and performing off-site replication to be able to power up the production VMs immediately after the source site goes down.

Thanks!

[GraPghaST]

Thanks for your prompt answer. The problem that we have when trying to failover a replica is that we get an error message during file lock: The object has not been created fully, or something. The failover job gets terminated. When I then try to perform a second attempt, the replica is still in faiover status. And that lock can't be removed. Only when I go to properties of that replica, and the force lock removal. But the replica is left in a red exclamation mark state, seems to me that it then is incomplete from viewing the replica's properties. The whole replication has to start over again.

What I have done so far:

The replica job replicates a VM to the ESXi host at the remote site. OK
I have imported the most uptodate SQL database to a new Veeam installation, disabled all the jobs and updated the passwords on all server connections. OK
I then disconnect the remote site ESX host and make a direct connection to both vCenter and ESX directly. Works both. OK
When I try to failover a replica, an error occurs and failover job gets cancelled. NOT OK
When I try to undo a failover, an error occurs and job gets cancelled. NOT OK
When I try to undo a failover with force lock removal, replica gets incomplete. Replication has to start over again. NOT OK
I then rerun the replication job from the main Veeam installation, it runs without errors. But the replica at the remote site is still marked with red exclamation mark, and incomplete. ?
When I then try to delete the replica from the remote site disk, it states that it is locked by the replica job I just ran. Which is finished. ?
I deleted all jobs, references and files from disk. When I try to delete the replica it still says the file is locked by <replicajob>, and cannot be deleted
I created a new replica job from the main office Veeam and tried to run it, but it then says that the replica already exists, and subsequently fails.

So how can I resync the replica? I need to do this to get the exact error I see when performing failover in the first place. i don't want to mess up another replica at the remote site.

is this problem ticket worthy? I'd like to get some help with this

[Vitaliy S.]

I then disconnect the remote site ESX host and make a direct connection to both vCenter and ESX directly. Works both. OK

Could you please elaborate it a little bit further? I guess this is the reason why you get errors on the next steps.

Taking into consideration that you've successfully moved Veeam configuration database, you needn't re-connect anything, as target ESX(i) host should already be presented in the Veeam backup console to perform a failover.

[GraPghaST]

Could you please elaborate it a little bit further? I guess this is the reason why you get errors on the next steps.

Taking into consideration that you've successfully moved Veeam configuration database, you needn't re-connect anything, as target ESX(i) host should already be presented in the Veeam backup console to perform a failover.

Of course. The remote site setup is still in the same serverroom as the production environnment, but separated by an ASA. So when this project is finished succesfully, we only have to move the equipment, change the IPsec VPN outside addresses and everything is configured. When I say disconnected, I simulate a situation that the main office is totally gone. The remote site environment can't reach production anymore. So I pull out the cable that connects the ASA to the "Outside" world. I the connect to the remote ESXi host or remote vCenter directly by connecting a network cable between my laptop and the server.

When a Veeam database is imported, the only thing that has to be configured in order to connect to the remaining ESXi host is the passwords used to connect to them. Safety feature, so that one can't get to the ESX hosts passwords when he has obtained the Veeam Database.

If you need to know more, please let me know!

[Vitaliy S.]

Robbert,

Well... based on the description you've given everything looks good. I wonder what type of error you get while trying to perform a failover. It would be best if you could shoot all the logs to our support team and update this topic with resolution you recieve from our tech guys.

P.S. By the way does your DR plan cover possible disasters that affect the entire site/server room? I've just remembered one video clip that speaks for itself.

[GraPghaST]

Robbert,

Well... based on the description you've given everything looks good. I wonder what type of error you get while trying to perform a failover. It would be best if you could shoot all the logs to our support team and update this topic with resolution you recieve from our tech guys.

P.S. By the way does your DR plan cover possible disasters that affect the entire site/server room? I've just remembered one video clip that speaks for itself.

Sounds good. Which logs do they need, and where can I send it to? Will be a fairly large amount. I already opened a support case, referring to this thread.

And yes, I know that DR equipment needs to go offsite, I'm not new to this. Otherwise I would be missing the whole point (which I'm not) Maybe I didn't explain it clear enough, but the equipment is only here to configure the DR setup correctly. And as long we haven't got a working configuration to go with (as in: This problem needs to be fixed first) the equipment isn't going anywhere. Only when we have a good, working DR setup and ditto plan the equipment wil move to an external datacenter.

[Vitaliy S.]

Maybe I didn't explain it clear enough, but the equipment is only here to configure the DR setup correctly. And as long we haven't got a working configuration to go with (as in: This problem needs to be fixed first) the equipment isn't going anywhere. Only when we have a good, working DR setup and ditto plan the equipment wil move to an external datacenter.

Oh...got it! You should update your ticket with veeamshell.log only and if additional logs are required you'll be notified.

[GraPghaST]

OK, ticket is updated with log. Now we wait till support has woken up

[GraPghaST]

Logs are dropped, mail is sent, solution not yet available. Vitaliy S., could you please see if Anna has received them?

[Vitaliy S.]

Robbert, I cannot find your ticket, can you please send me a PM with a support case number?