Replica failover but then Production returns unexpectedly

[roaima]

We use VBR with replication of our most critical systems. Yesterday our PAAS datacentre provider had problems that meant some of our production filesystems became unavailable on our primary VMware environment. No problem, we thought, and failed the affected VMs over to our DR environment. Happy users. Reasonably happy IT staff.

What we noticed at the time was that because the backing disks for these VMs was unavailable, we could not power off the affected production VMs. In fact we had no control over them at all - all operations were greyed out.

This morning the production filesystems were brought back online and we were notified the outage was over. Unfortunately that meant our production VMs kicked back in and continued running. When IT staff came in this morning we had both replica and production instances for each affected machine.

What should we have been able to do, to ensure that the recovered production instances didn't affect the live replica instances until we were ready?

Thanks
Chris

NB I've not logged a support case because I don't believe this is directly a Veeam issue. But as a Veeam customer I'd like to hear of anyone's suggestions ideally along with best practice recommendations.

[veremin]

What should we have been able to do, to ensure that the recovered production instances didn't affect the live replica instances until we were ready?

The best way would have been not to power up production VMs and failback replica VMs instead.

[roaima]

Thanks. But like I said, we could not power off the affected production VMs as the (vSphere) controls were all greyed out

[veremin]

If you have no control over production VMs, meaning they can be powered up and down randomly, there is little you can do to prevent this, using backup server. Thanks!

[roaima]

We usually have full control over all our VMs (including creation and disposal) and the vSphere environment. There was a disk failure such that the vSAN disks became inaccessible and this stopped us from being able to control the affected VMs.

Maybe I should have submitted a ticket to the PAAS provider to ask them to (attempt to) power down our affected VMs. I don't know.

[veremin]

You can ask provider to shut down production VMs and then perform replica failback operation.

However, changes done to production VMs (while they were active) will be gone and machines will be reverted to current replica states.

Thanks!