Can v6 replication perform an SRM like failover test?

[MattG]

The one task that is performed regularly with VMware SRM is failover tests. It is as simple as clicking a button and then verifying the recovery site VMs.

I have read through the v6 user guide and do not see this functionality in v6. Am I missing it?

Thanks,
-MattG

[Gostev]

Yes. Replica testing is covered in the sticky FAQ topic (VMware). Thanks.

[MattG]

From FAQ:
Q: I want to test replica failover. Can I simply power on any replica restore point with vSphere Client?
A: Yes. Because each restore point is a snapshot, its contents will remain the same (all disk changes from the running VM will go into the new snapshot file the host would automatically create). Actual replication job also won't brake. Unlike with v5, testing replicas with v6 is really a breeze - thanks to the new architecture.

Is there a way to perform test failover of multiple VMs or do you need to perform failover per VM?

Thanks,
-MattG

[Gostev]

For real failover, multi-select operations are supported (and not just for failover - more details in the What's New document).

However, since replica testing is done outside of product UI, the answer depends on the process you choose. You can do it one by one with vSphere Client. Or you can write short PowerCLI script that powers on multiple replica VMs at once. Depend on your DR site networking, you may also want to have this script disconnect network before powering on replica VM for testing.

[MattG]

Since replica testing is performed outside of the Veaam UI, does that mean you can't test the re-ip functionality without doing a real failover?

Thanks,
-MattG

[Gostev]

Yes, that is absolutely correct. Re-IP is a part of real failover process. But again - in many scenarios, you would want to test replicas with vNIC disconnected anyway. Just something to keep in mind.

[tsightler]

For real failover, multi-select operations are supported (and not just for failover - more details in the What's New document).

However, since replica testing is done outside of product UI, the answer depends on the process you choose. You can do it one by one with vSphere Client. Or you can write short PowerCLI script that powers on multiple replica VMs at once. Depend on your DR site networking, you may also want to have this script disconnect network before powering on replica VM for testing.

What do you mean when you say "replica testing is done outside of product UI"? Certainly with V6 you can still do "failover" and "undoFailover" via the UI and both of these features support multi-select and work great.

[MattG]

I want to be able to perform test failovers like SRM. In SRM I have a Recovery Plan that has Re-IP customizations. I click on the RP, click Test, and then have the RP test failover to the Recovery Site. On the Recovery site these VMs would startup connected either to a Private VLAN or to a default test bubble network. This functionality allows me to do full testing of the recovery and make sure the VMs come up as expected without any effect on the production VMs.

Can v6 do this?

-MattG

[Gostev]

What do you mean when you say "replica testing is done outside of product UI"? Certainly with V6 you can still do "failover" and "undoFailover" via the UI and both of these features support multi-select and work great.

I mean, when you want to test if your replicas VM are good and bootable without initiating real failover.

Can v6 do this?

You can do all this minus Re-IP. Reading through my response above, re-IP only kicks in during the "real" failover.

[MattG]

Gostev,

Could you add this to the feature request for the next version (the ability to do SRM like failover testing)?

It is great to have replication and the hope is that you'll never have to use it, but you should be able to fully verify failover on a regular basis. I say fully, because at least with SRM, the re-ip process is not always smooth and I don't want to find out after a prod-failover that half my VMs aren't re-ip'ing.

-MattG

[Gostev]

We have fully automated replica testing on our roadmap (we call this feature SureReplica, analogous to our existing SureBackup). Truth is, this was not a pressing feature request so far - probably because unlike backups, replicas are easy to verify because they are already stored in the native format.

I do see your point regarding the wish to validate the re-IP too as a part of the replica testing, of course. What helps here is that unlike in SRM, with Veeam re-IP rules are "global". Thus, you can test your re-IP rule on real failover of a single test VM, and if it works properly there, there is no reason why the same rule would not work correctly on all other VMs.

Thanks.

[tsightler]

Honestly I don't understand why you can't just do this with the V6 GUI. You can select multiple VM's in and perform failover, with re-IP, on all of them. Performing "failover" does not do anything to production, it simply preps the VM's at the target site and powers them on. After verifying that everything powers on and works as expected, simply "undo Failover".

The only thing to look out for is making sure that the VM network at the target site is not connected to production. I'd suggest just mapping the network to vSwitches that are not connected to the physical network and simply know that you will need to connect them during an actual failover event, but the other option is to simply temporarily disable the physical nics connected the the VM nics at the remote site during the test.

I just tested this in my lab with my DC, Exchange and SQL server and it worked great. Brought up the replica's on my isolated network, re-IP'd to a different subnet, with one quick trip through the "Failover" wizard. Production servers continued normally. After I finished testing I simply selected all three replica VM's from the replica screen and selected "Undo Failover" and the three replica VMs were powered down and put back into their pre-failover state, ready to resume replication. Sure, it's only two VM's, but the process would be almost identical for 10 VM's. Might not scale to dozens or hundreds of VM's, but works great otherwise.

[Gostev]

The only thing to look out for is making sure that the VM network at the target site is not connected to production. I'd suggest just mapping the network to vSwitches that are not connected to the physical network and simply know that you will need to connect them during an actual failover event, but the other option is to simply temporarily disable the physical nics connected the the VM nics at the remote site during the test.

This is the very reason why people do this outside the product: real failover process overwrites all your changes to VM networking configuration. At least, this used to be the case with all previous versions of the product (we even had some semi-tested registry hack at one time that would prevent VMX overwrite - but I am not sure on its current status and applicability).

I will find out tomorrow if anything had changed regarding VMX overwrite in v6 (although I am pretty sure everything is still the same), and whether that registry hack still works for v6.

[tsightler]

This is the very reason why people do this outside the product: real failover process overwrites all your changes to VM networking configuration. At least, this used to be the case with all previous versions of the product (we even had some semi-tested registry hack at one time that would prevent VMX overwrite - but I am not sure on its current status and applicability).

I'm not suggesting making changes to the VM network configuration, although I can see how you could read that from my posts. I'm suggesting making changes to the vSwitch network configuration to isolate the vSwitch that the replica VM's are connected to. We don't overwrite vSwtich configuration. This is what I've always suggested even with V5 and earlier versions. Where possible, replicate to an isolated vSwitch network on the target, and only connect the vSwitch during "real" failover. This has two advantages, it gives you an isolated network for failover testing, and limits the chances of a "mistake" by failing over a still running production machine to a "live" network. The one disadvantage is that it adds a small step during actual failover.

[MattG]

An SRM TEST failover creates a test bubble network (local private switch per host) by default and then tears it down after the test. The better practice is to configure the failover to use a defined non-routable VLAN on the reocovery side that all of the recovery hosts are configured to use. In SRM instead of the bubble network you can specifically say that the that the test recovery should use the non-routable VLAN.

This allows all test recovery VMs to be reachable no matter what recovery host they are on. This is important to do app testing (test sending Exchange email, test writing to SQL DB, etc).

-MattG

[tsightler]

Yes, I understand how SRM works, and I'm not trying to claim that Veeam has an exact match for this functionality, I'm only pointing out how, with a good setup and careful planning, you can get very similar functionality. The SRM "better practice" that you mention is pretty much exactly what I was recommending for Veeam replication to test failover. The feature we lack is the ability to map to this network only during testing, but you can get very close to this by simply having a procedure to turn this "nonroutable-VLAN" into a "routable-VLAN" during an actual failover event. Very close, but certainly not exactly the same, and not as automated.

[pmalinov]

[merged]

With version we now have the capability to re-IP and auto change VLAN setting on replicas - THIS IS GREAT!
But I have a question:
- Will this work if we start/boot up the replica at our remote site directly through the vSphere client???

I'm asking because at the remote site we do not have a Veeam Backup Server and also would like to test the DR plan from time to time, which requires breaking the link between the two sites during the test (dont want to have 2 mail servers or sql servers battling over each other)

Thanks

[Gostev]

@pmalinov also please be sure to review the sticky FAQ topic, as it addresses this very question as well.
This discussion above is related to the failover testing and might be helpful as well.

[pmalinov]

Yeah, I read through the thread but it is not clear to me if there is an easy way to work around our current DR/VEEAM setup to take advantage of this feature.
One of the things to consider when talking about a Disaster is that your main site may not be intact/reachable/equipment may be stolen or connected to the fail-over site. How would you initiate the fail-over if you are faced with a real Disaster and your Veeam Server is gone?

[Gostev]

Personally, I would simply adequately protect my Veeam server and make sure there is a replica of one in my DR site.

Or, I would place Veeam DB on remote SQL server (instead of using local), those are usually always well protected for high availability with clustering, native SQL replication, and so on.

And, if I lost both original and replica Veeam server (or database), I would just go ahead and power-on all replica VMs manually with vSphere Client. First, there is really no other options left; and second, it no longer matters that I won't be able to failback after this (since my Veeam deployment gone, failback is impossible anyway). And if I needed re-IP, I would do it manyally once those replica VMs are up.

[pmalinov]

Yes - that is exactly what we do now. We re-ip manually
Unfortunately we cannot replicate the veeam server as it is not virtual...but I can find some tool to pull an image regularly. We use Acronis for desktop deployment but I dont think it will work with a server.
Thanks

[Gostev]

Did you consider placing SQL on a VM, putting Veeam configuration DB there instead of local SQL, and replicating that VM?