CDP Planned failover?

[vmtech123]

Long time SRM user her.
SRM is quite simple. In a planned situation, VM's power down, volumes replicate, then VM's come back up. Replication changes direction, repeat.
If it is unplanned, boot up latest changes on a 5 minute RPO or so, things work good. There is also the ability to test SRM without actually effecting production.

What I don't get, is in CDP why we don't have a planned failover, or that option on the failover plans like we do for a replica.

What I'd like is on the failover plan itself, or the even by selecting the "Ready" CDP VM under Replicas, to do a planned failover and failback. (power off, replicate, power up, verify) then (Power off, failback changes power up)

To failback, If I manually select the active VM's to failback to production, it powers the VM down and boots it up which I'd assume would ensure no data loss. This is what I want for testing FROM production.

Am I missing something about this? I don't want the 30 seconds of data loss to test if my RPO is set to 30 seconds. Even in SRM with 5 minutes RPO, when you do a failover where the VM's are still accessible it powers them down then finishes the replication before bringing them up at DR ensuring no data loss.

[Andreas Neufert]

I think what you search for is our SRM like Software Veeam Disaster Recovery Orchestrator:
https://helpcenter.veeam.com/docs/vdro/ ... tml?ver=50

[vmtech123]

Interesting. That is totally overkill for what I need. I just wanted a way to test CDP without data loss the same way Replicas work.

I suppose using a bunch of replicas works for us. And keeping SRM for the last few critical VM's using SAN replication. The reason I wanted to switch to CDP was to stop paying for SRM.

CDP seems to work great, but that 30 seconds of data loss during a test is a deal breaker. I'd be fine with the downtime while the VMs powered off and finished the last bit of replication, but not data loss.

It seemed bizarre that on replicas we have the planned failover, but not on CDP. Now I see there is an upgrade to buy more software, it makes more sense why that feature isn't included anymore.

[veremin]

It was an initial release of brand new technology so we had to sacrifice certain features to make it happen (with planned failover being one of them).

We are aware of this request and planning to implement it in one of the next product versions - we cannot tell right now in which one exactly, though.

Thanks!

[Andreas Neufert]

@veremin do you know if the Orchestrator has the same limitations ?

[veremin]

Orchestrator also performs simple (not planned) failover, as far as I know.

[Andreas Neufert]

So my recommendation for Orchestrator was not addressing all your needs @vmtech123 was not the complete solution.

[vmtech123]

Thanks guys. Orchestrator looks amazing, don't get me wrong. We use SRM here and I totally understand having a big easy button to hit in a disaster is key.

I'm working on moving TSM, SRM etc to Veeam and there are just a few things that usually hold me back and planned failover is a big one but look forward to future releases as it should be pretty easy for you guys to implement.

If we can switch over, do some POC's and get workloads using replicas and CDP, the next step would be to obviously get Orchestrator to make life easy and bring things up in order, do planned testing etc.

Currently if I can't select a failover plan, have the VM's power down then come up at DR, then fail back. it's a no go.

By "TEST" we can't just bring stuff up at DR and assume it works. They want to fail over, test networks and run there for a bit.. Id obviously want to use failback to prod and not commit or undo this because data loss is 100% no, and committing would mean I need to replicate hundreds of TB again.


I will say, if I get approval without having to test, i'll start using CDP a bit. My test VM's were able to fail over without issue and work great. the RPO worth of Dataloss is going to be a no go when things are running though.

[veremin]

I believe sort of manual planned failover might work here. If you shutdown machine from inside the guest OS, wait till CDP activity stops for it (should happen rather fast) and then execute failover, there should not be any data loss.

[vmtech123]

haha, "sortof" and "should" are not things I want to hear when talking about dataloss. Just kidding

Thanks. I didn't even think of this. I could run a script to power down VM's in a failover group possibly, then do the CDP failover. I wonder if a pre-failover script would work.... There would have to be a way to check and make sure all vm's were down and CDP has stopped but this might be a solution. If that didn't work even a powercli to gracefully power down using vmware or force shut down if something was stuck.

[veremin]

I had a quick chat with QA team prior to posting my reply, according to them, this scenario should work fine, but they have not tested it thoroughly - so this's added a bit uncertainty to my replies

[vmtech123]

I believe sort of manual planned failover might work here. If you shutdown machine from inside the guest OS, wait till CDP activity stops for it (should happen rather fast) and then execute failover, there should not be any data loss.

I have a test CDP job with a single VM in it. Status = Syncing and SLA was at 100%.

When I powered down the VM at the main site, I received this error.

VM configuration for CDP completed with errors Error: Unable to continue replication: VM must be powered on
Source daemon issue detected: (4026531849:vvr) get-source-replication-state. Failed to get I/O filter command stream: I/O filter not connected (Time: 5/9/2022 7:53:13 AM)
Failed to process disks
Please power on the source VM to continue processing (Time: 5/9/2022 7:53:24 AM)

I assumed If I wait two minutes (RPO TIME) the status would changed to Synced, or stopped but it just stays at syncing.

I know Veeam requires the VM's to be powered on for CDP according to this so it seems this should be expected.
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

I may be wrong, but just powering off the VM's isn't going to work for this test. the last 2 things the job tells me are Please power on to continue, followed by Failed to process disks when the VM is off.


If you want I can open a ticket, but seeing that "CDP Testing" isn't really a thing, I don't know how far I'll get.

[veremin]

The error is expected, cause indeed the CDP policy can transfer traffic only from running VM.

However, before policy fails it still should transfer all of the changes happened prior to graceful shutdown - this is what I was discussing with the QA team.

So when you see the said error, it means that CDP activity has stopped and you can now failover the target VM. If it occurs that the latest changes have not been synced, you can open a support ticket and we will research with the support and QA engineers.

Thanks!

[vmtech123]

Thanks for the response again. Much appreciated.

So while this will be good to try on some test machines, there really is no way to KNOW if the last sync completed. Perhaps that could be my feature request,
1) CDP planned fail over is an obvious one. On the failover plan or individual VM. (power down, sync, confirm, power up at DR) Failback will be same in opposite direction keeping changes while at DR. When a DR test is done, we fail to DR, wait for next window, fail back. replicating only the changed blocks back would be a great bonus.

If not, at least try and implement the following.

2)In CDP after the VM is powered down and data transferred, change from "Syncing" to "Synced" or something along those lines saying VM powered down. It's not really an Error at this point.



It sounds like it should work, and it worked on my test VM, but in a 24hour 7 days a week critical business, with VM's that can range to 40+TB that won't fly with the execs "sorry I broke the VM during a DR test and your 2 hour window is now going to be 3 days" lol

[veremin]

Understood, some of your ideas have been already planned for future product versions, others will be noted. Thanks again for the feedback!