Anyone using cdp in production?

[FrenchBlue]

Hello,

I wonder if anyone is using cdp in production for a real drp and managed to stabilize it sustainably? We have been trying for about 8 months now, without success yet. We had nothing but problems, with several support cases. The last issue we had was supposed to be fixed by a new "private fix", which made things even worse and literally broke the cdp functionality. Case has been opened for a couple of weeks already, without success. It seems like we'll never be able to stabilize it, and I'm starting to doubt that it will be fixed in V12, so I really wonder if some customers are using it successfully, I don't have any colleagues or customers using this feature yet in my environment.

Thanks.

[veremin]

If you can share the current ticket number we can verify the investigation internally. Thanks!

[FrenchBlue]

Hello,

Sure, current case is 05706797.

[veremin]

Thanks for providing us with the case number. We will double-check it and post back with more information.

[FrenchBlue]

BTW, something I didn't mention yet in this ticket and the previous ones is that we have a high bandwidth available between both sites, but around 85ms latency. Could this be an issue?

[veremin]

According to our support team, the case is given due diligence and is not stuck in the "no response" state. It seems that you experience network issues that cause proxy servers to run in "CBT" mode (high latency might be the reason).

I'd recommend continuing investigation with the support engineer that looks to be in contact with you (the last response dates yesterday).

Thanks!

[jmc]

hello,

yes. we have been using it since the beginning with some customers. at first there were also some problems, but then everything was ok. however, we only have around 5 vm's running in 2 cdp jobs per customer. of these, most are equipped with more than 5 hard drives and over 1 tb.

if we have e.g. maintenance or downtimes with the vm's, the synchronization goes to cbt. after some time - when everything is in sync again - the vm switches back to sync.

however, we haven't had a real deployment to use yet. so we haven't started a cdp vm yet.

if you change something on the hard disk configurations, however, you have to recreate the cdp for that vm.

environment used is mainly hp bl460 gen9 and gen10 with hp msa sas and ssd storages over fc.

so we have no problems with the cdp at the moment.

jeff

[FrenchBlue]

Thanks for the info. Are you in plain 11a 20220302 or do you have subsequent private fixes? Do you monitor the size of your replica folders?

[jmc]

hello,

no privates patches. yes, and i have checked the function yesterday. i have startup a cdp vm without any problems. everything works pretty well.

jeff

[FrenchBlue]

OK, maybe you are more lucky due to a very low number of VMs, but at the moment we only have 20 VMs spread over 3 policies. We should go up to around 150 but we just can't scale up until it is stabilized... or we have switched to something else for cdp.

[fcp]

We are using it in production and it is working fine after issues with I/O filter and our with certs on our VMware hosts. We only do CDP with our seven critical file servers. I have each VM in their own policy. Have only two CDP proxies, one at each datacenter to handle all seven CDP jobs. We have two one gig fiber links between the datacenters with about 15ms ping times. Two of the VM's are 3.7TB and 4.1TB and the rest are under a TB. Only issues we have is if we expand the drives on them it has to do a full copy of the VM again to the remote site. Besides normal network traffic across this link to our other datacenter we do backup copy jobs across this link.

[veremin]

OK, maybe you are more lucky due to a very low number of VMs, but at the moment we only have 20 VMs spread over 3 policies. We should go up to around 150 but we just can't scale up until it is stabilized... or we have switched to something else for cdp.

How did the investigation end for you? Originally it was network-related issue that makes proxy server stuck in "CBT" mode and prevents CDP policies from running successfully, were there any other problems found? Thanks!

[FrenchBlue]

Hello,

Unfortunately, the main issue since the last private fix (cdp jobs all going and remaining in CBT mode in less than 24 hours, until they are restarted manually) is not yet solved and we are still struggling with it, case is still opened.

[FrenchBlue]

Hello,

Update for follow up: after migration to V12, the cdp issues are gone, it seems to be stable now, at least for a few days, which never happened in V11, and without any change to the underlying infrastructure. So anyone still having issues with cdp on V11 should probably migrate as soon as possible to V12.

[veremin]

We put a lot of effort to make CDP better and stable in v12, and I'm glad our hard work paid off. Thank you for your kind words (they have been already shared with the R&D team).

[gschaffer]

Hello,

Update for follow up: after migration to V12, the cdp issues are gone, it seems to be stable now, at least for a few days, which never happened in V11, and without any change to the underlying infrastructure. So anyone still having issues with cdp on V11 should probably migrate as soon as possible to V12.

Hi, how many VMs do you have in CDP now?
We have a lot of issues in v11, and we have a lot of new issues in v12.

How is the consumption of CPU by CDP Coordinatinator service in VBR server? And how much log CDP generate in VBR server?
I have now 415VMs Replication in CDP, but we have almost 6 new issues in v12 and 7 cases opened.

[FrenchBlue]

Hello,

Wow, this is a big setup. We only have 30 VMs on cdp at the moment, slowly growing up to around 150, but some big ones in the 3 to 8TB disk range and with high I/O write activity (databases, database dump repositories etc...). Only 1 source and 1 destination cdp proxy at the moment (Windows but we're migrating to RHEL soon) , it works fine with no specific issue since we migrated to V12.
How many proxies do you have ? What are the main issues for information? At what size did they start to happen? Did you already have the 415 VMs replicated in V11?

[philrandal]

We are using it in production and it is working fine after issues with I/O filter and our with certs on our VMware hosts. We only do CDP with our seven critical file servers. I have each VM in their own policy. Have only two CDP proxies, one at each datacenter to handle all seven CDP jobs. We have two one gig fiber links between the datacenters with about 15ms ping times. Two of the VM's are 3.7TB and 4.1TB and the rest are under a TB. Only issues we have is if we expand the drives on them it has to do a full copy of the VM again to the remote site. Besides normal network traffic across this link to our other datacenter we do backup copy jobs across this link.

How did you fix the I/O filter and cert issues?

[gschaffer]

Hello,

Wow, this is a big setup. We only have 30 VMs on cdp at the moment, slowly growing up to around 150, but some big ones in the 3 to 8TB disk range and with high I/O write activity (databases, database dump repositories etc...). Only 1 source and 1 destination cdp proxy at the moment (Windows but we're migrating to RHEL soon) , it works fine with no specific issue since we migrated to V12.
How many proxies do you have ? What are the main issues for information? At what size did they start to happen? Did you already have the 415 VMs replicated in V11?

Hello,

I have 2 proxies in each site, running on linux. But I didn't notice relevante difference between Windows or Linux proxies in performance or general behavior.
In v11 I had 210 VMs and stop to grow when Source VMs started to freeze and reboot because of DRS/VMotion migrations.

Main issues:
Inherited from v11:
1) High CPU usage caused by Veeam CDP Coordinator Service in VBR Server (CDP Coordinator Service) and VBR database Server. In v11 I was using SQL Express.
2) 12GB+/week of logs being generated by CDP Coordinator

New in v12:
3) We are unable to change proxy configuration in CDP Job. Doing it in UI stops replication. Via Powershell is not even possible, returning error.
4) In some circunstances Veeam is lost in file locks. Apparently the target proxy is sending replication to an ESXi host but files are locked by another. Turning that VMs unable to replicate. Basically, rebooting target ESXi host makes some issues.
5) In some VMs, Veeam Fail to create long-term restore points.
6) Veeam does not alert about VMs stoping to replication, you need to constantily check if all vms are or aren't replicating.

The issues started with the first CDP job.

We have some 3GB+ VMs also, and they are write intensive. I recommend dedicatin a CDP job and a target datastore for each of these VMs, because the size they will get in target site in umpredictible
For the rest of "normal" VMs, I am trying to maintaing 30VMs/job and target datastore.

[FrenchBlue]

Hello,

Wow, this reminds me at least partially of the issues we had in V11, apart from the load/logs size issues of course. How are you you doing the backups on the source sites? With Veeam too? I suppose you already use AAP? This is mandatory to avoid issues with cdp.
The lack of cdp monitoring and the ui which often doesn't reflect the current status are still issues for sure.

[gschaffer]

Backup is being done by another Veeam B&R server (we dedicate one VBR Server to CDP). Not using AAP. Why do you say it is mandatory?

[FrenchBlue]

Hello,

Because we had several of these issues before we were using aap on the source backups, support quickly suggested us to use it and it improved the situation significantly. At some point, we only had this orphan files issue where some vmdk and tlog files in the replicas folders were older than our 24H retention policy for cdp and stayed there forever, which caused more or less infinite growth of the replicas and was a serious issue. Then there was a private fix for this and it made things worse, replication kept crashing after a few hours. But since we upgraded to V12 all issues are gone. During the night, when backups run on the source site, rpo gets higher because there is a high network load, and some VMs can temporarily switch to CBT mode, but then it catches up and everything is OK again in the morning.
I suggest you try aap on the source VMs backups. I don't know if your backup and cdp jobs are aligned 1:1 , but try it (Windows VMs) at least on all the backup jobs of a given cdp policy VMs list (you have to disable VMware file quiesce in the options for these jobs and only use aap/vss) and you'll see what happens. Not sure it will work, but at this point, it is worth trying.

[jeg97]

Hello,

I'm going to implement CDP and I'm looking to use tags. Is anyone using tags with CDP and what's the simplest way setup tags to align with different Tiers for orchestrated recovery?