CDP with CBT mode - Warning

[FrenchBlue]

Hello,

We have the same problem here. CDP Job and all VMs in it it are syncing properly, and the day after, part of the VMs have switched to "CBT mode" without changing anything.
There is no network issue, and the other VMs in the CDP policy are syncing OK.
BTW, simply disabling and re-enabling the cdp policy <seems> to fix the issue, but not for long, a few minutes later some VMS go back to CBT mode.
Case has been opened, 05464156.

Thanks.

[veremin]

Are you also using backup jobs that create snapshots during CDP policy? If so, you can mention the issue 394736 and request a hotfix for it. If not, continue your investigation with the support team. Thanks!

[FrenchBlue]

Hello,

Yes, we are. But support never asked for this yet and keeps on blaming the infrastructure. I've updated the case (well, the IBM cloud case, they are in between us and Veeam support).

Thanks.

[veremin]

Based on the ticket notes, it's been submitted to the escalation. Once the ticket is escalated, mention the issue 394736 and ask whether you are affected by it (most likely, you are). In case of the positive answer you should be provided with the hotfix. Thanks!

[FrenchBlue]

Hello,

But is the fix already available or in progress? This is a bit unclear, because we got a response from support saying that they are working on a fix, but previously in this thread it seemed that it was already available.

Thanks.

[veremin]

We will double check the investigation internally and update the thread, once we have more information. Thanks!

[veremin]

QA team is currently reviewing the debug logs. If the issue mentioned above is confirmed, you will be provided with the said private fix. Thanks!

[veremin]

Based on the preliminary research, it does not seem that you are affected by the mentioned issue. Most likely there are some vSphere bug that we are currently trying to reproduce. We have already reached your engineer and requested additional details.

I will keep the thread updated.

[FrenchBlue]

Hello,

We were migrating from Vsphere 6.7u3 to 7.0u3d in parallel (completely new cluster with NSX-T, not in-place upgrade). The more VMs we migrated to the new cluster, the less the issue occurred. We have now completed the migration of all source VMs to the 7.0u3d cluster, and in the last 48H, the issue didn't occur at all, no more VMs stuck in CBT mode. From time to time, we can see a VM shortly switching to CBT, but then it quickly recovers to syncing, as expected. We're still monitoring this daily, but it really seems it only affects Vsphere 6.7, not 7.0 .

[FrenchBlue]

HI,

Finally we also hit the issue on 7.0. Occurrence seems much less frequent, but it happened anyway after a few days on one VM. Exactly when Veeam backup took its snapshot for the regular daily backup, error message is that cdp failed to connect to the source, and then game over, VM remains stuck in CBT mode.

[veremin]

The investigation still continues: originally we suspected some platform or environment specific issue and new logs that we got should help us to verify this assumption. Thanks!

[FrenchBlue]

Hello,

For those who are still fighting with this issue: do NOT use VMware tools filesystems quiescence in the "Vsphere" tab of the backup jobs on the source site for Windows VMs, but use AAP instead (which in turn uses VSS), even if your VM has no SQL, Exchange etc.. it must be used for all Windows VMs that are backed up by Veeam on the source cdp site. See kb 1065, it is an issue with snapshots that are done for backups, which impacts cdp too (the VMs who stay in CBT mode forever until you restart the cdp policy, with errors on disks processing in the UI).
We did this last week and the cdp CBT problem is gone for us, not a single issue in 1 week. From time to time a VM switches shortly to CBT mode due for example to very high I/O on it at this time, but no errors, and they now all recover fine to "Syncing" after a while.