CBT issues when VBR coexisting with other backup software?

[Moebius]

In this infrastructure I have been using Veeam B&R since the beginning of time. Now, for reasons beyond my control, the Powers That Be decided to transition to Avamar.
For the time being, I am still running VBR (managed by me) along with Avamar (managed remotely by somebody else).

I am noticing that sometimes a job that uses to take between 10 and 15 minutes to complete, suddenly takes 12 hours or more.

Here is an example:
job X (incremental): starts 12:45, ends 12:57. Duration=12'26"; processing rate 100 MB/s; processed 5.1TB; read 10.1GB; transferred 4.2GB.
following run of the same job:
job X (incremental): starts 13:45, ends 02:02. Duration=12h17'20"; processing rate 84 MB/s; processed 5.1TB; read 3.5TB; transferred 36GB.

Most of the times the job runs fast; periodically, the same job takes forever, as in the case above.
The huge difference is clearly in the "read" figure. This makes me suspect some sort of CBT issue due to the coexistence of another backup product (Avamar), since this never happened before Avamar came into play.
However, the statistics show no reference to CBT problems or CBT being reset.

How can I investigate further? Should I open a case about this?

[jorgedlcruz]

Hello,
There is a recommendation not to use multiple products to protect virtual environments, CBT and Snapshots running simultaneously or overlapping are just quick thoughts that come to my mind.

I would recommend you to open a support ticket, so they can investigate the logs deeply and point you to the warning, or error, of why it is taking this long. Can you see inside the Job, inside the VM, which step took this long? Was it reading the disk? Consolidating disks? Assigning Veeam resources, perhaps?

Please share with us the Support Case number, thank you.

[Moebius]

Hi Jorge,

will do, thank you. For what I can see it takes this long reading the disks of one vm.
As for using multiple products on the same environment, I also tried my best to avoid this, but to no avail.

[jorgedlcruz]

Yes, in case that is a CBT reset, it should be on the logs, that is for sure. I can see the 3.5TB read; that will definitely take time. It is quite a lot for an incremental, considering the final size it was just 6x the other job.

I am sure our Engineers would see what is happening on the logs, and you can pass the recommendations from the findings to your team.

Thanks

[HostedBDR]

Typically, the problem isn't 2 vendors using CBT (the same CTK file) to get the job done. Using it at the exact same time, that is where you run into problems. Just like with MS and VSS snapshots, having 2 sets of VSS backups is fine, having them run at the same time or overlap, not so much.

That gets to be a problem with scheduling yes, however keeping them from overlapping will resolve (typically resolve) any conflict. It can (does) work, as CBT is designed to work with multiple vendors reading/using it within the same environment, running at the same time (overlapping jobs) that will only lead to frustration.

[Moebius]

Case #05435190 (now closed).

Veeam tech support said that there is evidence that the CBT had been reset before the start of the Veeam job; however, Veeam cannot track what resets CBT, and when this happened.
I investigated various instances of this behavior and it looks like this happens when a Veeam job starts and completes within the duration of an Avamar job run.

More specifically, it seems that if Veeam creates a snapshot in presence of an already created Avamar snapshot, then when Avamar removes its snapshot it also resets CBT. So, the following Veeam job run takes forever as it re-reads the disks entirely. At least, this is the hypothesis I could formulate so far. I'd like to go ahead and prove it.

How can I look for evidence in the VMware logs of a CBT reset action?

[PetrM]

Hello,

I'd have a look at vmware.log of the VM itself, hostd and vpxd logs. Probably, VMware support could provide a better guidance to find CBT reset events in logs.

Thanks!