CBT in AHV

[Amarokada]

We are new users of Veeam on AHV, we've come into it at v6 (and Veeam 12.2) so have gone straight into using Prism as the management connection and utilising categories to select VMs in our jobs, all works very nicely.

However we're a few days into things and we notice incremental backups do not seem to be using CBT, as we see the whole size of the VM being read in the backup job details. I've read through the notes here: https://helpcenter.veeam.com/docs/vbahv ... html?ver=6 but we don't believe our cluster is using protection domains.

Our backup jobs are all using the "backup job" method, which leaves a snapshot on the platform but saves the data to a Veeam repo as usual. I believe this snapshot is used as part of the CBT mechanism? We see the following in the RecoveryPoints info for the VM in Prism and the expiry of that snapshot is a couple of months.

(third_party_backup_snapshot_2da9d573-aa24-4ac8-ab4d-2cf92c8bf751)

There isn't anywhere in the backup jobs themselves that relates to CBT like we would have in VMware jobs, so how can I check why we're not seeing CBT working?

Our Nutanix cluster is using AOS 6.8.0.5.

[ronnmartin61]

@Amarokada correct the 3rd party snapshot you see is left by Veeam to compute CBT on the next job run. Full reads each time are not the expected behavior of course. I've seen a similar case of this with v6.0 so it's best if you proceed with opening a case with support. Please note that we don't formally support AOS 6.8.x as yet. Nutanix unexpectedly changed one of the API's we rely on with 6.8. Everything should work as far as backup is concerned however restore to new location recoveries will throw an error (in place restores work as expected). Nutanix is working to correct the issue in an upcoming 6.8 release...

[Amarokada]

Awesome reply, thank you. Ticket raised. Case #07437300

[ronnmartin61]

@Amarokada we know we corrected a retry-related CBT issue in the v6.1 that we just posted. I'm not certain that it corrects your specific issue but we'd recommend upgrading in any case.

[Amarokada]

Upgraded thank you. I'm now wondering if the "processed" and "read" headings for each job statistic page are the same because it it's using the calculated CBT as the "processed" value and not the total size of each VM disk. Would I normally see something in relation to CBT in the VM summaries?

Also we're using 11 workers on the Nutanix cluster (22 nodes), each has 6 CPUs and are configured for 4 streams. When our jobs kick off it all starts ok but after a short while we get failures with:

27/09/2024 21:21:22 :: <VMNAME> : Request failed with status code TooManyRequests

The VMs that have this failure have already started backing up (we see GB of data transferred in Veeam before it fails). Would you know where this is coming from? Prism? The workers? or maybe the cluster itself?

[ronnmartin61]

@Amarokada let me check into processed vs. read a bit further as I may be seeing something similar in my lab. As far as the TooManyRequests error goes that would likely be one to work in conjunction with our tech support

[Amarokada]

My ticket I raised was shutdown by Veeam with:

We have reviewed your case and AOS 6.8 is not currently supported. Please review the article below for verification.
https://helpcenter.veeam.com/docs/vbahv ... html?ver=6

I'm disappointed with this reply but I do understand it. We didn't have control over the Nutanix build but have made sure no further updates are done without first checking Veeam compatibility. However I am left with 3 issues I want to get resolved, and 2 of those are to do with the Nutanix side. Will 6.8 automatically get supported (usually Veeam have a 90 day after release thing on other products such as VCD), is it the same for Nutanix? I'm not sure when 6.8 came out.

[Amarokada]

I'm guessing the 90 day thing doesn't apply to Nutanix as 6.8 came out over 4 months ago.

[ybarrap2003]

@Amarokada Nutanix has a KB written on this since they have not resolved the issue - https://portal.nutanix.com/page/documen ... 0003djB0AQ

[ronnmartin61]

@Amarokada just so you're aware we have QA resources standing by to validate the API fix we're eagerly waiting on in AOS 6.8 and it will not be anything like 90 days before we render a compatibility decision. We will get this done as quickly as we're able as soon as Nutanix ships fixed bits. Feel free to follow the 6.8 interoperability thread active at nutanix-ahv-f51/aos-6-8-veeam-ahv-suppo ... 93415.html

[ronnmartin61]

@Amarokada regarding your earlier question around "processed" and "read" AHV job statistics it appears there is a discrepancy vs. other image level backup job types which I've raised with engineering. While it doesn't represent any actual processing bug with job operations our aim is for consistency across platforms.

[Amarokada]

In my ticket where I asked whether Veeam should leave 1 snapshot behind after each backup (which shows in the VB&R interface) I got a contradicting answer from the support engineer.

He says that "backup" mode jobs should not be leaving a snapshot behind after the backup completes, but I then referred to the link: https://helpcenter.veeam.com/docs/vbahv ... =cbt&ver=6 which suggests it does as it's used for CBT.

RonMartin here also stated that the snapshot is used for CBT processing in order that the entire disk does not have to be read each day. Here is what the support engineer said:

The piece of documentation you've referred in your last e-mail is, indeed, correct.
But this does not mean that, for "Backup" type jobs, snapshots are kept permanently on the cluster.
In each backup session, the Proxy compares the current data with the one that is already present in your repository.

Almost all of our backups in "backup" mode are keeping the last snapshot (just 1) in place until the next day. When that VM is backed up again in incremental mode I see two snapshots during the backup, and then the latest one remains afterwards.

It seems I'm getting conflicting information from different people within Veeam on this point, please can we get this clarified? It doesn't help that the VB&R interface shows the stats for the whole job "processed and read" as being the same figure, which I've mentioned before, but when you click on individual VMs it only says how much data was backed up, which tallies with an incremental, but nowhere in the text does it validate that CBT was used.

[ronnmartin61]

@Amarokada support is correct that the "3rd party backup" type snapshots are not retained permanently but is incorrect in implying that the snapshot does not persist between job runs. After the first full you should always see a snapshot for a given VM which we use to derive CBT for the next run against a new snapshot after which the older snapshot is removed as you in fact are observing.

[Mostapha_Lakhdar]

Hello everyone,

I am experiencing the same CBT issue in my environment, even though I am running the latest supported software versions.

Environment:
Nutanix AOS: 7.5.1.2
AHV:11.0.1.1
Veeam Backup & Replication:13.0.2

The `third_party_backup_snapshot` is correctly created and remains on the VM between backup sessions. However, during every incremental backup, Veeam appears to read the entire virtual disk instead of only the changed blocks.

As a result:

The Read size is almost identical to the full VM size.
Incremental backups take much longer than expected.
It looks like CBT is not being used, although the snapshot required for CBT exists.

Has anyone experienced the same behavior with AOS 7.5.x and Veeam v13?

Are there any known issues or additional logs/diagnostics that I should collect to verify whether CBT is actually being used?

Any guidance from the Veeam R&D team would be greatly appreciated.

Thank you.

[ronnmartin61]

@Mostapha_Lakhdar there are some very rare cases where we ignore normal CBT (comparing the current and prior snapshots) and have to do a full disk read for instance if the actual disk size changes between backup job runs we have to re-read the entire updated disk. These are corner cases though and probably best to open a case with support if you suspect CBT is malfunctioning.

[Mostapha_Lakhdar]

Hi Ronn,

Thank you for your explanation.

In my case, the issue happens on every incremental backup, not occasionally.

The third_party_backup_snapshot remains attached to the VM between backup sessions, and no virtual disk resize or configuration changes are performed.

However, each incremental backup reads almost 100% of the virtual disk, resulting in read sizes nearly identical to the full backup.

Since this behavior is consistent across all incremental runs, it does not appear to be one of the corner cases you mentioned.

I will open a support case and provide the diagnostic logs so the team can investigate why Veeam is falling back to a full disk read.

Thank you for your guidance.

[Mostapha_Lakhdar]

Hi Ronn,
Please confirm which API response or internal condition causes the AHV plugin to emit "Unable to use CBT since the reference snapshot was not found". The logs confirm that the backup snapshot is successfully created and retained (Removed 0 backup snapshots), while the backup runs in incremental mode (ForceFullBackup=false). We would like to determine which exact Prism API response (missing snapshot, UUID mismatch, invalid snapshot chain, or another condition) leads the plugin to set CBT as unavailable, because this information is not present in the standard PlatformSvc, Worker, or Backup logs.

[Mostapha_Lakhdar]

Based on my log analysis, I found the following behavior:

After a full backup of a VM with Veeam, a third_party_backup_snapshot is successfully created on Nutanix. The logs show that this snapshot is created correctly and is intended to be used as the CBT reference for subsequent incremental backups.

However, I observed that after the snapshot is created, Nutanix appears to register or reference it internally using a different UUID than the CBT snapshot UUID initially created by Veeam.

During the next incremental backup, Veeam requests the CBT reference snapshot using the UUID from the previous backup. According to the behavior observed in the logs, Nutanix is unable to resolve this reference, and Veeam reports:

"Unable to use CBT since the reference snapshot was not found."

As a result, Veeam falls back to reading the entire virtual disk instead of using Changed Block Tracking (CBT).

Could you please clarify the following:

Is it expected that Nutanix internally maps the CBT snapshot to a different UUID after its creation?
Which UUID is actually used by the Changed Regions API during incremental backups?
Could this behavior indicate a known issue or bug in AOS 7.5.1.2 affecting CBT integration with Veeam?
Is there any Nutanix log or API that can confirm the UUID used by the Changed Regions request and the reason why the reference snapshot is reported as unavailable?

Any clarification from Veeam R&D team would be greatly appreciated.

Thank you.