CBT in AHV

[Amarokada]

We are new users of Veeam on AHV, we've come into it at v6 (and Veeam 12.2) so have gone straight into using Prism as the management connection and utilising categories to select VMs in our jobs, all works very nicely.

However we're a few days into things and we notice incremental backups do not seem to be using CBT, as we see the whole size of the VM being read in the backup job details. I've read through the notes here: https://helpcenter.veeam.com/docs/vbahv ... html?ver=6 but we don't believe our cluster is using protection domains.

Our backup jobs are all using the "backup job" method, which leaves a snapshot on the platform but saves the data to a Veeam repo as usual. I believe this snapshot is used as part of the CBT mechanism? We see the following in the RecoveryPoints info for the VM in Prism and the expiry of that snapshot is a couple of months.

(third_party_backup_snapshot_2da9d573-aa24-4ac8-ab4d-2cf92c8bf751)

There isn't anywhere in the backup jobs themselves that relates to CBT like we would have in VMware jobs, so how can I check why we're not seeing CBT working?

Our Nutanix cluster is using AOS 6.8.0.5.

[ronnmartin61]

@Amarokada correct the 3rd party snapshot you see is left by Veeam to compute CBT on the next job run. Full reads each time are not the expected behavior of course. I've seen a similar case of this with v6.0 so it's best if you proceed with opening a case with support. Please note that we don't formally support AOS 6.8.x as yet. Nutanix unexpectedly changed one of the API's we rely on with 6.8. Everything should work as far as backup is concerned however restore to new location recoveries will throw an error (in place restores work as expected). Nutanix is working to correct the issue in an upcoming 6.8 release...

[Amarokada]

Awesome reply, thank you. Ticket raised. Case #07437300

[ronnmartin61]

@Amarokada we know we corrected a retry-related CBT issue in the v6.1 that we just posted. I'm not certain that it corrects your specific issue but we'd recommend upgrading in any case.

[Amarokada]

Upgraded thank you. I'm now wondering if the "processed" and "read" headings for each job statistic page are the same because it it's using the calculated CBT as the "processed" value and not the total size of each VM disk. Would I normally see something in relation to CBT in the VM summaries?

Also we're using 11 workers on the Nutanix cluster (22 nodes), each has 6 CPUs and are configured for 4 streams. When our jobs kick off it all starts ok but after a short while we get failures with:

27/09/2024 21:21:22 :: <VMNAME> : Request failed with status code TooManyRequests

The VMs that have this failure have already started backing up (we see GB of data transferred in Veeam before it fails). Would you know where this is coming from? Prism? The workers? or maybe the cluster itself?

[ronnmartin61]

@Amarokada let me check into processed vs. read a bit further as I may be seeing something similar in my lab. As far as the TooManyRequests error goes that would likely be one to work in conjunction with our tech support

[Amarokada]

My ticket I raised was shutdown by Veeam with:

We have reviewed your case and AOS 6.8 is not currently supported. Please review the article below for verification.
https://helpcenter.veeam.com/docs/vbahv ... html?ver=6

I'm disappointed with this reply but I do understand it. We didn't have control over the Nutanix build but have made sure no further updates are done without first checking Veeam compatibility. However I am left with 3 issues I want to get resolved, and 2 of those are to do with the Nutanix side. Will 6.8 automatically get supported (usually Veeam have a 90 day after release thing on other products such as VCD), is it the same for Nutanix? I'm not sure when 6.8 came out.

[Amarokada]

I'm guessing the 90 day thing doesn't apply to Nutanix as 6.8 came out over 4 months ago.

[ybarrap2003]

@Amarokada Nutanix has a KB written on this since they have not resolved the issue - https://portal.nutanix.com/page/documen ... 0003djB0AQ

[ronnmartin61]

@Amarokada just so you're aware we have QA resources standing by to validate the API fix we're eagerly waiting on in AOS 6.8 and it will not be anything like 90 days before we render a compatibility decision. We will get this done as quickly as we're able as soon as Nutanix ships fixed bits. Feel free to follow the 6.8 interoperability thread active at nutanix-ahv-f51/aos-6-8-veeam-ahv-suppo ... 93415.html

[ronnmartin61]

@Amarokada regarding your earlier question around "processed" and "read" AHV job statistics it appears there is a discrepancy vs. other image level backup job types which I've raised with engineering. While it doesn't represent any actual processing bug with job operations our aim is for consistency across platforms.

[Amarokada]

In my ticket where I asked whether Veeam should leave 1 snapshot behind after each backup (which shows in the VB&R interface) I got a contradicting answer from the support engineer.

He says that "backup" mode jobs should not be leaving a snapshot behind after the backup completes, but I then referred to the link: https://helpcenter.veeam.com/docs/vbahv ... =cbt&ver=6 which suggests it does as it's used for CBT.

RonMartin here also stated that the snapshot is used for CBT processing in order that the entire disk does not have to be read each day. Here is what the support engineer said:

The piece of documentation you've referred in your last e-mail is, indeed, correct.
But this does not mean that, for "Backup" type jobs, snapshots are kept permanently on the cluster.
In each backup session, the Proxy compares the current data with the one that is already present in your repository.

Almost all of our backups in "backup" mode are keeping the last snapshot (just 1) in place until the next day. When that VM is backed up again in incremental mode I see two snapshots during the backup, and then the latest one remains afterwards.

It seems I'm getting conflicting information from different people within Veeam on this point, please can we get this clarified? It doesn't help that the VB&R interface shows the stats for the whole job "processed and read" as being the same figure, which I've mentioned before, but when you click on individual VMs it only says how much data was backed up, which tallies with an incremental, but nowhere in the text does it validate that CBT was used.

[ronnmartin61]

@Amarokada support is correct that the "3rd party backup" type snapshots are not retained permanently but is incorrect in implying that the snapshot does not persist between job runs. After the first full you should always see a snapshot for a given VM which we use to derive CBT for the next run against a new snapshot after which the older snapshot is removed as you in fact are observing.