Possible Bug - Post-upgrade to V12/AHV4 - AHV Backup Proxy Protection jobs not working

[HYF_JE]

Greetings,

I'm not sure if this is a bug in V12/AHV4 for certain, but wanted to ask if any other customers have this same behavior (or if Veeam developers can repro) or if it's just us. If others can confirm I will open a support case and update this thread.

We upgraded from V11/AHV3 to V12/AHV4 on Tuesday April 18th. We have a handful of Nutanix clusters, each with a single AHV proxy each. I know that during the component upgrade process of the V12 upgrade that the old AHV proxy VM is deleted and a new one is deployed.

Note we are in UTC-5 so a discrepancy in some logs between 10PM of the 17th and 3AM on the 18th is to be expected

See this screenshot - this is the statistics/log of the PD snapshot job for one of the backup proxies that was working prior to our upgrade on the 18th.



Now see this screenshot - this is the statistics/log of the same PD snapshot job after the upgrade. Note how it's ending with a warning because there's no VMs found.



If we look at the protection domains on the relevant cluster, we can see that there is a VM of the same name as the backup proxy in the protection domain....

Please ignore the number of warnings in the screenshot, these are not related to this issue and pre-exist this issue by many months.



....but if we look at the snapshot listing, they seem to stop on the day of the upgrade:



Here's my conclusion based on what I'm seeing:

1. After the V12 upgrade is completed, the component upgrade process executes.
2. During the component upgrade process, the AHV proxy VMs are re-deployed and the old VMs are deleted.
3. Unlike a normal proxy deployment, whatever process adds the AHV proxy VM to the VeeamBackupProtection on the Nutanix cluster does not run during the component upgrade.
4. While the name of the VM is unchanged in Nutanix, it gets a new vmuuuid internally on the Nutanix cluster, so nothing seems afoul in the Nutanix Prism webUI...
5. ...but the VM listed in the protection domain is the previous AHV proxy VM, not the newly created one.

Please comment if you're seeing the same behavior.

[Mildur]

Hi HYF_JE

I cannot try it in my lab, I'm already on the newest version and my lab doesn't have those snapshot jobs.

Can you please open a support case and provide them with the log file and your conclusion from this post?
Our support engineer will be able to check for know bugs. Please provide me with the case number, so I can follow up the case from my side.

Best,
Fabian

[HYF_JE]

Can you please open a support case and provide them with the log file and your conclusion from this post?

Support case #06022046 created with logs attached.

[HYF_JE]

Here's the update: Feature, not bug. Also I have a bit of egg on my face, this feature is documented in the release notes which I did read, but probably mentally skipped over as it wasn't a very "interesting" change.

https://www.veeam.com/veeam_backup_nuta ... tes_rn.pdf

Auto-protection PD snapshot job on appliance is no longer created after backup appliance deployment or registration. This has been replaced with Configuration Backup from the Veeam Backup & Replication server jobs. Existing PD jobs will not be automatically removed during appliance updates and should be removed once Configuration Backup is setup.

It's also documented here: https://helpcenter.veeam.com/docs/vbahv ... tml?ver=40

I deleted our AHV proxy protection jobs after verifying the AHV configuration backups new to v12 are working. I know we had an issue in V11 (not sure if other customers had it...) where deleting jobs via the VBR console wouldn't work as the AHV proxies would re-create the jobs after a restart or enough time had passed. But at least for me, after deleting the proxy protection jobs and restarting the AHV proxies, the jobs have not come back. It's also worth noting that you'll want to visit each Nutanix cluster and cleanup the old protection domain.

I'd like to give some feedback to the development/product team:

• Please work with the documentation teams to mention this change in the upgrade checklist(s). It's a very easy thing to skip past.
• This seems like the perfect kind of thing to add to the new BPA feature. Informing customers that they have stale jobs for protecting the v3 AHV proxies/appliances and giving guidance on how to clean them up (both within Veeam and on their clusters) would be a good feature.

To the forum moderators: please mark this thread as solved or won't fix. I don't think I have the ability to edit thread titles.

[Mildur]

Hi HYF-JE

Thanks for updating this topic.

Please work with the documentation teams to mention this change in the upgrade checklist(s). It's a very easy thing to skip past.

I will talk to our user guide team. It would be a good addition for post tasks and/or upgrade checklist.

This seems like the perfect kind of thing to add to the new BPA feature. Informing customers that they have stale jobs for protecting the v3 AHV proxies/appliances and giving guidance on how to clean them up (both within Veeam and on their clusters) would be a good feature.

Thank you for your request. We will talk about it.

Best,
Fabian