15 minute wait for Workers to respond

[Amarokada]

Hi

Today we upgraded to the latest versions of Veeam/AHV/plugin.

We have 12 worker nodes in our 22 node cluster, and they were all updated today by enabling the "Obtain updates from rpm repositories" open and then testing them. Then afterwards we switch off the automatic updates.

Now that our nightly backups have kicked off we're noticing that when workers are shutting down and then re-starting for another backup, they sometime hang for 15 minutes at this stage:

>3/31/2025 8:32:08 PM Success Connection to the worker service was established successfully 15 min 6 sec

We see this sometimes also:
>3/31/2025 9:22:12 PM Warning Failed to synchronize update settings of the worker: worker IRIS-WORK-10-N: Timeout for Ready state —
And even though we have automatic updates switched off it then hangs for another 15 minutes trying to:
>3/31/2025 9:22:12 PM Running Checking known repositories for updates —

Taking 15 or 30 mins to start 1 worker is a huge problem for us as no other workers get started during this time as Veeam only starts 1 worker at a time. It doesn't seem to be limited to the same workers getting stuck, it seems to be random at this stage.

This is going to have a huge impact on our backups tonight, pushing them to much later in the day tomorrow (1500 VMs total).

I know this is a new problem and will need to be looked into, but our customer is going to have a fit, we've been fighting different problems with Veeam for AHV since we started using it on Nutanix and the reliability just isn't there. We upgraded to this latest version to fix the scheduler issue that has hit us twice already, only to find now we have an issue with worker startups.

Is there a way to avoid workers from powering down for a set amount of time to allow another job to start using it, without the whole shutdown/startup cycle each time?

[ronnmartin61]

Definitely unexpected behavior and I'm not seeing any similar cases so of course best to work through support.

[Kochkin]

Also, if your network does not have access to rpm repositories, it is possible to disable automatic updates from network. That may speedup worker preparation in your case.
https://helpcenter.veeam.com/docs/vbahv ... html?ver=7

[mbmac]

We are having the exact same issue and have opened a case with support. This is a new out of the box nutanix prism central deployment with Veeam. Oddly enough if we use the embedded worker on the proxy it runs fine. It seems to only affect dedicated workers.

[Moderator: Case number 07658391]

[ronnmartin61]

@mbmac I see you've closed the case and perhaps pursued another direction however supporting a very large number of PC-managed clusters is something we want to insure we support. As far as PC sizing is concerned does your PC instance(s) conform to the cluster limits outlined at https://portal.nutanix.com/page/documen ... =pc.2024.3?

[mbmac]

@ronnmartin61 Yes we had to scrap the PC deployment as it was killing the entire PC environment. Our PC is scale out XLarge. We had numerous hours of support calls and they found a few minor things to adjust that made 0 difference. Our environment is 90 two node clusters with 4 VMs per cluster so the PC size at XLarge seems completely overkill IMO but it would literally crash when we hook it into Veeam. As soon as we pull Veeam from the environment PC works exactly as expected.

[ronnmartin61]

@mbmac thank you for the additional information. We're actively researching on our end and I've also raised this with my Nutanix counterpart