13.1 upgrade: workers destroyed, redeploy fails with local storage disabled

[hilto]

13.1 upgrade destroyed all PVE workers, then could not redeploy them: root cause is a hard dependency on local storage

Environment

- Veeam Backup & Replication: 13.1 (build 13.1.1.18)
- Veeam Plug-in for Proxmox VE: 4.0
- Proxmox VE: v9.2
- Guest storage: NFS
- PVE Worker VM storage: NFS

What happened

Upgraded to 13.1. Upgrade itself completed cleanly, no errors, no warnings.

Started an existing Proxmox VE backup job that had been running without issue prior to the upgrade. The first thing the job did was destroy every existing worker VM on the cluster. It then attempted to redeploy them, failed, and the job died. Every subsequent run of that job failed the same way, because there were now no workers left in the environment and none could be created.

I understand workers are disposable appliance VMs pinned to a plug-in build and that a redeploy on plug-in upgrade is by design. That is not the issue, the issue is that Veeam requires Local Storage Enabled to deploy/upgrade PVE Workers.

Root cause

Worker deployment fails unless the **local storage on the PVE nodes/cluster (`local/local-ZFS` on the OS boot device) is enabled and available**.

In this environment that local storage is deliberately disabled, because on a properly built PVE host you do not want the OS boot device presented as usable storage. If something writes to it, you fill the boot drive and take the node down. Guest workloads live on dedicated storage, which is exactly how it should be.

Enabling local storage and re-running the job made worker deployment succeed immediately and the backup completed normally. Nothing else was changed.

Why this is a problem, not just a config quirk

Requiring local storage to be enabled on a Proxmox host is asking administrators to undo a standard hardening decision in order to run a backup product. Separating the hypervisor OS device from guest storage is basic practice. It is also an odd requirement given that the worker is a throwaway VM that could just as easily be placed on whatever storage the administrator nominates.

There is no pre-upgrade check, no warning at job start, and the failure message does not point at storage. The only reason I found it was by trial and error when I first deploy v13 and remembered the issue I found which was the same again after v13.1 upgrade.

What I would like to see changed

1. Remove the hard dependency on local storage entirely. If a valid target storage is configured and reachable, deployment should use it.
2. Fail with an actionable error. "Failed to deploy the worker VM" with no storage context is not diagnosable.
3. Document it. If the local storage requirement is intentional and staying, it belongs in the upgrade checklist/release notes, not discovered by an admin at 11pm with no backups running.

Workaround for anyone else hitting this

1. Enable local storage (`local` / `local-zfs`) on the affected PVE nodes so it shows as available.
2. For each existing worker node run **Test** before touching/running the job to check if local storage enablement fixed the issue.
3. Run the job.

Happy to open a case and provide logs if that is the preferred path, but posting here first because I suspect anyone running a cluster with local storage disabled is going to hit exactly this on the 13.1 upgrade, and they will hit it *after* their workers are already gone.

[vnikiforov]

Hello, Hilton,

Redeploying all workers during the 13.1 upgrade is expected. The plug-in ships a new worker image, and the first job run removes the existing workers and redeploys them from it.

The local storage requirement is not about where the worker VM runs. You pick that storage yourself in the worker settings, and it can be file-level storage that supports snapshots, NFS included. Deployment stages the worker image through the node's default local storage, which is why the redeploy failed once that storage was disabled. That requirement is documented in the User's Guide VM settings step, together with the advice to contact Veeam Customer Support if the default storage cannot be enabled in your environment. Please open a support case and Veeam support will guide you further.

[hilto]

Hi Vladimir,

I will contact support I guess but will be difficult as i am in a trial/demo mode atm. I realise this is in the user guide but there is no reason for this to be setup the way Veeam has in the end. It's a shame that you have dismissed this issue as we wouldn't be talking or going off to support if Veeam didn't use a method requiring local storage being enabled. Any decent engineer in their right mind wouldn't leave local OS drive available to be used by VM's or ISOs etc as operators can easily slip and put data on it filling the OS drive and bricking the host. It is best practice to be disabled local storage on OS for PVE and Veeam hasn't taken that note. Instead have to spend time to do work arounds. It's a simple request that shouldn't have to go to support to do a workaround.

[vnikiforov]

Hello, Hilton,

I personally agree, but that is what we have officially and formally at the moment. Please be advised, I also went ahead and informed the relevant product manager, so they take a look at the issue more closely and suggest if there is a room for improvement. Thank you for your report and explanations.