Host replacement during backup - Host not found

[GabesVirtualWorld]

Question about replacing hosts during backup to better understand the inner workings. On a 10 host cluster we were replacing 6 hosts (remove from inventory, upgrade, re-add with same name) during the backup window. Because of this backups failed with "Error: Host was not found. Id: [xxxxxx], HostRef: [xxxxxx], HostType: [VC]".

First reaction would be that it sounds logical the host is no longer found because the host has been removed, but VMs have been VMotioned to other hosts, so why would that cause the job to fail? And it failed the complete job of 160 VMs.

I can imagine that during the process VMs have also been VMotioned back from old hosts to freshly re-added hosts and maybe Veeam didn't rescan the vCenter / Cluster for new information yet, so these VMs would have failed then, but I doubt this happened for all 160 VMs.

How often is the vCenter inventory updated automatically by Veeam? Should we have Veeam rescan the vCenter manually after re-adding hosts to prevent this in the future?

[david.domask]

Hi GabesVirtualWorld,

Unfortunately won't be able to tell why the backup failed specifically without a Support Case and a review of the debug logs.

However, I get that you're more about what is the best process for such upgrades here -- Veeam does periodic full rescans of the VMware environment and caches a lot of the information during incremental scans. But I'm doubtful that it's a best idea to try to time the rescan correctly, and instead probably setting a maintenance window for the backups would be best (gracefully shut down the backup server or simply disable the jobs (you can shift + click to select them all and then right-click and disable)

Once the maintenance on the VMware environment is done, then rescan the vCenter and re-enable the jobs.

Edit: fwiw, without seeing the logs and just that brief error you posted, my guess is the vcsa migrated between hosts and some lookup failed, but purely a guess.

[GabesVirtualWorld]

You're correct it is more about the best process.
Could you explain why the full job would then fail? If indeed a host is "missing", I'd expect a per VM error and not completely for the full job.
Would manually forcing a rescan in Veeam before I take a new host out of maintenance mode, solve this issue? I'd rather not move or pause the job because of short backup windows.

VCSA itself is in a different environment, not on this cluster and it didn't move.

[david.domask]

Got it re:vcsa, as noted it was just a guess

As for deeper explanation, won't be able to give more as noted in my answer above, it will require review of the debug logs to understand more specifically what operation failed to do the lookup and what it failed to find.

[GabesVirtualWorld]

Thanks, will discuss internally if we create a support call for this.