"host with id" "was not found" error after new hosts added

[TWuser]

Hi all -

Case #07109835 for those interested.

Have a VBR server with 20+ replication jobs and 130+ servers, replicating to our DR environment with multiple hosts and clusters (vmware). We have new hosts that were added to existing clusters, replacing the existing hosts. VM's (including replicas) were vmotioned off the old hosts onto the new hosts before they were removed from the cluster.

Almost all of the replication jobs now give me an error:

1/29/2024 5:01:42 AM :: Failed to create processing task for VM **vmname** Error: Host with id 'f*******-e***-4***-9***-1***********' was not found.

Implying the job needs the old host in order to run, even though the replica exists in the cluster and shouldn't be dependent on a host.

What is way more frustrating is that I cannot EDIT these jobs - it gives me the error below if I try to edit:

"Object reference not set to an instance of an object"

- If I try to clone the job, the cloned job gives me the same error.
- - - I tried cloning them in powershell and it has the same result
- I also can't DISABLE most of the jobs, so those jobs are continuing to run and fail repeatedly throughout the day, absolutely spamming my email with hundreds of failures.
- - - I tried disabling the jobs in powershell and it has the same result "Object reference not set to an instance of an object".
- We tried adding an "old" host back into vcenter temporarily to see if it would change anything, but no luck. I am thinking the "host id" doesn't stay the same when a host is removed and later added back to vcenter.

I had a similar issue with this last fall due to a host failure, and I ended up having to rebuild the jobs, as well as rebuild my Orchestrator Datalab from scratch.

Anyone know why a Replication Job, which does NOT point to any hosts directly, can't handle new hosts replacing old ones? It should be a pretty common situation, but I don't see a ton of forum posts out there with the issue, especially with no ability to edit, disable, or even clone jobs, which is extremely frustrating and non-sensical.

[tyler.jurgens]

When you check your "Backup Infrastructure -> vCenter Servers" do you see all your newly added hosts in that list? Have you tried rescanning your vCenter (right click on it and rescan, or select it and hit rescan from the ribbon menu)?

[TWuser]

Yes, I have rescanned vcenter probably a dozen times by now at least. It has no problems seeing the new hosts.
Rebooted multiple times, even removed and re-added the local proxies just in case.

I am working on making new jobs from scratch, using the Summary of the current jobs combined with the VeeamOne "Job Configuration Dump" to make sure I have consistent VM's, settings, schedule, etc. Then will map the existing VM's to the new jobs, though CBT will be broken and "calculating digests" will take a ton of time, but hopefully less time than fully re-seeding. So a disappointing and fairly labor intensive process if things go well, and if I have any issues I may have to delete it all and re-seed anyways.

This guy has a good post essentially working through the same thing: https://community.veeam.com/blogs-and-p ... found-5806 but also does not have a real solution.

[TWuser]

Digging into the rabbithole, I did search the database (currently on SQL still) for all instances of the host ID in the error (I used this guys script: https://www.meziantou.net/find-a-guid-i ... tabase.htm)

Not a SQL expert, but I found it in Backup.Model.Backups, BJobs, and TombStones tables. Some of the VM's in bad jobs already had the HostID of the new host, so I didn't see anything obvious there, and am not in a rush to break the database, so haven't changed anything.

Going off past experience of a broken Datalab, I checked the replica VMX files, and also found a ton of Veeam data stuffed into the replica VM's .vmx file, including the host ID under "UniqueDigestRef". But once again, a broken job's VM seemed to have a new host listed here, so I don't see anything obviously wrong. I may dig a bit more comparing DB tables and the VMX file, but without better understanding I am unlikely to find a consistent solution I can trust.

[TWuser]

After more digging and testing, I ended up editing the database and was able to fix all my very broken replication jobs without having to spend hours manually recreating them (with plenty of risk of human error to miss something).

The SQL table dbo.BJobs has entries of all my jobs, and the "target_host_id" column had entries matching the "host with id..was not found" error.
I created basic tests jobs and pointed them to each cluster (we have 3). I then took the target_host_id values for those (new) servers and used those values to replace the broken ones.  

I first did this with a smaller less critical job, and it fixed it right away. All my jobs are back and happy again.

I found this guy afterwards and it looks like he essentially did the same thing:
https://jc-lan.org/2022/06/29/veeam-11- ... esxi-host/

Support did not offer to take a look or try to edit my database and said it was a "known issue".
When asking how to avoid the issue in the future since adding and removing hosts is not that unusual, they suggested "adding the new hosts and confirming that the replication jobs run to the VM on the new hosts prior to removing the old hosts".

The next time we remove hosts, I will be disabling all replication jobs, re-scanning vcenter in Veeam both before and after a host change, and running multiple test jobs.