Feature Request: Veeam Service Broker Auto-Recovery to a vCenter after Network Outage

[TimLawhead]

Case # 03444871

At a secondary site network switches were replaced last week causing an outage between the Veeam B&R server and the vCenter at that site. According to Veeam support, during the network outage the Veeam B&R server's Veeam Broker Service got hung up when communicating with that vCenter and was no longer able to communicate with it after network outage was over. All backups failed at that site until the Veeam B&R server was rebooted, though a reset of the service may have been sufficient. Backups at the primary site and communications with the primary site's vCenter were not affected.

The error message on all the jobs, which Google wasn't turning up in my searches: Tag *VM Tag Name* is unavailable, VMs residing on it will be skipped from processing. Nothing to process. All machines were excluded from task list.

When I went into the B&R client to check on the Backup Jobs, I attempted to Retry and Start the job with them failing. When I went into the Backup Job properties I found that the VM Tag for the Backup Job would be unable to recalculate the VMs sizes. I was able to use Powershell and pull VM Tags from the vCenter without issue.

While network outages shouldn't be happening, it'd be nice if the Broker Service was able to auto-recover or reset itself after it goes into a hung state after a network outage between the B&R Server and a vCenter.

TIA
Tim

[HannesK]

Hello Tim,
and welcome to the forums.

Thanks for bringing this up. I will investigate further whether it's an issue or a feature request.

Thanks,
Hannes

[HannesK]

Hello,
we tried to reproduce and the broker service reconnected. He tries it every 30 seconds.

Best regards,
Hannes

[TimLawhead]

During the outage a Copy job was started at the primary site by the B&R, it was unable to reach the target repository and the job failed with "Unable to access target repository Error: The RPC server is unavailable. RPC function call failed. Function name: [GetSvcVersion]. Target machine: [*FQDN of target repository*:6160]. I can see 2 other copy and one backup job after that with the same error, then there's a Copy job that has a "Cannont connect to target backup repository" error. Then all other jobs after that have the "Tag "name" is unavailable..." message.

The issue happened on 2/27 and wasn't caught until 3/4. I tested VM Tag calls with Powershell and had no issues getting them. During this process our primary site backups which were to a different vCenter continued without issue. Ultimately I waited till there wasn't a job running and then rebooted the B&R server and the issue with the VM Tags at our secondary site went away.