Backup job takes a long time to start

[kevin.boddy]

Hi,

My Hyper-V backup jobs take between 20-50 minutes to start. VBR seems to get stuck when starting the job on 'Building list of machines to process'. Once that completes the rest of the job runs quickly.
It's the same for a job with 1 VM vs a job that contains 20 VMs. VBR is connected to our VMM server and our hosts are in a 2016 failover cluster. All VMs in the backup job are added as individual VMs, not as tagged VMs or VM groups.

Any idea why this one process takes so long to run? What exactly is VBR doing when this process is running? Surely VBR knows what VMs need to be processed because they are part of the backup job configuration?

Thanks
Kevin

[PetrM]

Hi Kevin,

If I remember correctly, the list of objects is obtained via PS in case of SCVMM. I'd try to test this cmdlet, most probably it will take the same time to complete. Anyway, I don't think that we can determine the root cause over the forum posts so you may open a support case, upload debug logs as per this instruction and let our engineers to examine logs. Also, please don't forget to paste a support case ID over here for our reference.

Thanks!

[kevin.boddy]

Hi,

I have logged a case yesterday #05395595. VBR was initially taking hours to complete the 'Building list machine to process' due to slow response of the WMI on the Hyper-V failover cluster owner node. After rebooting the VBR server the WMI reponse times came back down to normal levels but the time spent during this process is still very long in my opinion. To me to doesn't look like VBR is issuing PowerShell cmdlet's against the SCVMM server, it seems to rather be bypassing it can talking straight to the failover cluster.

Thanks
Kevin

[PetrM]

Hi Kevin,

Ok, thanks for information. Maybe it's worth trying to run the same WMI queries manually but I think our engineers will figure out the action plan for troubleshooting.

Thanks!

[kevin.boddy]

Hi,

If VBR is configured with the SCVMM server, why would the log bundles not include any logs from the SCVMM server? There only appear to be logs for the Hyper-V hosts. The WMI proxy logs are only available on the Hyper-V hosts and the failover cluster owner node appears two have of them. Surely VBR should be running queries against SCVMM not the failover cluster directly?

The failover cluster owner node is obviously also processing compute tasks for hosted workloads so this would appear to be putting undue extra load on that Hyper-V host.

Thanks
Kevin

[PetrM]

I doubt that my answer would be 100 % accurate as I don't have access to debug logs and I cannot say what is the root cause of this issue. Indeed, there are some queries executed against SCVMM directly as well, however I'd prefer to wait for the outcome of logs analysis from the support team to understand what exactly slows down the process. Also, I think our engineers will be able to collect all necessary logs.

Thanks!

[Larry s]

Hello - I'm just wondering if anything was ever found around this, as I've had similar performance issues, and it would be great if there was a way to get some improvements.

[PetrM]

Hello,

I don't have any ideas besides slow WMI responses but may I ask you to work on the issue with our technical support team and share a case ID? Our possibility to troubleshoot issues on forum is very limited.

Thanks!

[kevin.boddy]

Hi,

I have tested the WMI query manually as instructed by the Veeam engineer and there is nothing wrong there.

The issue appears to be whenever a VBR backup job starts, it runs a WMI query for every single VM in the cluster, not just the VMs in the backup job. If you have hundreds and hundreds of VMs and hundreds of jobs then that turns out to be a lot of WMI queries having to be run.

Apparently it's by design but that makes absolutely no sense at all. Why do that? The backup job is only going to backup the VMs configured in the backup job. Why check every single VM in the cluster?

Regards
Kevin

[PetrM]

Hi Kevin,

At the moment, I don't have an explanation about the purpose to check every single VM in the cluster. It seems that you're right and it makes sense to check only those VMs that must be protected by the job itself. Please let me clarify the details with our support team, I'll update the topic once I hear back from my colleagues.

Thanks!

[PetrM]

Hi Kevin,

As far as I see, the case has already been escalated and our senior engineers and R&D team work together on the resolution. Let's wait for the results.

Thanks!

[kevin.boddy]

Hi,

Private hotfix was supplied to allow us to toggle the checking of VM groups on/off. This has reduced the backup job startup time from 20-50 minutes down to 2.

Thanks
Kevin