v9 replication jobs getting stuck on 'Building VMs list'

[mvalpreda]

Case # 01690842

2x Windows 2012 R2 Hyper-V servers on the same subnet. Veeam running on the host with production VMs. Second Hyper-V server is old server and now being used as a replication target.

Upgraded from v8 to v9 and had an issue with replication jobs. Futzed around and wound up re-creating the replication job. Got one replication job to run fine and when the next replication job goes to run, it gets stuck at 'Building VMs list' forever. It usually takes <10 seconds to build the VM list and then run the job. If I kill all the Veeam services, run a backup job, then a replication job it will run. If I kill the services and run a replication job, it gets stuck on 'Building VMs list'.

This worked flawlessly in v8. Can I roll back to v8?

[Gostev]

There is no rollback for the existing installations, but you can deploy the new instance of v8. But I would first confirm with support that this is indeed a v9-specific issue, and not some other change in the environment. Thanks!

[mvalpreda]

No other changes. In-place upgrade to v9.

[foggy]

Job logs should tell what it is doing while it is stuck, so please working with support on this.

[ptoro]

I am on version 9 of Veeam and loving it (coming from v8)!

One thing I have noticed though is that the building VMs list has increased dramatically (maybe I just wasn't paying attention before, but nonetheless).

HyperV environment (clusters) going back to scale out file servers that serve up SMB3 shares (no SCVMM).

What is this "building VMs list" step actually doing? Any info would be greatly appreciated.

I have multiple jobs that just focus on the VMs that reside on that shares CSV (volume) and at the most I am backing up 17 VMs per job. That one job with 17VMs takes almost 2 hours just to build this VMs list!!!!!!!!! (insane). And then it starts to backup the VMs..... It's crazy town

All my jobs are now using per-vm backup chains (I checked the repo and the individual files have been created). I'm not really concerned about the backup time when it is actually backing up the vms but of the time to build these lists.

I just created a support ticket but wanted to know if anyone else was experiencing these issues. From what i've read, doesn't appear to be an issue with VMware/vCenter. I'm hoping it's something dumb on my end and not a norm with hyperV in cluster not controlled by SCVMM.

[Gostev]

one job with 17VMs takes almost 2 hours just to build this VMs list

This is not normal.

I just created a support ticket

What is the support case ID?

[ptoro]

Veeam Support - Case # 01736028 - Building VMs List....

[ptoro]

So still working with support on this. quick recap of what has been done:

- enabled creating the VM list from the actual hosts and not from veeam server (through reg key)
- disabled guest processing

no difference what so ever.

Just sent over another set of logs, will keep you guys posted. From what Veeam tech said, building the VM list should only be taking a couple of minutes. So there is something definitely going on...

[FUBIT]

I am having the exacty same issue but I even have identical servers and the jobs get stuck at building VMs list. It works so well when it runs but is so unreliable now.

[Vitaliy S.]

I cannot find any resolution in the OP case as it just started to work, so please contact our technical team for troubleshooting directly.

[mkaec]

I have experienced a similar problem 3 times with backup copy jobs in v9. The job would get stuck at "Building VMs list" and sit there indefinitely. I would recover by rebooting the backup server. Support case 01705801.

If it occurs again after installing update 1, I will revisit with support.

[mbrinkho]

Having the same issue here but with a standard backup. Veeam 8 was great, 9 has given us some major issues.

Aside from rebooting the server is there anything else I can do for this?

[foggy]

Do you man reboot doesn't help or rebooting the server is not an option for you?

[mbrinkho]

Rebooting makes the job run for a few days and then it gets stuck at building VMs list again. If I try to cancel the job it never cancels.

[foggy]

Then I suggest letting our engineers review your setup and logs.

[djesko]

Same problem here , did you get a fix ?

EDIT : did see now that the respons was from today

Did you create a case ?

Br Niklas

[foggy]

Niklas, it is always recommended to open your own case, since the reasons of this behavior can vary from one environment to another.

[MarkWatson]

Did you get anywhere with this? I'm facing a similar but worse issue where the job doesn't get past the "Building VMs list", I'm working with Veeam suport on this, case # 01767317.

[mbrinkho]

I'm still psyching myself up to put a call in to support.

[jazzoberoi]

Same thing is happening to us. Works beautifully for a few days and sometimes just sits at building the VM's list till the server is restarted.

[Vitaliy S.]

Jazz, in order to troubleshoot this issue quicker, please open a support case.

[mbrinkho]

Opened a case, 01771011.

[peter_cosa]

any fix for this?

[bcann]

I found that if you backed up via selecting the host rather then the individual VM's, AND you had application aware on, it would happen a lot more often. when I selected the individual VM's and only the application aware items on the specific servers it seemed to disappear.

[mbrinkho]

Installing Update 1 fixed this for us.

[kamiwa]

Installing Update 1 fixed this for us.

Installing Update 1 introduced this bug for me.

As general advice seems to open a support ticket with Veeam:
How do I open a support ticket?

[Vitaliy S.]

You can do that via our website > https://www.veeam.com/support.html

[kamiwa]

Thanks. Already found out by myself and support got back to me with a hotfix that I'm currently testing.

[mkaec]

I had this happen twice this weekend. This was the first time it happened with backup jobs. Previously, it had only happened with BCJs. It happened both times with the same job.

[Vitaliy S.]

Marc, if you still experience this issue, then please reach out to your support team for the hotfix.