Host-based backup of Microsoft Hyper-V VMs.
mvalpreda
Enthusiast
Posts: 77
Liked: 3 times
Joined: May 06, 2015 10:57 pm
Full Name: Mark Valpreda
Contact:

v9 replication jobs getting stuck on 'Building VMs list'

Post by mvalpreda »

Case # 01690842

2x Windows 2012 R2 Hyper-V servers on the same subnet. Veeam running on the host with production VMs. Second Hyper-V server is old server and now being used as a replication target.

Upgraded from v8 to v9 and had an issue with replication jobs. Futzed around and wound up re-creating the replication job. Got one replication job to run fine and when the next replication job goes to run, it gets stuck at 'Building VMs list' forever. It usually takes <10 seconds to build the VM list and then run the job. If I kill all the Veeam services, run a backup job, then a replication job it will run. If I kill the services and run a replication job, it gets stuck on 'Building VMs list'.

This worked flawlessly in v8. Can I roll back to v8?
Gostev
Chief Product Officer
Posts: 31804
Liked: 7298 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by Gostev »

There is no rollback for the existing installations, but you can deploy the new instance of v8. But I would first confirm with support that this is indeed a v9-specific issue, and not some other change in the environment. Thanks!
mvalpreda
Enthusiast
Posts: 77
Liked: 3 times
Joined: May 06, 2015 10:57 pm
Full Name: Mark Valpreda
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by mvalpreda »

No other changes. In-place upgrade to v9.
foggy
Veeam Software
Posts: 21138
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by foggy »

Job logs should tell what it is doing while it is stuck, so please working with support on this.
ptoro
Influencer
Posts: 21
Liked: 5 times
Joined: Jul 24, 2015 4:41 pm
Contact:

[MERGED] Building VMs list = SLOW

Post by ptoro »

I am on version 9 of Veeam and loving it (coming from v8)!

One thing I have noticed though is that the building VMs list has increased dramatically (maybe I just wasn't paying attention before, but nonetheless).

HyperV environment (clusters) going back to scale out file servers that serve up SMB3 shares (no SCVMM).

What is this "building VMs list" step actually doing? Any info would be greatly appreciated.

I have multiple jobs that just focus on the VMs that reside on that shares CSV (volume) and at the most I am backing up 17 VMs per job. That one job with 17VMs takes almost 2 hours just to build this VMs list!!!!!!!!! (insane). And then it starts to backup the VMs..... It's crazy town 8)

All my jobs are now using per-vm backup chains (I checked the repo and the individual files have been created). I'm not really concerned about the backup time when it is actually backing up the vms but of the time to build these lists.

I just created a support ticket but wanted to know if anyone else was experiencing these issues. From what i've read, doesn't appear to be an issue with VMware/vCenter. I'm hoping it's something dumb on my end and not a norm with hyperV in cluster not controlled by SCVMM.
Gostev
Chief Product Officer
Posts: 31804
Liked: 7298 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Building VMs list = SLOW

Post by Gostev »

ptoro wrote:one job with 17VMs takes almost 2 hours just to build this VMs list
This is not normal.
ptoro wrote:I just created a support ticket
What is the support case ID?
ptoro
Influencer
Posts: 21
Liked: 5 times
Joined: Jul 24, 2015 4:41 pm
Contact:

Re: Building VMs list = SLOW

Post by ptoro »

Veeam Support - Case # 01736028 - Building VMs List....
ptoro
Influencer
Posts: 21
Liked: 5 times
Joined: Jul 24, 2015 4:41 pm
Contact:

Re: Building VMs list = SLOW

Post by ptoro » 1 person likes this post

So still working with support on this. quick recap of what has been done:

- enabled creating the VM list from the actual hosts and not from veeam server (through reg key)
- disabled guest processing

no difference what so ever.

Just sent over another set of logs, will keep you guys posted. From what Veeam tech said, building the VM list should only be taking a couple of minutes. So there is something definitely going on...
FUBIT
Lurker
Posts: 1
Liked: never
Joined: Mar 29, 2016 5:05 pm
Full Name: FUB IT
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by FUBIT »

I am having the exacty same issue but I even have identical servers and the jobs get stuck at building VMs list. It works so well when it runs but is so unreliable now.
Vitaliy S.
VP, Product Management
Posts: 27371
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by Vitaliy S. »

I cannot find any resolution in the OP case as it just started to work, so please contact our technical team for troubleshooting directly.
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by mkaec »

I have experienced a similar problem 3 times with backup copy jobs in v9. The job would get stuck at "Building VMs list" and sit there indefinitely. I would recover by rebooting the backup server. Support case 01705801.

If it occurs again after installing update 1, I will revisit with support.
mbrinkho
Enthusiast
Posts: 28
Liked: 2 times
Joined: Jan 03, 2014 5:14 pm
Full Name: M Brinks
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by mbrinkho »

Having the same issue here but with a standard backup. Veeam 8 was great, 9 has given us some major issues.

Aside from rebooting the server is there anything else I can do for this?
foggy
Veeam Software
Posts: 21138
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by foggy »

Do you man reboot doesn't help or rebooting the server is not an option for you?
mbrinkho
Enthusiast
Posts: 28
Liked: 2 times
Joined: Jan 03, 2014 5:14 pm
Full Name: M Brinks
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by mbrinkho »

Rebooting makes the job run for a few days and then it gets stuck at building VMs list again. If I try to cancel the job it never cancels.
foggy
Veeam Software
Posts: 21138
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by foggy »

Then I suggest letting our engineers review your setup and logs.
djesko
Novice
Posts: 4
Liked: never
Joined: Jan 14, 2015 9:28 am
Full Name: Niklas Engberg
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by djesko »

Same problem here , did you get a fix ?

EDIT : did see now that the respons was from today :)

Did you create a case ?

Br Niklas
foggy
Veeam Software
Posts: 21138
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by foggy »

Niklas, it is always recommended to open your own case, since the reasons of this behavior can vary from one environment to another.
MarkWatson
Lurker
Posts: 1
Liked: never
Joined: Apr 20, 2016 4:44 pm
Full Name: Mark Watson
Contact:

Re: Building VMs list = SLOW

Post by MarkWatson »

Did you get anywhere with this? I'm facing a similar but worse issue where the job doesn't get past the "Building VMs list", I'm working with Veeam suport on this, case # 01767317.
mbrinkho
Enthusiast
Posts: 28
Liked: 2 times
Joined: Jan 03, 2014 5:14 pm
Full Name: M Brinks
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by mbrinkho »

I'm still psyching myself up to put a call in to support. :(
jazzoberoi
Enthusiast
Posts: 96
Liked: 24 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by jazzoberoi »

Same thing is happening to us. Works beautifully for a few days and sometimes just sits at building the VM's list till the server is restarted.
Vitaliy S.
VP, Product Management
Posts: 27371
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by Vitaliy S. »

Jazz, in order to troubleshoot this issue quicker, please open a support case.
mbrinkho
Enthusiast
Posts: 28
Liked: 2 times
Joined: Jan 03, 2014 5:14 pm
Full Name: M Brinks
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by mbrinkho »

Opened a case, 01771011.
peter_cosa
Lurker
Posts: 1
Liked: never
Joined: Apr 06, 2016 12:11 am
Full Name: Peter
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by peter_cosa »

any fix for this?
bcann
Novice
Posts: 6
Liked: never
Joined: Oct 30, 2015 3:28 am
Full Name: Brad Cann
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by bcann »

I found that if you backed up via selecting the host rather then the individual VM's, AND you had application aware on, it would happen a lot more often. when I selected the individual VM's and only the application aware items on the specific servers it seemed to disappear.
mbrinkho
Enthusiast
Posts: 28
Liked: 2 times
Joined: Jan 03, 2014 5:14 pm
Full Name: M Brinks
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by mbrinkho »

Installing Update 1 fixed this for us.
kamiwa
Novice
Posts: 3
Liked: 3 times
Joined: Apr 30, 2016 3:10 pm
Full Name: Kai Michael Wadsack
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by kamiwa »

Installing Update 1 fixed this for us.
Installing Update 1 introduced this bug for me.

As general advice seems to open a support ticket with Veeam:
How do I open a support ticket?
Vitaliy S.
VP, Product Management
Posts: 27371
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by Vitaliy S. »

You can do that via our website > https://www.veeam.com/support.html
kamiwa
Novice
Posts: 3
Liked: 3 times
Joined: Apr 30, 2016 3:10 pm
Full Name: Kai Michael Wadsack
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by kamiwa » 2 people like this post

Thanks. Already found out by myself and support got back to me with a hotfix that I'm currently testing.
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by mkaec »

I had this happen twice this weekend. This was the first time it happened with backup jobs. Previously, it had only happened with BCJs. It happened both times with the same job.
Vitaliy S.
VP, Product Management
Posts: 27371
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v9 replication jobs getting stuck on 'Building VMs list'

Post by Vitaliy S. »

Marc, if you still experience this issue, then please reach out to your support team for the hotfix.
Post Reply

Who is online

Users browsing this forum: No registered users and 22 guests