Building VMs list

[1-0-1]

Any specific why a backup job would hang for a long time on 'Building VMs list'. I have a backup job with its destination over a WAN link and noticed it runs for a long time.

[Gostev]

I believe what it does at this point is just talking to vCenter server.

[1-0-1]

Aha - In this particular example of mine it is running for 1 hour now and looking at the vCenter server I do not see any activity in terms of CPU/memory usage. I have a feeling this job will fail ...

[1-0-1]

Running now for several hours - unless somebody has a suggestion I am going to log a call.

[Vitaliy S.]

Yes, logging a support call with our technical team is definitely a good idea.

[ejleipold]

Hi 1-0-1,

Did you find a solution for this? I too am having jobs getting stuck on "Building VMs list" when the destination is across the WAN.

It looks like it's actually trying to download something, I'm guess it's trying to merge the incrementals with the full backup (I'm using reverse incrementals)

[Vitaliy S.]

On "Building VMs list" step Veeam backup establishes connection to the source vCenter Server, destination folder is checked on the other step...

[lars@norstat.no]

Exactly what data are you fetching during building VM List and why is this necessary ? Your already know what servers to replicate,i can't imagine that when starting a backup job is the best time to update list's ....

I'm asking because this task accounts for about 90% of the processing time (non data transfer time)

I've just really cleaned up both Vcenter database and Veeam database. Removed 50 GB of data from the Vcenter database and 15 GB from the Veeam database and rebuilt all tables and indexes. Of course i had to delete all my 70 replicas because of the bug that causes many jobs to run extremly slow after upgrade from 6.5 to 7 ...

Even so the UI in both the Vsphere Client and the Veeam Console is now blazing fast and theres is almost a negative delay

That is why i don't understand why this task is so slow when the connection to the Vcenter is so fast. Everything is hosted on the same datacenter and there is no network problems of any kind.

One theory i have is that you somehow are collecting a list of every ESXi server and virtual machine connected to the Vcenter even if i only replicate from the two that are local, since i have about 30 other ESXi servers located all over europe (some with delays above 20 ms) this would be very bad.

This is further strengthened by the fact that editing a job is still very slow and searcing for a machine inside the job editor is also slow.

Seems that either the problem is what i'm describing or there is some bug in Veeam that causes problems when connecting to the Vcenter.

I just want to repeat that everything else is blazing fast on both systems and other systems connecting to Vcenter (Mainly web client and Vsphere client) have no delays doing any tasks.

[Gostev]

Exactly what data are you fetching during building VM List and why is this necessary? Your already know what servers to replicate,i can't imagine that when starting a backup job is the best time to update list's ....

We don't necessarily know what servers to replicate. For example, if your job is configured by adding container, we need to expand it each time the job runs in order not to miss any newly added VMs. Otherwise, you will have much bigger complaints to us ("why the job reports success, but my new VM is not being backed up?")

And even when you add VMs explicitly to the list, we still need to look up lots of information, such as current VM disks (hot add makes it really easy to add new disks to a running VM), or their current location (think Storage VMotion).

[lars@norstat.no]

Of course ... Thnx

Will try to find a way to speed this up then, should not take this long to check for current disk and location ... 2-3 seconds maybe ... i have the whole Vcenter database in RAM and Veeam server is running on the same ESXi host ...

[Gostev]

I've been however pushing developers to add some sort of infrastructure caching, and update the cache only once a few hours, instead of having each job read the information from vCenter. We will, however, need to be careful with the cache expiration period, for the reasons I have stated above. I do have some creative ideas around this (some advanced vCenter capabilities might help here), but my theories must be verified first.

Hopefully, devs will have time and resources for this feature sooner, rather than later. Again, I am pushing for this feature hard.

[lars@norstat.no]

Yes because when you have jobs that are running for 5 minutes total, that minute is 1/5 of the total replication time. But even if you collect the information every time it should not take this long. There is one VM in every job and asking the database what disks it have and where they are located are 2-3 lines of text ... Hard to understand the extra 58 seconds

[Gostev]

Talking to vCenter is indeed not a fast thing. Partly because it is a web service API (SOAP), so there is lots of chit-chat involved, nothing like a quick direct query into the database.

[lars@norstat.no]

Would it be possible to set up a system where the customer can choose and allow Veeam to talk directly to the Vcenter database ? Or would Vmware not allow it ?

[Gostev]

In theory, anything is possible... and VMware would probably be OK with read-only access. But it certainly would be nightmare to maintain, because database changes every version. For example, I always recommend everyone against hooking up directly into Veeam configuration database for this very reason. Even though our DB is a few times smaller, and is very easy to understand (or so said the users who did not listen to me)

[lars@norstat.no]

I also haven't listen and think your database is indeed nice

Hope you never stop trying to get more speed because even minutes for every job translates in to a lot more jobs pr day.

I have seen jobs where more time is being spent on "paper work" rather than transferring data, although it's become by mangitude better after database optimizations thanks to Alexander at Veeam Support

[lars@norstat.no]

I was thinking. I understand that to you DatacenterESX01, FinlandESX01 and GermanyESX01 could be all the same or to put it another way. You could imagine that i would move a server that i'm backing up from Datacenter to Finland. And from Central_Storage to Germany_Storage. They could all be named E01, E02 etc for that matter ....

But for me i would know that would never happen ... Because they are different countries and there would be no reason for me to do that. I would know that the only backups that i do is from ESX03 and ESX04 to DESX01 and DESX02 ....

Is there a hack, database edit or something that i can do to exclude all my other ESXi host from all checks and updates like the one performed in Building VM list ?

[chrisdearden]

If your other environments are split up via datacenters , you can create a scoped account that only has admin permissions to that datacenter.

[lars@norstat.no]

Of course, now why didn't i think of that

Just hope there won't be a time out issue where Veeam will detect the rest but not have access .... I'l try right away ...

Thnx

[lars@norstat.no]

Not too much improvement, but it now takes 48 seconds to build VM list where it before always took exactly 1 minute.

[chrisdearden]

48 Seconds about right there Lars, I'm not sure how much faster we can get it unless you switch your vcenter to an all SSD database The above trick is very handy where there is a single VC, covering hosts over the globe ( sometimes over a poor quality link )

[lars@norstat.no]

I could possible do that, but also my whole Vcenter database is in RAM so it should be faster ... We did a test today where my colleague changed the storage placement for a server and i had to find out where it was moved manually via the vcenter client and i did it in about 30 seconds

Editing jobs are much faster though ....

[Gostev]

BTW, I had a meeting on this issue with the devs last week, and they promised me some minor optimizations (that should signficantly reduce the time of building VMs list) for the next minor release. So, let's wait and see what they will come up with

[lars@norstat.no]

Great news !! Thnx

I'm down to about 4 minutes pr replica now, and i'm aiming for 3 minutes. Then there will almost just be the snapshots and data transfer left and that is something we can never get rid of

[lars@norstat.no]

i found this from VMware:

http://kb.vmware.com/selfservice/micros ... Id=1024874

Are you also using Browse Datastore to build the vm list ? If so i can maybe find a way to speed it up.

[Gostev]

No, we are not.