VM stuck at "Getting VM info from vSphere"

[mmcgrath]

Hi all -

Running latest build of versino 8 (8.0.0.917) and have been having an issue where single VMs within a replication job get stuck at "Getting VM info from vSphere" and the duration does not move either. This is happening almost daily, so not only do I have to babysit my daily jobs -- but my backup/replication jobs are not finishing. This specific replication job ran fine in version, but has been problematic in version 8. It has 52 objects in it.

Any thoughts?

[Gostev]

Hi, Max. Please include support case ID for this issue, as requested when you create the new topic. Thanks!

[foggy]

Max, does the job get stuck on the same VMs each time? Do you need to stop the job manually to resolve this? What about manual retries or Quick Backup performed against the stuck VMs? And yes, logs review is required, so please contact support for assistance with this.

[Delo123]

I think we are also experiencing something like this... Backups start running, and will become "stuck" after a while. Our (virtual) Veeam Server will not respond anymore, (no rdp/console) and cpu usage is around 0. I then have to reboot the esx host where the veeam server resides, only after that i can remove snapshots on the vm's where the vm's are running... This has been since a few days, and dosen't affect all jobs. We ae using Hot Add
Veeam VM then must be unregisterd and reregistered. Upon start i notice the veeam server was actually still counting backup time in the jobs, but doing nothing...
I am off tomorrow and tuesday, but will create a Tcket on Wednesday.
For the meantime, i have quickly setup a physical veeam server and configured direct san access and am running full backups on all vm's now, hoping the most important ones finish until tomorrow morning

[mmcgrath]

Gostev, I apologize, I didn't know that was the protocol. I will open a support case tomorrow!

Foggy, it is generally 1 of 2 VMs -- but I have seen it be a number of different VMs. A lot of the time stopping the job will do nothing. I have been rebooting my Veeam server almost everyday to get jobs to "finish". I have never tried a quick backup against the stuck VM. As mentioned, I will open a case tomorrow.

[Primetrack]

Hi everyone.

We have exactly the same here since about two or three weeks.
Just opened a support case.

The VM's are failing randomly. Most with error 'Getting VM info from vSphere' and the lots of other errors like:

- Error: Agent with the specified identifier '{b3dc2fd8-e150-4b89-a693-955828c63b76}' does not exist. Agent with the specified identifier '{b3dc2fd8-e150-4b89-a693-955828c63b76}' does not exist.
- Error: Application is shutting down. Failed to download disk. Eine vorhandene Verbindung wurde vom Remotehost geschlossen. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

Hope Veeam Support can help out soon.

Patch 1 is installed and did not help at all.

Regards

[mmcgrath]

Gostev, my support case ID is 00719911.

Max

[Primetrack]

Mine is: Case # 00719600

[anthonysomerset]

I had the same issue with one particular VM

i turns out when i logged into vSphere client, i had to answer a VM question about locked CD drive, once i answered that for the affected vm, things started progressing normally, not sure if that affects you guys but might be worth checking out

[Vitaliy S.]

Hi Anthony, just out of curiosity - what was that question about? Did you have an ISO image attached to the VM or it was something else?

[anthonysomerset]

yes it was saying about locked cd tray and suggested that the VM might have locked the ISO for use and was asking if i wanted to remove the lock and disconnect the iso or not

[j.vanledden]

same error.
my SupportCase ID 00735825

regards,

[Primetrack]

It's really a strange thing.

Actually the support made already several remote sessions but could not resolve it yet.
We tried to disable CBT, Indexing, Guestprocessing, parallel processing... problem persists.
I turned it down that only one VM could write serial to the NAS... problem persists.

What I see is, that most jobs fail almost every time once. Some are on success state after retry 1 and some after retry 2.
But there are some VM's which are not backed up at all anymore.

Support added a registry key to the Veeam Server based on the following: https://msdn.microsoft.com/en-us/librar ... .105).aspx

It worked once because we backed up only one disk (out of five of this VM).. but now its not working anymore neither with all disks.
While testing this I came to the conclusion that I could split up this VM to backing up only one disk per job. So I added 5 new jobs (again with parallel processing, guest processing, indexing etc enabled!) and this night they were all but one on state success!
But this is the first full backup. Now I will see in this night if the normal reverse incremental backup is working also....

[LarsWulf]

Hi everybody,

since 2 days now I experience the same/similar issue. Veeam on version 8.0.0.917

Randomly backup jobs start failing with looking up the vSphere info and afterwards I get the error: Agent with the specified identifier '{d67c4f38-6f11-4e67-ba0b-9a8ceb0cd9f2}' does not exist.
After this happened to the first VM in a job the rest of the VMs for this job will fail. But lucky me the first retry always was successful for all Jobs and VMs.

In my case it is not related to specific jobs or VMs. I have different jobs and VMs failing.
The strange thing is I just moved my environment to different more powerful physical Veeam servers on 21.01. after that I only had successful jobs. But suddenly after no changes to anything not even Windows updates or vSphere updates have been installed these error occurs. The only thing changed is the number of VMs to backup.

Support case is open: 00739800

[Primetrack]

Like stated in my post before I splited one big VM into 5 different new jobs. Since then this VM is getting backed up again permantenly.
But other VMs still randomly fail.

[JamezUK]

Hi, Was there ever a resolution to this issue? I have started to experience it and was hoping for a quick fix without having to bother support.

Cheers

[foggy]

James, I've checked all the cases mentioned in this thread and in one of them the error went away after rebooting the server, another user switched to setting the proxy server manually, and in the third case error went away by itself. So no quick solution and I recommend to contact support for deeper investigation of your specific issue (logs review will be required).

[hbrusset]

Hi guys,

I used to have the same issue on a customer backup server, even after upgrading to 8.0.0.2030.

After analysing, in my case, VeeamVssSupport Service was running and still running after a reboot of the backuped VM.

Here is what I do to solve my issue :

• I just kill this service
• set startup to manual
• open a cmd as admin
• run : sc delete VeeamVssSupport
• and then finaly restart or retry the job

hope it can help you

[mikegmi]

We're getting the same error.

[foggy]

Mike, have you already contacted technical support? If no, please do.

[skrbnik]

Hi,

we have solved the issue after setting the proxy server manually and selecting the main Veeam Backup Server as the proxy.

What we don't understand is why this is working and not when we select our other Veeam Backup Proxy which is the host where the second repository disk in mounted...

Is this some kind of bug?

Thank you.

[foggy]

Only deep logs investigation can tell the actual reason of this behavior.

[fraack]

Hi Guys,

I have the same issue at one of my customers. I opened a ticket yesterday : Case # 01148396
We are running a centralized Veeam Infrastructure on Version 8 Update 3. We integratd for the first three offsite locations for the customer. Every offsite location (Itlay, Spain and Switzlerand) contains three ESXi 6.0 Hosts with a vSAN Datastore and a virtual machine acts as Proxy, Repository and WAN Acc. First we have local Backup Job in Hot Add Mode to this virtual machine. After that we perform a backup copy job to the HQ in Germany. Since a couple of days the local backup job stucks at "getting info from vSphere". This occurs only for Spain and Switzlerand. The Backup in italy runs very well. The bandwith is with 2 Mbit/s nearly the same for all locations.

The first support response was that there is a vCenter disconnect described in our logs. But I was wondering whay the itlian jobs makes no trouble. The Job defination is quite simple for the local Job:
- Reverse Incremental
- 10 RTs
- no active Fulls
- compression optimal
- block size default 1MB
- VSS enabled
- hotadd


May one of you guys have an idea.

best regards

Florian

[fraack]

I spend some further time with Troubleshooting yesterday and I figured out the main reason for this issue. First I set the Proxy for Job processing from automatic to Manual and I disabled VSS for this Job. With this Setting the Job ran successfull like estimated. After the Job was finished I enabled VSS again to double check this Setting. With VSS enabled the Job still stucks at getting info from vSphere. It seems that the Connection to the offsite Locations is to weak. So it takes to Long time to upload the guest Agent.

I know that this Problem is adressed with v9. So I discussed it with my customer and we diabled VSS for offsite Jobs till v9 is GA.

That works for me.

[actiefnet]

Having the same issues with v 9.0.0.1491. Anybody found solid solution for this?

[Vitaliy S.]

Please contact our support team for the debug logs review, as it is almost impossible to guess the reason for this issue via forums. Thanks!

[millahjovich]

Max, does the job get stuck on the same VMs each time? Do you need to stop the job manually to resolve this? What about manual retries or Quick Backup performed against the stuck VMs? And yes, logs review is required, so please contact support for assistance with this.

[hyvokar]

Same problem here with v9 u2 after moving from vcenter 5.5 to 6. Opening a support ticket.

[FMOFMO]

Same problem here with v9 u2 after moving from vcenter 5.5 to 6. Opening a support ticket.

I've got the same issue. I am running VBR v9u2, vCenter 6.0u1 with 2 HA clusters containing ESXi running 5.5u3p9 Build 4722766 or v6.0u1a.
Was your issue solved by support? I am considering upgrading to v9.5...

[foggy]

Hi Francoise, I'd recommend opening your own case as well. The reasons of the issue might differ between setups.