Disaster recovery orchestration for the Enterprise (formerly Veeam Availability Orchestrator)
Post Reply
Markus.K1985
Veeam Vanguard
Posts: 103
Liked: 28 times
Joined: Dec 08, 2014 2:30 pm
Full Name: Markus Kraus
Contact:

Lab Test is stuck at waiting for VM

Post by Markus.K1985 »

Hello,

I have created a simple Lab with one VM on a Singe ESXi Host. The Lab is tarting, but the Lab Test is stuck at waiting for VM Heartbeat. VMware Tools are running and ESXi Hosts receives VM Heartbeat. Veeam B&R Job "Vm Start Job" is in Status -
Working.

Code: Select all

The File Log of the Job repeats:
[i][06.06.2018 11:13:25] <01> Info     [StableIp] Searching for real IP-address
[06.06.2018 11:13:25] <01> Info     [StableIp] Network adapter: Index=0, DeviceConfigId=4000, IpAddress='fe80::250:56ff:feb1:dce1'
[/i]

And exits with:
[i][06.06.2018 11:14:37] <01> Info     [StableIp] IP-address hasn't been detected
[06.06.2018 11:14:37] <01> Info     [StableIp] ===============================================================================================================================================
[06.06.2018 11:14:37] <01> Info     [StableIp] ====
[06.06.2018 11:14:37] <01> Info     [StableIp] |  |
[06.06.2018 11:14:37] <01> Info     [StableIp] ====
[06.06.2018 11:14:37] <01> Info     [StableIp] |  |
[06.06.2018 11:14:37] <01> Info     [StableIp] ====
[06.06.2018 11:14:37] <01> Error    Results: OS did not boot in the allotted time (System.TimeoutException)
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.Core.CViVmStartStableIpAction.<>c__DisplayClass3.<Execute>b__0()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CIpDetectionWaitTask.CheckDeadline(DateTime deadline)
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CIpDetectionWaitTask.RunImpl()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CStableIpWaitTask.Run()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CAnalyzeAlgorithm.Execute()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CAnalyzeAlgorithm.Analyze(CStableIpAddress address, DateTime deadLine, Int32 timeOut, TimeOutDelegate timeOutDelegate, CheckStoppedDelegate checkStoppedDelegate, CLogDelegate logDelegate, CAlgorithmStateChangedDelegate stateChangedDelegate, CStableIpAlgorithmParams parameters)
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CRebootTolerantAnalyzeAlgorithm.Execute()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CStableIpAlgorithm.AnalyzeMode()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.Core.CViVmStartStableIpAction.Execute()
[06.06.2018 11:14:37] <01> Info         [Soap] Connection 'esxi-10.lab.local:443:root:False::0:1' is disposing.
[06.06.2018 11:14:37] <01> Error    Results: OS did not boot in the allotted time (System.TimeoutException)
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.Core.CViVmStartStableIpAction.<>c__DisplayClass3.<Execute>b__0()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CIpDetectionWaitTask.CheckDeadline(DateTime deadline)
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CIpDetectionWaitTask.RunImpl()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CStableIpWaitTask.Run()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CAnalyzeAlgorithm.Execute()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CAnalyzeAlgorithm.Analyze(CStableIpAddress address, DateTime deadLine, Int32 timeOut, TimeOutDelegate timeOutDelegate, CheckStoppedDelegate checkStoppedDelegate, CLogDelegate logDelegate, CAlgorithmStateChangedDelegate stateChangedDelegate, CStableIpAlgorithmParams parameters)
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CRebootTolerantAnalyzeAlgorithm.Execute()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.SureBackup.Common.CStableIpAlgorithm.AnalyzeMode()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.Core.CViVmStartStableIpAction.Execute()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.Core.CViVmStartEngine.Run()
[06.06.2018 11:14:37] <01> Error       at Veeam.Backup.Core.CVmStartJobExecuter.Execute()
[06.06.2018 11:14:37] <01> Info     Session completed with error. Job: [Vm Start Job]
[/i]
Is a real IP in the VM necessary für the VAO LAB (VMware Heartbeat does not have this requirement)?
Any hints where I can start troubleshooting this situation?

Best regards,
Markus
Alec King
VP, Product Management
Posts: 1445
Liked: 362 times
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Lab Test is stuck at waiting for VM

Post by Alec King »

Hi Markus,

Is the VM configured to get IP v4 address?
Markus.K1985
Veeam Vanguard
Posts: 103
Liked: 28 times
Joined: Dec 08, 2014 2:30 pm
Full Name: Markus Kraus
Contact:

Re: Lab Test is stuck at waiting for VM

Post by Markus.K1985 »

I have configured a IPv4 to my Replication VM (test) and now, as expected, the VM Heartbeat was detected. But I am wondering about this behavior, because a configured IP is not VM Heartbeat. I would expect a separate Test for that.

After "Fixing" this issue my Lab Test is still exiting with warning:
Recovery Step Details
Check VM license and availability (Warning)
Timestamp Details
12:53:06 License has not expired
12:53:06 License is not exceeded
12:53:06 Waiting VM for availability...
12:53:06 VM is already testing
Process Replica VM (Skipped)
Timestamp Details
12:53:06 Skipped: VM is already testing
Check VM Heartbeat (Skipped)
Timestamp Details
12:53:06 Skipped: VM is already testing

The File log only throws one warning:
[06.06.2018 14:10:27] <01> Warning Unknown guest id vmwarePhoton64Guest

Any hint why "VM is already testing" ?
Alec King
VP, Product Management
Posts: 1445
Liked: 362 times
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Lab Test is stuck at waiting for VM

Post by Alec King »

Our initial check called 'Heartbeat' has been present in SureBackup / SureReplica for years, and is actually a check of -
* VM Powered on successfully
* Network initialised
* VMTools heartbeat
This algorithm is built into B&R and not easily changed!

Regarding 'VM already testing' - try Power OFF the lab in VAO UI, and make sure that Lab Appliance in B&R and vCenter also shows powered off, and all VM replicas powered off and reverted to Ready state.
Markus.K1985
Veeam Vanguard
Posts: 103
Liked: 28 times
Joined: Dec 08, 2014 2:30 pm
Full Name: Markus Kraus
Contact:

Re: Lab Test is stuck at waiting for VM

Post by Markus.K1985 »

I have Reset the State of the whole Lab -> same Issue

In my config I have not enabled the Network Appliance in Lab Settings, is that a possible problem?
Alec King
VP, Product Management
Posts: 1445
Liked: 362 times
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Lab Test is stuck at waiting for VM

Post by Alec King »

You definitely need to Enable the lab (for your desired Site, if more than one Site)
In Configuration - Plan Components, on Virtual Labs tab, check the box for the lab.
After it's checked, you should see this lab in the Home area of UI, in Virtual Labs node.

Select the lab and Edit to configure it with a 'test environment' (it looks very similar to editing a Plan). I believe at a minimum you will need to add a Domain Controller to the test environment so that your plan VMs can boot in the isolated network. (note -the DC must be replicated by B&R)

With above complete, you can select your plan in Failover Plans page and choose Verify - Run Lab Test.
You will see everything start in order -
1. Lab Appliance
2. Lab Groups (e.g. domain controller)
3. Your chosen Plan

Hope that helps!
Alec King
Vice President, Product Management
Veeam Software
matteo.ghadirpour
Novice
Posts: 3
Liked: 2 times
Joined: Oct 10, 2019 10:04 am
Full Name: Matteo Ghadirpour Galogir
Contact:

Re: Lab Test is stuck at waiting for VM

Post by matteo.ghadirpour »

Markus.K1985 wrote: Jun 06, 2018 12:18 pm I have configured a IPv4 to my Replication VM (test) and now, as expected, the VM Heartbeat was detected. But I am wondering about this behavior, because a configured IP is not VM Heartbeat. I would expect a separate Test for that.

After "Fixing" this issue my Lab Test is still exiting with warning:
Recovery Step Details
Check VM license and availability (Warning)
Timestamp Details
12:53:06 License has not expired
12:53:06 License is not exceeded
12:53:06 Waiting VM for availability...
12:53:06 VM is already testing
Process Replica VM (Skipped)
Timestamp Details
12:53:06 Skipped: VM is already testing
Check VM Heartbeat (Skipped)
Timestamp Details
12:53:06 Skipped: VM is already testing

The File log only throws one warning:
[06.06.2018 14:10:27] <01> Warning Unknown guest id vmwarePhoton64Guest

Any hint why "VM is already testing" ?
I have the same problem, but I correctly configured the network appliance. Any idea on how to solve it?
Alec King
VP, Product Management
Posts: 1445
Liked: 362 times
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Lab Test is stuck at waiting for VM

Post by Alec King »

Hi Matteo,
This error usually means that the replica VM is in some way busy. For example, already testing in another plan/group, or in-use by B&R server.

What's the status of your VM in the B&R console, in Replicas node?
matteo.ghadirpour
Novice
Posts: 3
Liked: 2 times
Joined: Oct 10, 2019 10:04 am
Full Name: Matteo Ghadirpour Galogir
Contact:

Re: Lab Test is stuck at waiting for VM

Post by matteo.ghadirpour »

Hi,
I would like to better explain my situation.
What I found peculiar is that in the Plan Group Details section of the Default Template Report I find this for every VM:
"Recovery Step Details
Check VM license and availability
Timestamp Details
11:31:16 License has not expired
11:31:16 License is not exceeded
11:31:16 Waiting VM for availability...
11:31:16 [Warning]VM is already testing
Process Replica VM
Timestamp Details
11:31:16 [Warning]VM is already testing
Check VM Heartbeat
Timestamp Details
11:31:16 [Warning]VM is already testing
Ping VM Network
Timestamp Details
11:31:16 [Warning]VM is already testing"

While in the DataLab Group Details section for the same VM i find this:
Recovery Result And Duration
Result Step Name Start Time Duration
 Success Check VM license and availability 11:12:42 00:00:00
 Success Process Replica VM 11:12:42 00:02:34
 Success Check VM Heartbeat 11:15:16 00:00:08
 Success Ping VM Network 11:15:24 00:00:18

Look at the timestamp, I powered on the DataLab at 11:11:40 and the VMs test were successful, Why 15 minutes later when I run the DataLab Test i get this [Warning]VM is already testing"?

To answer to your question "What's the status of your VM in the B&R console, in Replicas node?"
I didn't look at the B&R status and now I'm in ready status because I completely ran with no error nor warning the Orchestration Plan, in the VAO I'm in commit failback status.
Oleg Patrakov
Enthusiast
Posts: 84
Liked: 11 times
Joined: Mar 01, 2018 5:09 pm
Full Name: Oleg Patrakov
Contact:

Re: Lab Test is stuck at waiting for VM

Post by Oleg Patrakov » 1 person likes this post

Hi Matteo
It looks like you have the same VM in the Lab Group and in the plan itself. This will not work, as VM is already "in testing" as part of your Lab Group.
In general, the purpose of the Lab Group is to prepare a lab when you require some services VMs (such as DC, for example) to be started up and running for testing different plans (so you don't need to stop/start these service VMs for each and every plan).
In your case, if you only want to test a VM, just have it in the plan and remove it from the Lab Group.
If it's not the case, let me know.

At the same time you should not be testing the plan that is in "Commit failback" status. There's nothing to test in that case. Can you Reset the plan state and try testing without executing it first?

Thank you
matteo.ghadirpour
Novice
Posts: 3
Liked: 2 times
Joined: Oct 10, 2019 10:04 am
Full Name: Matteo Ghadirpour Galogir
Contact:

Re: Lab Test is stuck at waiting for VM

Post by matteo.ghadirpour » 2 people like this post

Thank you so much Oleg, I finally managed to solve my problem. I've reset the plan and removed the VMs from the Lab Group.
In this way the test was immediately successful with no warnings nor errors.

Thank you and have a good day.

Matteo
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest