surebackup "vmware tools did not start in a timely manner"

[fredjjg]

Hello,

I installed sure backup this week and it is a great product!
Since, I test and each time I start a job, it failed for 90% of my server.
Error is the same : "vmware tools did not start in a timely manner"

I tried to increase the "start up time" for "maximum boot time and application time out but it is the same.
I tried to change the "boot verification", same problem.

I'm sure that VMWare tools are installed on production VM.
When the VM is restored, it takes time to start the VM but at the end, the VMware tools are running.

Could you help me?

Thank you

Fred

[Gostev]

Hi Fred, strange - this has not been reported before. Please open a support case to let our technical staff inverstigate this deeper. Thanks!

[fredjjg]

Thank you Gostev, I will open a support case.

[ptmartin]

Hi again Anton -- add me to this list as well. Same symptoms ... "vmware tools did not start in a timely manner." This is while starting the DC in the lab environment. Timeout is set at 2100 seconds for the machine in the application group (35 minutes) so there should be ample time for the machine to startup I would think. I've also disabled vmware tools as a requirement for a successful startup and tryied again. Interestingly enough, the failure message was the same ... I've been throught the user guide, the v5 FAQ document, and researched the forum without success.

I've opened a support ticket as well - #542616

Session Information Follows:
11/19/2010 11:30:01 AM Getting virtual lab configuration
11/19/2010 11:30:34 AM Starting virtual lab routing engine
11/19/2010 11:30:45 AM PGI-DC1: Publishing
11/19/2010 11:30:50 AM PGI-DC1: Updating configuration
11/19/2010 11:30:56 AM PGI-DC1: Registering
11/19/2010 11:31:40 AM PGI-DC1: Configuring DC
11/19/2010 11:31:42 AM PGI-DC1: Configuring networks
11/19/2010 12:06:59 PM Fail PGI-DC1: Powering on
VMware Tools did not start in a timely manner
11/19/2010 12:06:59 PM Fail
VMware Tools did not start in a timely manner
11/19/2010 12:06:59 PM Sending email report
11/19/2010 12:07:04 PM PGI-DC1: Powering off
11/19/2010 12:07:05 PM PGI-DC1: Unregistering
11/19/2010 12:07:08 PM PGI-DC1: Cleaning up redo logs
11/19/2010 12:07:09 PM PGI-DC1: Unpublishing
11/19/2010 12:07:14 PM Stopping virtual lab routing engine
11/19/2010 12:07:14 PM Fail
VMware Tools did not start in a timely manner
11/19/2010 12:07:17 PM Job finished

Thanks again in advance,
Paul

[Gostev]

Hi Paul, did you try to open PGI-DC1 console around 10 minutes after the VM has been powered on, and see whats up with this VM and whether VMware Tools are in fact running?

[ptmartin]

I did and it looked ok once it had completed "rebuilding indices" presumably for active directory(?). The Vsphere client also confirmed that VMware tools were running so I'm at a loss on this one ...

[MSoft]

Hi, any feedbackup on this issue as I'm having the same problem?

[Vitaliy S.]

Hello, as far as I know Paul's case is still being investigated, so please contact our technical team for assistance as well. By the way, please make sure you're running the latest tools in the VM which you've added to SureBackup job.

[Arnold]

I had the same issue as the OP, what I forgot was that before I took a backup of the VM I disconnected the vNIC. On restore it displayed this exact same error.

Even disabling VMWare Tools and Ping as a pre-req of the Surebackup job made no difference.

If I connected the vNIC back and then took another backup, upon restore everything worked fine.

[ptmartin]

That is correct Vitality - it is being escalated to level 3 and involving some dev assistance potentially I am told. So far the only real progress has only been made in getting the lab to come up properly. We had to manually configure the proxy device rather than use the automatically configured settings. It does not look like the automatic configuration can work. The assigned IP should be an IP from the virtual machine network and for whatever reason, the automatic configuration gives it a 192.168.1.x IP address.

Now regarding VMWare tools .. I had to disable tools heartbeat detection in the configured application group servers (a DC and my email server) in order to start these properly and continue. These take an incredibly long time to come up by the way. I am loking at something between 30 and 60 minutes each time I make a change to test. Nobody knows why yet as far as I can tell.

I can get to a point where a machine is finally starting to be backed up but again, for whatever reason it fails to detect that VMware tools are installed and results in a failed backup. I have even disabled this check on the individual servers and so far have not had a lot of success - it still checks and fails even though virtual center says tols are there and running (the proper version by the way).

That's about it so far. The length of time it takes for the environment to come up makes testing and troubleshooting a real test of patience and does not allow for quick progression toward a solution. More to follow as progress is made ...

[cbdbwalker]

I am having this same issue as well on both my DC and Exchange server. I cannot even get the SureBackup for DC to work if I configure it as just a linked job and not part of an application group. That server also takes an incredibly long time to start. I'm going to open up a case of my own but I will follow this thread, hopefully we can get a resolution to this...

[ptmartin]

First thing to try is increase the start time delay timer. The servers in my app group are set at 2100 seconds for the DC (be sure to deselect VMware Tools heartbeat is present), and Exchange is set at 1800 seconds (again with deselecting VMware Tools). The servers in the backup operation are set at 1200 seconds delay.

[Alexey D.]

Folks,

Paul is right, first thing to try is increasing these timeouts.
Also, please take in account following points:
- make sure that target datastore where VBK resides has a good link to backup console, and there are sufficient IOPS left to serve VMs' startup from compressed/deduped VBK;
- also, enough resources (mem, CPU) are available at ESX hosting your virtual lab;

The lack of resources may be a direct showstopper.
Please check this and let us know how things go.

[Gostev]

You can simply open temp VM console in vSphere client, and see if it is booted, and if VMware Tools are in fact running in system tray.

[cbdbwalker]

I've set my timeouts for 45 minutes and I've been watching the console. The DC will boot into safemode, attempt to shutdown and then hang. It will never reboot into normal mode to continue the SureBackup job. There is no CPU, Disk or Network activity, but I can move the mouse around in the console. After the timeout, the VM powers down and the job fails.

[Gostev]

The DC will boot into safemode, attempt to shutdown and then hang. It will never reboot into normal mode to continue the SureBackup job.

Here is your issue (SureBackup only helped to detect it). I would assume that the same thing will happen if you try real production restore. I would start troubleshooting from trying to perform new full backup of this server.

[ptmartin]

In my lab environment, the DC initially boots into safe mode then reboots. Mine does not hang on reboot, though. It starts fine the second time and eventually the email server starts. This server starts normally. Again, this entire process takes anywhere between 30 and 45 minutes.

[Gostev]

Wow, it always takes less than 10 mins for me to start DC w/all reboots required, even when running SureBackup VM in virtualized ESXi inside VMware Workstation on regular desktop (arguably slowest VMware deployment you can imagine). I wonder why it takes so long on real hardware for you.

Or, are you saying it takes 30 to 45 mins to start multiple VMs?

[ptmartin]

It takes that long for the DC to finally finish its configuration in the application group and then I get to wait another 20 minutes or so for the email server. When these 2 are done, it usually takes close to an hour, the servers in the surebackup job then start. As of yesterday they are still failing but I have another attempt going that I started just about an hour ago. Check this out:

11/24/2010 8:01:13 AM Getting virtual lab configuration
11/24/2010 8:01:51 AM Starting virtual lab routing engine
11/24/2010 8:02:05 AM PGI-DC1: Publishing
11/24/2010 8:02:11 AM PGI-DC1: Updating configuration
11/24/2010 8:02:16 AM PGI-DC1: Registering
11/24/2010 8:02:54 AM PGI-DC1: Configuring DC
11/24/2010 8:02:58 AM PGI-DC1: Configuring networks
11/24/2010 8:27:34 AM PGI-DC1: Powering on (IP address: 192.168.254.11, network: 192.168.254.0, mask 255.255.255.0, gateway: 192.168.11.156)
11/24/2010 8:27:35 AM PGI-DC1: Pinging
11/24/2010 8:29:35 AM PGI-DC1: Application initialization
11/24/2010 8:29:37 AM PGI-DC1: Running test scripts
11/24/2010 8:29:36 AM Global Catalog script, path VmConnectionTester.exe, ErrCode 0
11/24/2010 8:29:36 AM Domain Controller script, path VmConnectionTester.exe, ErrCode 0
11/24/2010 8:29:37 AM DNS Server script, path VmConnectionTester.exe, ErrCode 0
11/24/2010 8:30:02 AM PGI3: Publishing
11/24/2010 8:30:05 AM PGI3: Updating configuration
11/24/2010 8:30:11 AM PGI3: Registering
11/24/2010 8:30:24 AM PGI3: Configuring networks
11/24/2010 8:47:40 AM PGI3: Powering on (IP address: 192.168.254.16, network: 192.168.254.0, mask 255.255.255.0, gateway: 192.168.11.156)
11/24/2010 8:47:41 AM PGI3: Pinging
11/24/2010 8:49:41 AM PGI3: Application initialization
11/24/2010 8:49:42 AM PGI3: Running test scripts
11/24/2010 8:49:42 AM Mail Server script, path VmConnectionTester.exe, ErrCode 0
11/24/2010 8:56:09 AM PGI4: Publishing
11/24/2010 8:51:01 AM PGI8: Publishing
11/24/2010 8:49:45 AM PGI2: Publishing
11/24/2010 8:51:05 AM PGI8: Updating configuration
11/24/2010 8:51:05 AM PGI8: Registering
11/24/2010 8:56:13 AM PGI4: Updating configuration
11/24/2010 8:56:13 AM PGI4: Registering

Fun Fun Fun ....

[Gostev]

Yes, this is about 3 times slower than virtual ESXi can do it on my desktop but I have backup located directly on Veeam Backup server. Sounds like your backup storage, or connection to it has some serious problems. 30 minutes is well beyond expected numbers.

Since using DC adds complexity, I recommend getting clean Windows install to test this on (not part of domain, so it does not wait to hear for DC), and experiment with different backup storage placement. To start, try backing up this server locally to Veeam Backup server disk. In this scenario, boot time (until logon screen) should be around 1 min, definitely no more than 2 minutes. Then, move backup to another backup storage, and try the same again.

We actually did similar testing while certifying 3rd party storage last week, with backup stored on CIFS share connected to backup server via 1Gb LAN boot time was 2:15 (and this was deduplicating storage device, so its read IOPS and random read capabilities were noticeably lower than ones of raw disk). The original VM boot time was 1:35 (same VM but in actual production environment, with disks located on production storage).

[ptmartin]

As a test, I am increasing the block size on the datastore that houses my virtual lab. I have a 2mb block size configured for one of my datastores and a 1mb block size on the volume that the lab belongs to. As you can imagine this is taking a while to migrate my machines around so I can destroy and recreate the datastore. I doubt I'll have time today to do any further testing once things are back to an operational standpoint so I'll check back here on Monday and check/post any further results. I know that one of my machines is too big for a 1mb block size. Trying to eliminate all possibilities.

Anton - I discovered this while trying your suggestion. Until I go further I want to eliminate this first.

[tsightler]

One thing you might want to make sure of is that there is a DNS server in the virtual lab with your DC. DC's perform a lot of DNS lookups and registrations and if your don't have a DNS server in the virtual lab the DC can be VERY slow to come up, and may fail to come up at all. I know in many environment the DC's also double as DNS servers, but if they are separate this can be a very big issue.

Exchange also needs a DNS server, as well as access to a global catalog DC so make sure the DC you include with your Exchange lab is also a global catalog.

[ptmartin]

My DC is a DNS server and a DHCP server and has a copy of the Global Catalog.

[ptmartin]

Interesting results today. I finally got to test after changing datastore block sizes. That seemed to help one of the issues I had. Now that I have a consistent block size across datastores I am at a point where vmware tools are not registering in a timely manner. I'll try again tomorrow and disable the heartbeat check on all machines. I keep coming back to the same problem and it all seems to be speed related. I'll try again tomorrow and update this board with the results.

[ptmartin]

Hello All -
As was originally thought, it all came down to speed. I worked with Vitaliy last Thursday and after verifying the environment was set up correctly, he increased the startup timers on all backups to double or nearly double what they are by default. It is a painfully slow process but the results were 100% successful. He said he would have an explaination for the speed issue so I am very much looking forward to that. SureBackup is an amazing accomplishment so I am very happy to have this finally closer to a resolution.

[Gostev]

Where do you store your backups? This is first and last thing to look at if there are any speed issues with vPower engine

[ptmartin]

My Veeam backups are going to a NAS device currently. If that were the cause don't you think I would see the same or similar problems in the daily backups as well ... especially since I can run several daily backups at a time and not see an issue as significant as when running a single surebackup job? Not to be argumentative but what is the real reason because that does not make sense?

[Gostev]

Think again... these are two absolutely different workloads from backup storage perspective.

Backup job = write I/O on backup storage
SureBackup job = read I/O on backup storage

For a sake of experiment, try creating 2 jobs backing up the same VM locally to backup server, and to NAS. Then, perform instant VM recovery from both backups (no need to connect VM to network), and compare time it takes for VMs to boot.

[fredjjg]

Hello, I'm back!
Finally after many test in my environement, it works very well.
My solution was to change the "maximum allowed boot time" to 2100 sec for all servers (I will test with less time later) AND to change the "process simultaneously up to " 1 VM only.

[Alexey D.]

Frederic, thanks for update!