Discussions specific to the VMware vSphere hypervisor
Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by Gostev » Nov 23, 2010 9:40 pm

cbdbwalker wrote:The DC will boot into safemode, attempt to shutdown and then hang. It will never reboot into normal mode to continue the SureBackup job.
Here is your issue (SureBackup only helped to detect it). I would assume that the same thing will happen if you try real production restore. I would start troubleshooting from trying to perform new full backup of this server.

ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by ptmartin » Nov 24, 2010 1:34 pm

In my lab environment, the DC initially boots into safe mode then reboots. Mine does not hang on reboot, though. It starts fine the second time and eventually the email server starts. This server starts normally. Again, this entire process takes anywhere between 30 and 45 minutes.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by Gostev » Nov 24, 2010 1:47 pm

Wow, it always takes less than 10 mins for me to start DC w/all reboots required, even when running SureBackup VM in virtualized ESXi inside VMware Workstation on regular desktop (arguably slowest VMware deployment you can imagine). I wonder why it takes so long on real hardware for you.

Or, are you saying it takes 30 to 45 mins to start multiple VMs?

ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by ptmartin » Nov 24, 2010 1:57 pm

It takes that long for the DC to finally finish its configuration in the application group and then I get to wait another 20 minutes or so for the email server. When these 2 are done, it usually takes close to an hour, the servers in the surebackup job then start. As of yesterday they are still failing but I have another attempt going that I started just about an hour ago. Check this out:

11/24/2010 8:01:13 AM Getting virtual lab configuration
11/24/2010 8:01:51 AM Starting virtual lab routing engine
11/24/2010 8:02:05 AM PGI-DC1: Publishing
11/24/2010 8:02:11 AM PGI-DC1: Updating configuration
11/24/2010 8:02:16 AM PGI-DC1: Registering
11/24/2010 8:02:54 AM PGI-DC1: Configuring DC
11/24/2010 8:02:58 AM PGI-DC1: Configuring networks
11/24/2010 8:27:34 AM PGI-DC1: Powering on (IP address: 192.168.254.11, network: 192.168.254.0, mask 255.255.255.0, gateway: 192.168.11.156)
11/24/2010 8:27:35 AM PGI-DC1: Pinging
11/24/2010 8:29:35 AM PGI-DC1: Application initialization
11/24/2010 8:29:37 AM PGI-DC1: Running test scripts
11/24/2010 8:29:36 AM Global Catalog script, path VmConnectionTester.exe, ErrCode 0
11/24/2010 8:29:36 AM Domain Controller script, path VmConnectionTester.exe, ErrCode 0
11/24/2010 8:29:37 AM DNS Server script, path VmConnectionTester.exe, ErrCode 0
11/24/2010 8:30:02 AM PGI3: Publishing
11/24/2010 8:30:05 AM PGI3: Updating configuration
11/24/2010 8:30:11 AM PGI3: Registering
11/24/2010 8:30:24 AM PGI3: Configuring networks
11/24/2010 8:47:40 AM PGI3: Powering on (IP address: 192.168.254.16, network: 192.168.254.0, mask 255.255.255.0, gateway: 192.168.11.156)
11/24/2010 8:47:41 AM PGI3: Pinging
11/24/2010 8:49:41 AM PGI3: Application initialization
11/24/2010 8:49:42 AM PGI3: Running test scripts
11/24/2010 8:49:42 AM Mail Server script, path VmConnectionTester.exe, ErrCode 0
11/24/2010 8:56:09 AM PGI4: Publishing
11/24/2010 8:51:01 AM PGI8: Publishing
11/24/2010 8:49:45 AM PGI2: Publishing
11/24/2010 8:51:05 AM PGI8: Updating configuration
11/24/2010 8:51:05 AM PGI8: Registering
11/24/2010 8:56:13 AM PGI4: Updating configuration
11/24/2010 8:56:13 AM PGI4: Registering

Fun Fun Fun ....

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by Gostev » Nov 24, 2010 2:42 pm

Yes, this is about 3 times slower than virtual ESXi can do it on my desktop :) but I have backup located directly on Veeam Backup server. Sounds like your backup storage, or connection to it has some serious problems. 30 minutes is well beyond expected numbers.

Since using DC adds complexity, I recommend getting clean Windows install to test this on (not part of domain, so it does not wait to hear for DC), and experiment with different backup storage placement. To start, try backing up this server locally to Veeam Backup server disk. In this scenario, boot time (until logon screen) should be around 1 min, definitely no more than 2 minutes. Then, move backup to another backup storage, and try the same again.

We actually did similar testing while certifying 3rd party storage last week, with backup stored on CIFS share connected to backup server via 1Gb LAN boot time was 2:15 (and this was deduplicating storage device, so its read IOPS and random read capabilities were noticeably lower than ones of raw disk). The original VM boot time was 1:35 (same VM but in actual production environment, with disks located on production storage).

ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by ptmartin » Nov 24, 2010 7:29 pm

As a test, I am increasing the block size on the datastore that houses my virtual lab. I have a 2mb block size configured for one of my datastores and a 1mb block size on the volume that the lab belongs to. As you can imagine this is taking a while to migrate my machines around so I can destroy and recreate the datastore. I doubt I'll have time today to do any further testing once things are back to an operational standpoint so I'll check back here on Monday and check/post any further results. I know that one of my machines is too big for a 1mb block size. Trying to eliminate all possibilities.

Anton - I discovered this while trying your suggestion. Until I go further I want to eliminate this first.

tsightler
VP, Product Management
Posts: 5294
Liked: 2146 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by tsightler » Nov 24, 2010 7:47 pm

One thing you might want to make sure of is that there is a DNS server in the virtual lab with your DC. DC's perform a lot of DNS lookups and registrations and if your don't have a DNS server in the virtual lab the DC can be VERY slow to come up, and may fail to come up at all. I know in many environment the DC's also double as DNS servers, but if they are separate this can be a very big issue.

Exchange also needs a DNS server, as well as access to a global catalog DC so make sure the DC you include with your Exchange lab is also a global catalog.

ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by ptmartin » Nov 24, 2010 7:50 pm

My DC is a DNS server and a DHCP server and has a copy of the Global Catalog.

ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by ptmartin » Dec 01, 2010 10:17 pm

Interesting results today. I finally got to test after changing datastore block sizes. That seemed to help one of the issues I had. Now that I have a consistent block size across datastores I am at a point where vmware tools are not registering in a timely manner. I'll try again tomorrow and disable the heartbeat check on all machines. I keep coming back to the same problem and it all seems to be speed related. I'll try again tomorrow and update this board with the results.

ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by ptmartin » Dec 06, 2010 5:27 pm

Hello All -
As was originally thought, it all came down to speed. I worked with Vitaliy last Thursday and after verifying the environment was set up correctly, he increased the startup timers on all backups to double or nearly double what they are by default. It is a painfully slow process but the results were 100% successful. He said he would have an explaination for the speed issue so I am very much looking forward to that. SureBackup is an amazing accomplishment so I am very happy to have this finally closer to a resolution.

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by Gostev » Dec 06, 2010 5:41 pm

Where do you store your backups? This is first and last thing to look at if there are any speed issues with vPower engine ;)

ptmartin
Enthusiast
Posts: 46
Liked: 10 times
Joined: Aug 17, 2010 3:35 pm
Full Name: Paul Martin
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by ptmartin » Dec 06, 2010 6:12 pm

My Veeam backups are going to a NAS device currently. If that were the cause don't you think I would see the same or similar problems in the daily backups as well ... especially since I can run several daily backups at a time and not see an issue as significant as when running a single surebackup job? Not to be argumentative but what is the real reason because that does not make sense? :-)

Gostev
SVP, Product Management
Posts: 24017
Liked: 3254 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by Gostev » Dec 06, 2010 6:52 pm

Think again... these are two absolutely different workloads from backup storage perspective.

Backup job = write I/O on backup storage
SureBackup job = read I/O on backup storage

For a sake of experiment, try creating 2 jobs backing up the same VM locally to backup server, and to NAS. Then, perform instant VM recovery from both backups (no need to connect VM to network), and compare time it takes for VMs to boot.

fredjjg
Novice
Posts: 3
Liked: never
Joined: Nov 18, 2010 3:53 pm
Full Name: frederic joneau
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by fredjjg » Dec 15, 2010 1:29 pm

Hello, I'm back!
Finally after many test in my environement, it works very well.
My solution was to change the "maximum allowed boot time" to 2100 sec for all servers (I will test with less time later) AND to change the "process simultaneously up to " 1 VM only.

Alexey D.

Re: surebackup "vmware tools did not start in a timely manne

Post by Alexey D. » Dec 15, 2010 1:40 pm

Frederic, thanks for update!

Post Reply

Who is online

Users browsing this forum: Majestic-12 [Bot] and 16 guests