Discussions specific to the VMware vSphere hypervisor
RumataRus
Enthusiast
Posts: 78
Liked: never
Joined: Jun 21, 2010 5:30 am
Full Name: Dmitry Prokudin

Re: surebackup "vmware tools did not start in a timely manne

Post by RumataRus » Mar 18, 2011 11:00 pm

Hi!
I have the same problem.
Is there common solution or the problem must be resolved personally?
I do not understand why I should change the "maximum allowed boot time" to 2100 sec for all servers AND to change the "process simultaneously up to " 1 VM only.

Gostev
SVP, Product Management
Posts: 24016
Liked: 3253 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by Gostev » Mar 19, 2011 12:23 am

The issue is typically caused by vPower NFS performance problems, so the above "fixes" should be easy to understand. Since VM takes too long to boot, let's maximize all timeouts to give it enought time to boot. Also, let's reduce simultaneous processing to 1 VM to reduce concurrent load on already slow performing vPower NFS server.

I would strongly suggest finding and fixing vPower NFS performance first (easiest way to do this is using Instant VM Recovery w/out connecting VM to network) before attempting to setup SureBackup jobs and attempting to resolve issues caused by slow vPower NFS performance. While settings above might help, it still not going to do you any good, because SureBackup job performance will still be awful unless you can ensure optimal vPower NFS server performance and thus fast VM boot up times (comparable to the corresponding production VM) first.

RumataRus
Enthusiast
Posts: 78
Liked: never
Joined: Jun 21, 2010 5:30 am
Full Name: Dmitry Prokudin

Re: surebackup "vmware tools did not start in a timely manne

Post by RumataRus » Mar 19, 2011 11:29 am

Gostev wrote:The issue is typically caused by vPower NFS performance problems
Anton, you are right. During Instant VM Recovery I have found that disk read & write latencies on VeeamBackup NFS datastore exeed 100ms. :(
And I wonder why, because this datastore is placed on fast RAID10 disk group.
Obviously, we need to ask for support.

RumataRus
Enthusiast
Posts: 78
Liked: never
Joined: Jun 21, 2010 5:30 am
Full Name: Dmitry Prokudin

Re: surebackup "vmware tools did not start in a timely manne

Post by RumataRus » Mar 19, 2011 1:20 pm

Gostev wrote: I would strongly suggest finding and fixing vPower NFS performance first
vPower NFS performance is low. :(
Is there any guide for troubleshooting?
I found no options in Veeam Backup with regards to vPower NFS.

Gostev
SVP, Product Management
Posts: 24016
Liked: 3253 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by Gostev » Mar 19, 2011 4:22 pm

Troubleshooting vPower NFS performance requires reviewing full log files to understand which operations take longer than expected, and determining bottleneck based on this information. AFAIK, vPower NFS server logging is disabled by default for performance reasons, but can be temporarily enabled by support engineer during troubleshooting.

If I wanted to troubleshoot the issue myself, assuming you have physical backup server, I would try creating new test backup on another storage (for example, system drive on backup server) and test the instant recovery. Fast storage is not really required, even 5400 rpm consumer hard drive should provide sufficient performance for on-the-fly backup data rehydration engine to become primary bottleneck. But this way, you are removing your current backup storage from the picture, which is often what is causing the problems. Even if you have local RAID array - non-optimal RAID cache settings, bad firmware, not up-to-date drivers, or even hardware issues... think Intel P67 :mrgreen:

If this does not change performance, I would look at network between backup server and ESX that vPower NFS is mounted on. This one is less likely to be the issue, nevertheless 1 Gb links like to failover to 100 Mb because of bad connection, etc.

And finally, I would try using different ESX server.

pcrebe
Enthusiast
Posts: 94
Liked: 1 time
Joined: Dec 06, 2010 10:41 pm
Full Name: CARLO
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by pcrebe » May 01, 2011 9:43 pm

hI,
i've the same problem.

i tried to performa an IstantRestore of mine SBS2008 (DC+Exchange+Sharepoint) from my Qnap (raid5) destination backup: 30 mins for first boot and another 30 mins to have Ctrl+Alt+Canc window... Not really Istant!

The read and write latency of the veeam datastore is 150 or more msec, is it the bottleneck?

Do you think the SurebBackup fails for this NAS latency? A local destination in the backup server (maybe iscsi drive mounted) could help me for really IstantRestore and Surebackup check?

thanks

pcrebe
Enthusiast
Posts: 94
Liked: 1 time
Joined: Dec 06, 2010 10:41 pm
Full Name: CARLO
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by pcrebe » May 02, 2011 7:54 am

Only setting Startup Time to 5400 the test has Success.
89 mins for powerig on the VM

bye

Vitaliy S.
Product Manager
Posts: 22281
Liked: 1411 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by Vitaliy S. » May 02, 2011 9:05 am

pcrebe wrote:89 mins for powerig on the VM
That's a way too much. Should be at least 10x faster under normal circumstances, unless there is performance bottleneck affecting vPower NFS.
pcrebe wrote:A local destination in the backup server (maybe iscsi drive mounted) could help me for really IstantRestore and Surebackup check?
Worth trying, however I would also recommend trying another ESX(i) host for that. Please let us know how it works for you, if you need any assistance please contact our support team.

pcrebe
Enthusiast
Posts: 94
Liked: 1 time
Joined: Dec 06, 2010 10:41 pm
Full Name: CARLO
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by pcrebe » May 03, 2011 7:27 pm

Hi,
i've changed the host and the IstantRestore time reduces to 40 min.
thanks

Gostev
SVP, Product Management
Posts: 24016
Liked: 3253 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by Gostev » May 03, 2011 8:09 pm

Hi Carlo, thanks for the update... this makes it close to regular replica boot time, which you said is taking 30 mins. So just 33% performance overhead from vPower engine comparing to raw storage, which is more inline with what is expected.

Another question is why it takes 30 min to start a regular VM from raw disk ;) I guess this is specific to SBS having multiple applications on it, including Exchange.

Thanks.

iroche
Enthusiast
Posts: 30
Liked: never
Joined: May 20, 2011 8:27 am
Full Name: Ian Roche
Location: Ireland
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by iroche » Sep 08, 2011 11:42 am

Just to add I was seeing this very issue, I dialled down the simultaneous processing from 3 to 1 vms and the job is completing without error. It runs slower obviously but speed is not really an issue I just want the verification to run correctly so at this point I am happy :D . For info are backup server is a Dell R710 connected via SAS cable to a MD3000 & MD1000 (18TB of usable storage).

pcrebe
Enthusiast
Posts: 94
Liked: 1 time
Joined: Dec 06, 2010 10:41 pm
Full Name: CARLO
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by pcrebe » Sep 08, 2011 12:16 pm

Hi,
i think that host CPU and RAM are important for SureBackup jobs, infact my job fails on one host older than another also if less consumed by VMs workload. Try to change virtual lab location.

Thanks
Carlo

iroche
Enthusiast
Posts: 30
Liked: never
Joined: May 20, 2011 8:27 am
Full Name: Ian Roche
Location: Ireland
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by iroche » Sep 08, 2011 12:21 pm

pcrebe wrote:Hi,
i think that host CPU and RAM are important for SureBackup jobs, infact my job fails on one host older than another also if less consumed by VMs workload. Try to change virtual lab location.

Thanks
Carlo

Ill give it ago I am runing my lab on a beefy Dell R900 4 quad core CPU and 128GB of RAM running a very light load so I would be surprised if that makes a difference.

Patje
Novice
Posts: 7
Liked: never
Joined: Jul 08, 2011 11:20 am
Full Name: Patrick van Beek
Contact:

Re: surebackup "vmware tools did not start in a timely manne

Post by Patje » Mar 20, 2012 12:30 pm

Did you guys eventually resolved the issue?
I'm currently experiencing the same issue.
Veeam server is a Dell R510 with local disks.
There seems to be no issue on the server itself, I don't see any bottlenecks at the moment.
I've tried with a new virutal lab on a different host, same issue.
Read & write latency averages around 100ms on the veeam nfs store.
I've uninstalled antivirus software from the backup server but it didn't make any difference as well.

mpasaa
Enthusiast
Posts: 36
Liked: 2 times
Joined: Sep 08, 2009 3:28 pm
Full Name: Mike Audet

Re: surebackup "vmware tools did not start in a timely manne

Post by mpasaa » Mar 20, 2012 4:34 pm

I tried this option a long time ago and it simply does not work in a way that is useful to us. My goal was to schedule surebackup jobs and have them periodically and randomly check backups and provide some sort of report that I could add to our ticketing system as a validation for backup jobs (i.e. when an auditor asks if we check restores). This option only ever had errors when they ran. The VMs usually booted up in this lab set up BUT if I have to manually check every job it defeats the purpose of using it and I just rolled back to standard version since there was no ROI for these "added features". I can see from many post similar issues are still evident in this new version too. Too bad...it's a great concept but needs much work.

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 19 guests