surebackup "vmware tools did not start in a timely manner"

[RumataRus]

Hi!
I have the same problem.
Is there common solution or the problem must be resolved personally?
I do not understand why I should change the "maximum allowed boot time" to 2100 sec for all servers AND to change the "process simultaneously up to " 1 VM only.

[Gostev]

The issue is typically caused by vPower NFS performance problems, so the above "fixes" should be easy to understand. Since VM takes too long to boot, let's maximize all timeouts to give it enought time to boot. Also, let's reduce simultaneous processing to 1 VM to reduce concurrent load on already slow performing vPower NFS server.

I would strongly suggest finding and fixing vPower NFS performance first (easiest way to do this is using Instant VM Recovery w/out connecting VM to network) before attempting to setup SureBackup jobs and attempting to resolve issues caused by slow vPower NFS performance. While settings above might help, it still not going to do you any good, because SureBackup job performance will still be awful unless you can ensure optimal vPower NFS server performance and thus fast VM boot up times (comparable to the corresponding production VM) first.

[RumataRus]

The issue is typically caused by vPower NFS performance problems

Anton, you are right. During Instant VM Recovery I have found that disk read & write latencies on VeeamBackup NFS datastore exeed 100ms.
And I wonder why, because this datastore is placed on fast RAID10 disk group.
Obviously, we need to ask for support.

[RumataRus]

I would strongly suggest finding and fixing vPower NFS performance first

vPower NFS performance is low.
Is there any guide for troubleshooting?
I found no options in Veeam Backup with regards to vPower NFS.

[Gostev]

Troubleshooting vPower NFS performance requires reviewing full log files to understand which operations take longer than expected, and determining bottleneck based on this information. AFAIK, vPower NFS server logging is disabled by default for performance reasons, but can be temporarily enabled by support engineer during troubleshooting.

If I wanted to troubleshoot the issue myself, assuming you have physical backup server, I would try creating new test backup on another storage (for example, system drive on backup server) and test the instant recovery. Fast storage is not really required, even 5400 rpm consumer hard drive should provide sufficient performance for on-the-fly backup data rehydration engine to become primary bottleneck. But this way, you are removing your current backup storage from the picture, which is often what is causing the problems. Even if you have local RAID array - non-optimal RAID cache settings, bad firmware, not up-to-date drivers, or even hardware issues... think Intel P67

If this does not change performance, I would look at network between backup server and ESX that vPower NFS is mounted on. This one is less likely to be the issue, nevertheless 1 Gb links like to failover to 100 Mb because of bad connection, etc.

And finally, I would try using different ESX server.

[pcrebe]

hI,
i've the same problem.

i tried to performa an IstantRestore of mine SBS2008 (DC+Exchange+Sharepoint) from my Qnap (raid5) destination backup: 30 mins for first boot and another 30 mins to have Ctrl+Alt+Canc window... Not really Istant!

The read and write latency of the veeam datastore is 150 or more msec, is it the bottleneck?

Do you think the SurebBackup fails for this NAS latency? A local destination in the backup server (maybe iscsi drive mounted) could help me for really IstantRestore and Surebackup check?

thanks

[pcrebe]

Only setting Startup Time to 5400 the test has Success.
89 mins for powerig on the VM

bye

[Vitaliy S.]

89 mins for powerig on the VM

That's a way too much. Should be at least 10x faster under normal circumstances, unless there is performance bottleneck affecting vPower NFS.

A local destination in the backup server (maybe iscsi drive mounted) could help me for really IstantRestore and Surebackup check?

Worth trying, however I would also recommend trying another ESX(i) host for that. Please let us know how it works for you, if you need any assistance please contact our support team.

[pcrebe]

Hi,
i've changed the host and the IstantRestore time reduces to 40 min.
thanks

[Gostev]

Hi Carlo, thanks for the update... this makes it close to regular replica boot time, which you said is taking 30 mins. So just 33% performance overhead from vPower engine comparing to raw storage, which is more inline with what is expected.

Another question is why it takes 30 min to start a regular VM from raw disk I guess this is specific to SBS having multiple applications on it, including Exchange.

Thanks.

[iroche]

Just to add I was seeing this very issue, I dialled down the simultaneous processing from 3 to 1 vms and the job is completing without error. It runs slower obviously but speed is not really an issue I just want the verification to run correctly so at this point I am happy . For info are backup server is a Dell R710 connected via SAS cable to a MD3000 & MD1000 (18TB of usable storage).

[pcrebe]

Hi,
i think that host CPU and RAM are important for SureBackup jobs, infact my job fails on one host older than another also if less consumed by VMs workload. Try to change virtual lab location.

Thanks
Carlo

[iroche]

Hi,
i think that host CPU and RAM are important for SureBackup jobs, infact my job fails on one host older than another also if less consumed by VMs workload. Try to change virtual lab location.

Thanks
Carlo

Ill give it ago I am runing my lab on a beefy Dell R900 4 quad core CPU and 128GB of RAM running a very light load so I would be surprised if that makes a difference.

[Patje]

Did you guys eventually resolved the issue?
I'm currently experiencing the same issue.
Veeam server is a Dell R510 with local disks.
There seems to be no issue on the server itself, I don't see any bottlenecks at the moment.
I've tried with a new virutal lab on a different host, same issue.
Read & write latency averages around 100ms on the veeam nfs store.
I've uninstalled antivirus software from the backup server but it didn't make any difference as well.

[mpasaa]

I tried this option a long time ago and it simply does not work in a way that is useful to us. My goal was to schedule surebackup jobs and have them periodically and randomly check backups and provide some sort of report that I could add to our ticketing system as a validation for backup jobs (i.e. when an auditor asks if we check restores). This option only ever had errors when they ran. The VMs usually booted up in this lab set up BUT if I have to manually check every job it defeats the purpose of using it and I just rolled back to standard version since there was no ROI for these "added features". I can see from many post similar issues are still evident in this new version too. Too bad...it's a great concept but needs much work.

[Gostev]

it's a great concept but needs much work

What kind of work specifically do you suggest it needs? This technology is 1.5 years old already, and by now we have many customers using SureBackup on massive scale with great success. Of course, you cannot use it if your backup storage is so slow that it takes 89 minutes to boot a single VM, such as in case with poster above. For example, storing backups on deduplicating storage device, attempting to boot a VM from remote backup repository, or some networking issues may be the cause.

Me personally, I've never seen this issue in my lab, where it takes 45 seconds to boot clean install of Windows Server 2003 from backup to logon screen. The lab is my 8 GB RAM desktop with VMware Workstation 8.0 running backup server VM, and ESXi host VM. This virtualized ESXi starts the instantly recovered VM (second level virtualization, hehe). All of that is running on a single spindle of a regular 7200 rpm hard drive. So really, it is hard to imagine slower environment

The first and foremost thing before using SureBackup must be quick Instant VM Recovery speed test. People tend to spend much time tweaking SureBackup timeouts, when they should really make sure vPower engine is performing well before doing anything else.

[mpasaa]

For one thing, you need to make this feature error free in terms of alerting. When I set it up, the vmtools piece ALWAYS showed a warning for the surebackup job and the setting to ignore vmtool failures or whatever it was at the time didn't resolve the issue. While you claim the feature is useful in testing backup jobs to ensure they truly boot the VM I say--it does NOT help me to schedule automatic surebackup jobs if every time I receive a warning or error alert in email and have to check the job--defeats the purpose of 'automatic' wouldn't you say.

With all due respect sir, this product is generally good BUT needs work. I'm sure if you are a developer you probably take criticism a little hard but from a customer's standpoint, I would probably tell colleagues and friends to test other products in addition to Veeam. There are simply too many issues (albeit usually minor) with this version. I have to stick with reverse incrementals because the other option just 'blows up' backups to the point where I have to manage disk space almost monthly if not more. I don't have unlimited disk space to retain backup images and I don't have time to babysit this application on a weekly basis or have to adjust and readjust rollback counts, etc. to save space. I also have to disable all of the application indexing and guest processing options because having them in place, again, always results in email alerts with warning or failed status for jobs forcing me to manually check each job for failure reasons. Basically, I have it configured in a minimal setup with many options disabled just so I can get successful, basic backups--period. Throw in intermittent GUI exception errors that your support cannot resolve and constantly tell me to uninstall and reinstall and it just becomes tedious to support this application.

At the current time, I have minimal settings and many things disabled and my backups work fine as I have tested basic restores and imports. Your older versions were more solid in my opinion and frankly I can do without these "new" features.

Sorry, I wish I could give this a glowing review but it's just not that great a product. It does what it does and all of these new features are just bloat in our environment and have never really worked in my tests.

[Cokovic]

Sorry, I wish I could give this a glowing review but it's just not that great a product.

Sorry but cant agree with that. It's one of the greatest products i've seen for the last years. And support was great so far too. Backing up 1500+ VMs every day without major problems.

[Gostev]

This only shows that this mostly depends on the environment. As I like to say, Ferrari is not meant to perform well off road. You need most basic truck (which is what you turn Veeam into by disabling most features). Otherwise, you A/C get clogged, your SRS bags go off when you hit some stone, your ESP will get you stack in the sand, etc. Not a pleasant experience for sure, and you will definitely recommend your friends to evaluate trucks based on your experience - this is quite understandable.

Likewise, in software world - when the environment has a lot of issues by itself, many product features will be failing left and right. And vice versa, in well maintained environment you will be backing up thousands of VMs without any issues.

No argument that we should make the product more resilient to environmental issues, and we have been doing this all the time to improve over reliability via lots of error handling and robust design. For example, we are adding features you will not find with any other vendors (like bottleneck analysis), to help customers pinpoint performance issues with their environments (which used to be the most common type of support case until v6).

The truth is, it is very rarely the product fault, according to the support statistics - which is the ultimate source of truth. Of course, from the customers perspective, it is always a product fault when something does not work. Nobody thinks their baby is ugly...

I think we can agree that we all can improve few things here and there that includes both us fixing bugs in our product, and you addressing issues in your environments.

[smcauley]

[merged]

I have a long standing support case that has been with Veeam since the 19/04/12. We have reached the stage now where we have a senior engineer talking to VMWare to help troubleshoot. Given what has come back from VMWare it looks like both parties are pointing the finger at each other. So I thought I would try the forum hoping some of you may have some ideas.

The SureBackup has 3 servers, DC/DNS, SQL and Sharepoint. DC starts fine, processing of SQL begins and we can see the server is up and waiting in the lab. Veeam fails to detect as the VMWare Tools service does not start. Log errors state that it did not start in a timely fashion. Backups work perfectly with no errors and are restorable.

To date we have tried:-

Re installing VMWare Tools
Changed the NIC on the VM from vmxnet3 to E1000
Increasing timeouts
Turned off firewall
Disabled AV software
Changed ESX hosts
Created new lab in different datastore
Updated Veeam to version 6.0.0.181
Updated Veeam to version 6.1
Changed VMWare Tools to login as administrator
Performed full backup with new job
Ran SB job with SQL server only
Ran SB job, when VM powered on, opened VM settings and uncheck 'Connected' check-box for all vNics, in order to avoid possible TCP timeouts.

Still no luck. Anyone out there with any suggestions? Case number is 5185983

Thanks!