SureBackup Powering on Taking Forever

[dcs115]

Has anyone else experienced Surebackup's taking forever to change from "Starting" status to "Started"?

I am booting my DC and Exchange server, and it appears my first VM in the boot list (my dc) is going to take around 30 minutes to be detected as "Started" even though the VM is up and running and I can login etc.

Is there a way to change the default "Waiting for OS to boot for up to 2100 seconds (stable IP algorithm)..." 2100 seconds limit to something shorter? Obviously you want to make sure the VM is running before starting something else, but an hour to boot 2 VM's seems a bit excessive. Could I have a config issue?

Thanks!

dave

[Gostev]

You can change these timeouts in the SureBackup application group settings, however this is maximum timeout (up to). If the SureBackup job detects that OS is booted up earlier, it will proceed immediately. What is need here is to investigate why SureBackup job cannot see OS as running when OS is in fact started. This could be VMware Tools version issue, however best would be to open a support case and let our technical staff investigate your debug logs. Thanks!

[dcs115]

Thanks Gostev. I have been poking around a bit more and it appears that you are correct and the vmware tools service is not starting. What is odd is that it was working a few days ago and "nothing" has changed. If i login to the VM that SureBackup started and manually start the vmware tools service, it starts fine, but nothing seems to happen on the lab side (it still says "starting"). I will attempt a few more things and then give support a call. Thank you again!

[idroesse]

[merged]

We have installed vmware vsphere 5.0 and veeam backup and replication 6 in our network. all these setup are new. we newly installed exchange server.
We have backup of all these servers from VMs.
Today We setup surebackup job. Every thing was ok. Then We ran the surebackup job. surebackup job was for exchange. so from backup i added VMs for domain controller and exchange. it worked everything for fine. it booted the domain controller from backup. but then it stopped. after half an hour i found 2100 secs it stopped with error.

Fail says taht OS didnt boot in alloted time. but it booted. i can see when i clicked from link. or from vcenter console. but veeam surebackup cannot recognize that OS has booted or not.

Please help me why its not working for me.

[idroesse]

the logs are as follows:

12/14/2011 7:00:22 PM Fail Error: Results: OS did not boot in the allotted time

[idroesse]

I had no issue with vmware tools. its installed in the guest operating system. i found they are working fine. still why veeam is not recognizing that OS has boot up. Even thought the OS is powered on it stay as powering for 30 min(2100 seconds). after that it wait for next step to complete 30 min. after that it gives fail error as mentioned above realtime statistic log.

[Vitaliy S.]

idrosse,

The VM might be actually powered on, but I've seen lots of cases where VMware Tools had an up to date and healthy status in vSphere Client, but were still reporting incorrect Guest OS status. Please involve our support team into this as Anton has advised.

Thank you.

[idroesse]

I got reply from support that you need to double the boot time. I doubled the boot time for DC from default 2100 sec to 4200. then i got the following error:

12/15/2011 1:21:31 AM Error Summary: impossible to reconfigure Windows Registry

one hour time for one vm to boot is really a long time. and also im getting error. I dont know why its not communicating properly with VMware tools properly in veeam 6

[Gostev]

One possible reason is DC using unsupported disk type - our support can say for sure based on the debug logs.

[idroesse]

i checked the VM there is no dynamic disk in it. then why it need to show this problem.

[Gostev]

You need to contact our support for them to find the real problem.

[idroesse]

gostev,
today i found that vmware tool service for DC Vm that started from backup is not running. but i cross checked with DC VM in the production is running. why vmware tool service is not running from backup. i manually started service but it again stopped in ping test. why it is like that ?

Do i need to assign this switch to physical network adaptor?

[Gostev]

Not sure, cannot really comment on VMware Tools behavior and why they may not work. Try re-installing them on source VM and re-taking the backup, usually helps in this sort of situations.

[lwarner]

I've been having similar issues; I just started using SureBackup so I'm not an expert, but here's what I found so far. For my DCs, I had to set Veeam to run under an account with enough privileges to configure the DCs.

I also noticed that VMs with the older 'flexible' vNICs were not booting in time, and the jobs failed. The ones with vmxnet3 vNICS work fine, E1000 vNIC works fine. Note that if you have Win2008R2 / Win7 and use the vmxnet3 vNIC, you need the MS hotfix KB2550978 (or KB2344941 if you don't have SP1), otherwise when the restored VM gets a new NIC, and if you have a static IP (as with most servers), you won't get a valid IP address, and then the SureBackup job fails.

[idroesse]

i now removed the vmware tools and updating the windows server 2008 R2 SP1. then ill install it again. Test only passed heartbeat test but ping and script test failed.
Lets see what will happen after reinstalling vmware tools

[Gostev]

idroesse, do you have built-in firewall enabled on this server, since that would kill ping and any external connection attempts of course.

[idroesse]

yes the built in firewall is enabled on this server. but in the production DC i can ping without any problem. so it has to ping from the backup DC as well rite?

let me come to the matter. i tried as you told. i uninstalled vmware tools and installed it back. but it again failed. it the same results.

another thing i noticed is that every time when i try to start the dc from backup it booting as if there had been hard reset. that it says windows didnt shutdown properly message. if i manually log in vmware console i can see a message pop up and says windows need to restart. why this is coming? when i checked i found same message was there in dc as well. but i restarted and then uninstalled vmware tools and updated windows. now i cannot see any message like that. i made a backup after reinstalling vmware tools and then run the surebackup job. but the result was same.
Should i need to run a full backup? is this some thing related to full backup? and then try again for this?

Do i need to assign this surebackup network to any physical NIC? I didnt assign anything like that. its not connected to any physical NIC.

For your information i didnt get any tech support for past 2 days. no reply no updates. really poor tech support as of now.

[muhaynes]

When I setup my Virtual Lab, the Virtual Lab VM could not ping the restored VM instance unless I manually specified it's networking in the Advanced settings of the lab itself. This caused the lab to always fail the ping test even if the restored VM was up and running. I think this is because I run a funky expanded subnet on my local LAN 192.168.x.0/23) - maybe give that a try?

[idroesse]

I checked the windows log of DC for vmware tools service. I found the following error for vmware tool service .

Log Name: System
Source: Service Control Manager
Date: 12/17/2011 12:40:43 AM
Event ID: 7030
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: DC.idroesse.local
Description:
The VMware Tools Service service is marked as an interactive service. However, the system is configured to not allow interactive services. This service may not function properly.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
<EventID Qualifiers="49152">7030</EventID>
<Version>0</Version>
<Level>2</Level>
<Task>0</Task>
<Opcode>0</Opcode>
<Keywords>0x8080000000000000</Keywords>
<TimeCreated SystemTime="2011-12-16T20:40:43.505576800Z" />
<EventRecordID>3241</EventRecordID>
<Correlation />
<Execution ProcessID="464" ThreadID="3436" />
<Channel>System</Channel>
<Computer>DC.idroesse.local</Computer>
<Security />
</System>
<EventData>
<Data Name="param1">VMware Tools Service</Data>
</EventData>
</Event>

I found in the forum as


"From the description, it seems that the VMware Tools Service has been configured as an interactive service. The error occurs because support for interactive services has been removed to mitigate security risk in Windows Vista and Windows Server 2008.



As a workaround of the issue, you may cancel the "Allow service to interact with desktop" on the VMware Tools Service.



Event ID 7030 — Basic Service Operations

http://technet.microsoft.com/en-us/libr ... S.10).aspx



If you have any questions or concerns, please do not hesitate to let me know."

The forum link is http://social.technet.microsoft.com/For ... a4f7ab1258

vmware tool service is log on as 'local system account' with 'allow service to interact with desktop' checked.
Also for your kind information, i didnt assign the surebackup job network to any physical NIC. I also allowed ip address to assign automatically. i didnt go for advance options.

[idroesse]

When I setup my Virtual Lab, the Virtual Lab VM could not ping the restored VM instance unless I manually specified it's networking in the Advanced settings of the lab itself. This caused the lab to always fail the ping test even if the restored VM was up and running. I think this is because I run a funky expanded subnet on my local LAN 192.168.x.0/23) - maybe give that a try?

hmm that was the same thing i was thinking of. because i used the vm network to automatically assign ip address. but my local LAN of the range 192.168.x.x/24. i didnt assign any physical NIC for this network.

I found the following error for the vmware tools service

"The VMware Tools Service service is marked as an interactive service. However, the system is configured to not allow interactive services. This service may not function properly."

i dont know whether i want to run the vmware tools service using log on as local system account or domain account .

[lwarner]

What's a reasonable boot time for a regular file server? For a DC? It seems like it's taking forever for mine to boot also, but I don't have a reference point.
All our backup files, and the vPower NFS are on fairly quick DAS, and everything including the Veeam server and Virtual Lab is on the same physical server.

[Gostev]

About 45 sec for Windows Server 2003 file server in my lab - which is my desktop with Veeam B&R installed on it, and VMware Workstation running virtualized ESXi. VM boots up from vPower NFS inside this virtualized ESXi. The backup file for this VM resides on the local hard drive.

[idroesse]

after 2 days i found the real problem. i was thinking about it initially itself. The problem was due ot symantec end point protection 12.1 anti virus. It has only anti virus role ie no firewall and other roles. after i remove the anti virus and run the backup and then i did surebackup .its now working fine.
But now my problem is how can i install my antivirus without any problem for surebackup job?

[Vitaliy S.]

You need to add VMware tools process to exclusion lists, besides you should allow to access registry settings for external processes, such as Veeam. This is required to verify domain controllers in the SureBackup job.

[dcs115]

From my talks with support and personal findings, it looks like the VM (normally the DC) never boots because vmware tools does not start, ever (obviously noted in the above comments). If you login to the vm and look through the windows logs (i am on server 2008 r2), you will see that vmware tools hit its timeout threshold and will not try to start again. My personal opinion is that it times out because the network service (or some other service) takes so long to start due to the fact that the backup is being run off of slow storage (in my case 1 7200RPM SATA hard drive).

My workaround for this was to create a batch file, tell it to "net start vmtools" and add a scheduled task to run it on startup of the VM. By the time the scheduled task service is up and running, anything that VMtools relies on (not service dependencies, but actual windows components) is running and about 5-8 minutes after you see the CTRL+ALT+DEL screen, vmtools will start and the the VM will proceed to its checks. I am running static IP's on all of my Server's.

[dcs115]

You need to add VMware tools process to exclusion lists, besides you should allow to access registry settings for external processes, such as Veeam. This is required to verify domain controllers in the SureBackup job.

What exclusion list? On the backup job, or somewhere in windows?

[idroesse]

dcs115,
Exclusion list is for symantec endpoint protection. there is an option to to put the processes under exclusion list.

@Vitaliy S
how to allow to access registry settings for external processes? i didnt understand that? how to do that?

[Gostev]

I am afraid Vitaly knows about Symantec EndPoint Protection configuration as much as I do = nothing

[dcs115]

@idroesse when a dc boots (I am not sure about non-dc's) Veeam changes some registry settings to make it more compatible with Surebackup (this is my understanding).

I would think these changes would be made on behalf of the system, not as a separate user account, so "allowing external process to make registry changes" probably isnt even an option. With Windows, everything is done by a user account or on behalf of a user account, nothing is done externally per se. I think your install Endpoint contributed to your problem, due to the fact that it is trying to do so much to the system on boot. I believe when we talked before, vmtools was not starting? Try reinstalling endpoint, taking your backup, running a SureBackup job, logging into your booted SureBackup DC, manually starting vmtools from the services console and wait. I suspect it will work.

[idroesse]

@dcs115,
Its working now. i put the vmware folder in the exclusion list (C:\Program Files\VMware) for type security risk scans. also i included in the exclusion list (C:\Program Files\VMware\VMware Tools\vmtoolsd.exe) for All Scan . i tried just vmtoolsd.exe file for type security risk scans.but it failed. but i dont think its a nice option for vmware folder to put in exclusion list. any way thats the only way. its working fine. now i need to put antivirus for exchange server as well and do the same as mentioned.