Discussions specific to the VMware vSphere hypervisor
dsimpkins
Influencer
Posts: 21
Liked: 1 time
Joined: Jun 12, 2012 3:12 am
Full Name: Daniel
Contact:

Re: SureBackup Powering on Taking Forever

Post by dsimpkins » Jul 04, 2012 2:04 am

i've had the same problem (i have had every problem mentioned that is regarding surebackup, im not convinced Veeam is any good...) but i found that opening a console in vSphere client and logging in forced veeam to recognise it was booted.

Vitaliy S.
Product Manager
Posts: 23423
Liked: 1665 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: SureBackup Powering on Taking Forever

Post by Vitaliy S. » Jul 04, 2012 9:01 am

I believe logging in to the VM console has forced VMware Tools to update their status, which led to successful power state in the SB job.

jeremyh8
Enthusiast
Posts: 81
Liked: 11 times
Joined: Jun 17, 2012 1:28 am
Full Name: Jeremy Harrison
Contact:

Re: SureBackup Powering on Taking Forever

Post by jeremyh8 » Aug 17, 2012 1:22 am

someone mentioned unsupported disks for surebackup jobs. Are dynamic disks supported in surebackup jobs? If dynamic disks are supported then are dynamic disks that are spanned supported? 4 dynamic disks presented as 1 drive letter. I am aware of the instant level restore options but am HOPING that this does not carry over to surebackup jobs. Please advise

tsightler
VP, Product Management
Posts: 5541
Liked: 2359 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: SureBackup Powering on Taking Forever

Post by tsightler » Aug 17, 2012 1:36 am

Dynamic disk, even if spanned, should work fine in Surebackup. This is one of the workarounds for performing restores from VMs with large, spanned dynamic disks.

jeremyh8
Enthusiast
Posts: 81
Liked: 11 times
Joined: Jun 17, 2012 1:28 am
Full Name: Jeremy Harrison
Contact:

Re: SureBackup Powering on Taking Forever

Post by jeremyh8 » Aug 17, 2012 1:49 am

awesome ty sir. i am having a hard time getting these things to boot so i disabled vmtools heartbeat and i think things are coming together. I am very very happy to hear that this is going to work! i was getting worried that i had to tell my boss that i couldnt build an exchange lab!

jeremyh8
Enthusiast
Posts: 81
Liked: 11 times
Joined: Jun 17, 2012 1:28 am
Full Name: Jeremy Harrison
Contact:

Re: SureBackup Powering on Taking Forever

Post by jeremyh8 » Aug 18, 2012 1:18 am

I have been fighting getting my surebackup lab to work. I am trying to boot 3 dcs and exchange and 1 xp client. They all take forever to boot 1800-5500 seconds. They are pingable pretty quick and i can rdp into the machines but veeam still says they are not booting up. I have turned off vmware heartbeat and the ping test. They still fail even tho they are actually turned on and operational. Of course when this happens they all revert and are gone so i have to start again. It takes over an hour just to get the first 2 machines up then i am back at 0. I have tried this from 2 different B&R servers. Please advise.

tsightler
VP, Product Management
Posts: 5541
Liked: 2359 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: SureBackup Powering on Taking Forever

Post by tsightler » Aug 18, 2012 1:57 am

Sounds like networking isn't setup properly but it's important to note that this is not an official support forum. I'd suggest contacting support and having them help you.

jeremyh8
Enthusiast
Posts: 81
Liked: 11 times
Joined: Jun 17, 2012 1:28 am
Full Name: Jeremy Harrison
Contact:

Re: SureBackup Powering on Taking Forever

Post by jeremyh8 » Aug 18, 2012 1:59 am

i will open a support ticket. While i am waiting on support... do you know a way to test the networking?

tsightler
VP, Product Management
Posts: 5541
Liked: 2359 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: SureBackup Powering on Taking Forever

Post by tsightler » Aug 18, 2012 2:19 am

So you said you could RDP to the servers, how did you do this?

To test the networking you should see if you can ping the servers NAT addresses from the Veeam server itself. For example, if your servers live on a subnet such as 10.10.1.0/255.255.255.0, and you configured your vLab to use a mask address like 10.10.255.0/255.255.255.0 then to test the networking you would do something like this. If your DC is normally IP address 10.10.1.5, then, once you feel that the DC is booted and should be ready, attempt to ping the NAT address from the Veeam server which would be 10.10.255.5.

I'm curious, why are you booting so many DC's? Those are usually the servers that take the longest. Also, do you have a DNS server in your vLab, and specifically a DNS server that the other servers are configured to use?

jeremyh8
Enthusiast
Posts: 81
Liked: 11 times
Joined: Jun 17, 2012 1:28 am
Full Name: Jeremy Harrison
Contact:

Re: SureBackup Powering on Taking Forever

Post by jeremyh8 » Aug 18, 2012 2:31 am

I am able to ping the servers using the .255 ips from any machine in our network including the b&r server since we setup the needed routes. I am booting 3 dcs just to be doing it and they are used for dns and other servers have them set. The B&R servers i amusing are virtual using the backup repository off of a physical server. I tried moving my proxy to the same host as my virtual lab but after 20 minutes waiting for the first machine to get past configuring network settings i gave up. case # 5211516

tsightler
VP, Product Management
Posts: 5541
Liked: 2359 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: SureBackup Powering on Taking Forever

Post by tsightler » Aug 18, 2012 4:20 pm

When you configured the app group did you actually select the correct test for DCs?

jeremyh8
Enthusiast
Posts: 81
Liked: 11 times
Joined: Jun 17, 2012 1:28 am
Full Name: Jeremy Harrison
Contact:

Re: SureBackup Powering on Taking Forever

Post by jeremyh8 » Aug 19, 2012 12:09 am

i did select the correct tests but then turned them all of so that there would be less tests to fail (we arent trying to test backups we are trying to get a lab up for an exchange patch). I finally got it working by installing veeam B&R on my physcial proxy and creating the appplication group and Lab there. It is still very slow but it is working. I am going to see if support can see why it is crawling so badly.

tsightler
VP, Product Management
Posts: 5541
Liked: 2359 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: SureBackup Powering on Taking Forever

Post by tsightler » Aug 19, 2012 2:13 am 1 person likes this post

Turning on the test for a DC is actually an important step, it will actually make the DC boot faster because we will automatically reconfigured the DC to start replication services without waiting to talk to it's replication partners, which can take 15 minutes or longer (probably why your sitting at the "Configuring Network" step for so long). You will see an extra step in the log that says "Reconfiguring DC" before the DC is powered on in the vLab. You should at least select this one test so that you get this extra piece of "magic". Also, selecting the test automatically increases the timeout appropriately for a DC.

jeremyh8
Enthusiast
Posts: 81
Liked: 11 times
Joined: Jun 17, 2012 1:28 am
Full Name: Jeremy Harrison
Contact:

Re: SureBackup Powering on Taking Forever

Post by jeremyh8 » Aug 19, 2012 4:29 am

I took your advice and enabled the dc tests and i do see where it says configure DC when it starts bringing them up. It still sits for 15-25 minutes before hitting ctrl-alt-del and is very slow. I will see what support says next week. I really appreciate the info and helpful information.

lwarner
Influencer
Posts: 23
Liked: 2 times
Joined: May 29, 2011 6:05 pm
Full Name: Leigh Warner
Contact:

Re: SureBackup Powering on Taking Forever

Post by lwarner » Aug 27, 2012 11:12 pm 1 person likes this post

In my case, I discovered that my Anti-Virus on the guest the was cause. I had the exact same problems as stated in this thread. I completely removed AV on the all guests in my SureBackup jobs and Application Groups, and my DC startup times went from ~40 minutes to under 10 minutes. Member servers went from ~30 minutes to 5 minutes (give or take). Also noticed that I don't get the "Not Running (Not Installed)" VMWare Tools status in vCenter anymore. Works like a charm now.

But of course there's still the issue of not having AV on the servers, so I have to work backward from here. I think that the AV was somehow causing the VMTools to hang, which in turn caused the network to hang. I did try excluding the C:\Program Files\VMWare folder from scanning, but that didn't help, neither did setting the AV to 'delayed start'.

lwarner
Influencer
Posts: 23
Liked: 2 times
Joined: May 29, 2011 6:05 pm
Full Name: Leigh Warner
Contact:

Re: SureBackup Powering on Taking Forever

Post by lwarner » Aug 28, 2012 10:10 pm

I also discovered something odd: If you check the box "Keep application group running once the job completes" the DC to take forever to boot (30 + minutes). Unchecking it makes it start quickly. I don't think this is right, so I might contact support about this after I verify that I set everything else correctly.

jeremyh8
Enthusiast
Posts: 81
Liked: 11 times
Joined: Jun 17, 2012 1:28 am
Full Name: Jeremy Harrison
Contact:

Re: SureBackup Powering on Taking Forever

Post by jeremyh8 » Aug 29, 2012 1:12 am

i already had symantec set to exclude the vmtools service/process and file location. I have performed an instant vmware recovery and removed antivirus... I have seen a huge speed increase on boot and login. I am going to test again using 2 machines with instant recovery and see what happens. After that i will try it in a lab. I will let you know the results.

jeremyh8
Enthusiast
Posts: 81
Liked: 11 times
Joined: Jun 17, 2012 1:28 am
Full Name: Jeremy Harrison
Contact:

Re: SureBackup Powering on Taking Forever

Post by jeremyh8 » Aug 29, 2012 3:09 am

uninstalling symantec has improved boot up and performance of instant recovery and vlabs by 300-400% in my current tests. anyone have an ideas on how to resolve this? uninstalling isnt a very good option :)

stuartmacgreen
Expert
Posts: 111
Liked: 22 times
Joined: May 01, 2012 11:56 am
Full Name: Stuart Green
Contact:

Re: SureBackup Powering on Taking Forever

Post by stuartmacgreen » Sep 17, 2012 11:10 am

I had a surebackup job that was taking beyond the 600secs I had specified for the boot time.
While powering on the VM I noticed the following:
1. Windows Recovery screen with the 30 secs countdown.
2. Server then displayed "Applying update operation xxx of 41,540" (this took 5 minutes)
- What is this doing.? Why I ask is that patching on this server happened back in July. And the server was rebooted then. Windows Update on the server is set Not to download updates. So it wasnt as if these were pending patches. I also checked the running prod VM had no Pending File Actions.
3. The server completed the update operations and then powered down and started back up again. Even more time..

While this is not a Veeam issue it is for awareness that timings set in the Application Group for a VM might need to be further increased when you get situations like this.
Further when you hit this with more than 1 VM in the App Group and this is the first VM the rest of the job fails. We have resorted to moving VM's into a 1 VM per App Group. As we also seem to suffer from slow VMware Tools service starting. No rhyme or reason or pattern.

I understand I can increase the boot time set for the VM, its more learning and awareness.

These are not DC or Exchange servers.
Backup Infrastructure: Veeam B&R 9.5 U4 • vSphere 6.5 u2 • HPE MSA 2040 [FC] (Backup Target) • HPE MSL6480 LTO-6 (Tape Storage)
(Veeam Agent for Windows protecting Veeam Proxy / Tape / Repository servers)

hasnuman
Novice
Posts: 4
Liked: 2 times
Joined: Nov 30, 2012 8:00 pm
Full Name: Hasan Numanoglu
Contact:

Re: SureBackup Powering on Taking Forever

Post by hasnuman » Dec 05, 2012 8:16 am 2 people like this post

Hi,

I had the same problem I solved it in two steps;

First VM heartbeat was failing, I solved it via excluding VmWare folder from Symantec Endpoint Security. But ping attempts and scripts were still failing, I changed NIC of VmGuest from E1000 to VMXNET3. The guest was 2008 R2, on Wİndows 7 guests I did not experienced that kind of problem. Currently Surebackup jobs are working fine...

Greetings,

Hasan

aeccles
Enthusiast
Posts: 80
Liked: 6 times
Joined: May 01, 2012 3:00 pm
Contact:

Re: SureBackup Powering on Taking Forever

Post by aeccles » Oct 04, 2013 12:55 pm

hasnuman wrote:Hi,

I had the same problem I solved it in two steps;

First VM heartbeat was failing, I solved it via excluding VmWare folder from Symantec Endpoint Security. But ping attempts and scripts were still failing, I changed NIC of VmGuest from E1000 to VMXNET3. The guest was 2008 R2, on Wİndows 7 guests I did not experienced that kind of problem. Currently Surebackup jobs are working fine...

Greetings,

Hasan
Sorry for resurrecting an old thread, but I have a Symantec related question that I figured you had probably come across given the above post.
I recently upgraded Symantec Endpoint 11 to Endpoing 12.1.3 and since then, Veeam backups of Exchange extremely slow - processing rate of 5MB/s.
Any chance you ran into anything similar? Did you have make any exclusions for Veeam within Endpoint?

foggy
Veeam Software
Posts: 18673
Liked: 1631 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: SureBackup Powering on Taking Forever

Post by foggy » Oct 04, 2013 2:36 pm

Is SEP installed in the Exchange VM? What the bottleneck stats for the job and how read and transferred data counters have changed since the SEP upgrade?

aeccles
Enthusiast
Posts: 80
Liked: 6 times
Joined: May 01, 2012 3:00 pm
Contact:

Re: SureBackup Powering on Taking Forever

Post by aeccles » Oct 07, 2013 2:53 pm 1 person likes this post

This ended up being a network issue that was unrelated to Symantec and Veeam.

foggy
Veeam Software
Posts: 18673
Liked: 1631 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: SureBackup Powering on Taking Forever

Post by foggy » Oct 07, 2013 3:38 pm

Thanks for taking your time to update the thread, much appreciated!

cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

[MERGED] What's happening during "Powering on" stage of Sure

Post by cparker4486 » Dec 11, 2013 10:42 pm

Hello,

I've created a SureBackup job to use as a test bed for an upgrade of a 3rd party component for Dynamics GP. I've got three VMs in the Application Group (DC, SQL, and Remote Desktop Services). I am able to login to the machines well before SureBackup recognizes it can move on from the Powering On stage. I have turned off all "role" tests for the machines.

I've had to increasingly give higher and higher timeouts for each VM and run the job almost 6 times before I got to a high enough amount to get past Powering On of the second machine (it's wildly different on each run; last time was just 14 minutes). At this point I have a 60 min timeout for each VM and, although I'm logged into the final machine, and it's working well, it has been waiting at Powering On for 50 minutes now.

I understand that performance is going to be significantly slower but I'm pretty sure it's powered on now. :)
-- Chris

foggy
Veeam Software
Posts: 18673
Liked: 1631 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: SureBackup Powering on Taking Forever

Post by foggy » Dec 12, 2013 5:48 am 1 person likes this post

Chris, please review this thread for some hints and feel free to open a case if nothing mentioned here really helps.

cparker4486
Expert
Posts: 231
Liked: 18 times
Joined: Dec 07, 2009 5:09 pm
Full Name: Chris
Contact:

Re: SureBackup Powering on Taking Forever

Post by cparker4486 » Dec 12, 2013 5:41 pm

Looks like I am experiencing the same issue(s) discussed here. Mostly that VMtools is not coming online reliably. Thanks.
-- Chris

Post Reply

Who is online

Users browsing this forum: Slater and 39 guests