SureBackup Powering on Taking Forever

[dsimpkins]

i've had the same problem (i have had every problem mentioned that is regarding surebackup, im not convinced Veeam is any good...) but i found that opening a console in vSphere client and logging in forced veeam to recognise it was booted.

[Vitaliy S.]

I believe logging in to the VM console has forced VMware Tools to update their status, which led to successful power state in the SB job.

[jeremyh8]

someone mentioned unsupported disks for surebackup jobs. Are dynamic disks supported in surebackup jobs? If dynamic disks are supported then are dynamic disks that are spanned supported? 4 dynamic disks presented as 1 drive letter. I am aware of the instant level restore options but am HOPING that this does not carry over to surebackup jobs. Please advise

[tsightler]

Dynamic disk, even if spanned, should work fine in Surebackup. This is one of the workarounds for performing restores from VMs with large, spanned dynamic disks.

[jeremyh8]

awesome ty sir. i am having a hard time getting these things to boot so i disabled vmtools heartbeat and i think things are coming together. I am very very happy to hear that this is going to work! i was getting worried that i had to tell my boss that i couldnt build an exchange lab!

[jeremyh8]

I have been fighting getting my surebackup lab to work. I am trying to boot 3 dcs and exchange and 1 xp client. They all take forever to boot 1800-5500 seconds. They are pingable pretty quick and i can rdp into the machines but veeam still says they are not booting up. I have turned off vmware heartbeat and the ping test. They still fail even tho they are actually turned on and operational. Of course when this happens they all revert and are gone so i have to start again. It takes over an hour just to get the first 2 machines up then i am back at 0. I have tried this from 2 different B&R servers. Please advise.

[tsightler]

Sounds like networking isn't setup properly but it's important to note that this is not an official support forum. I'd suggest contacting support and having them help you.

[jeremyh8]

i will open a support ticket. While i am waiting on support... do you know a way to test the networking?

[tsightler]

So you said you could RDP to the servers, how did you do this?

To test the networking you should see if you can ping the servers NAT addresses from the Veeam server itself. For example, if your servers live on a subnet such as 10.10.1.0/255.255.255.0, and you configured your vLab to use a mask address like 10.10.255.0/255.255.255.0 then to test the networking you would do something like this. If your DC is normally IP address 10.10.1.5, then, once you feel that the DC is booted and should be ready, attempt to ping the NAT address from the Veeam server which would be 10.10.255.5.

I'm curious, why are you booting so many DC's? Those are usually the servers that take the longest. Also, do you have a DNS server in your vLab, and specifically a DNS server that the other servers are configured to use?

[jeremyh8]

I am able to ping the servers using the .255 ips from any machine in our network including the b&r server since we setup the needed routes. I am booting 3 dcs just to be doing it and they are used for dns and other servers have them set. The B&R servers i amusing are virtual using the backup repository off of a physical server. I tried moving my proxy to the same host as my virtual lab but after 20 minutes waiting for the first machine to get past configuring network settings i gave up. case # 5211516

[tsightler]

When you configured the app group did you actually select the correct test for DCs?

[jeremyh8]

i did select the correct tests but then turned them all of so that there would be less tests to fail (we arent trying to test backups we are trying to get a lab up for an exchange patch). I finally got it working by installing veeam B&R on my physcial proxy and creating the appplication group and Lab there. It is still very slow but it is working. I am going to see if support can see why it is crawling so badly.

[tsightler]

Turning on the test for a DC is actually an important step, it will actually make the DC boot faster because we will automatically reconfigured the DC to start replication services without waiting to talk to it's replication partners, which can take 15 minutes or longer (probably why your sitting at the "Configuring Network" step for so long). You will see an extra step in the log that says "Reconfiguring DC" before the DC is powered on in the vLab. You should at least select this one test so that you get this extra piece of "magic". Also, selecting the test automatically increases the timeout appropriately for a DC.

[jeremyh8]

I took your advice and enabled the dc tests and i do see where it says configure DC when it starts bringing them up. It still sits for 15-25 minutes before hitting ctrl-alt-del and is very slow. I will see what support says next week. I really appreciate the info and helpful information.

[lwarner]

In my case, I discovered that my Anti-Virus on the guest the was cause. I had the exact same problems as stated in this thread. I completely removed AV on the all guests in my SureBackup jobs and Application Groups, and my DC startup times went from ~40 minutes to under 10 minutes. Member servers went from ~30 minutes to 5 minutes (give or take). Also noticed that I don't get the "Not Running (Not Installed)" VMWare Tools status in vCenter anymore. Works like a charm now.

But of course there's still the issue of not having AV on the servers, so I have to work backward from here. I think that the AV was somehow causing the VMTools to hang, which in turn caused the network to hang. I did try excluding the C:\Program Files\VMWare folder from scanning, but that didn't help, neither did setting the AV to 'delayed start'.

[lwarner]

I also discovered something odd: If you check the box "Keep application group running once the job completes" the DC to take forever to boot (30 + minutes). Unchecking it makes it start quickly. I don't think this is right, so I might contact support about this after I verify that I set everything else correctly.

[jeremyh8]

i already had symantec set to exclude the vmtools service/process and file location. I have performed an instant vmware recovery and removed antivirus... I have seen a huge speed increase on boot and login. I am going to test again using 2 machines with instant recovery and see what happens. After that i will try it in a lab. I will let you know the results.

[jeremyh8]

uninstalling symantec has improved boot up and performance of instant recovery and vlabs by 300-400% in my current tests. anyone have an ideas on how to resolve this? uninstalling isnt a very good option

[stuartmacgreen]

I had a surebackup job that was taking beyond the 600secs I had specified for the boot time.
While powering on the VM I noticed the following:
1. Windows Recovery screen with the 30 secs countdown.
2. Server then displayed "Applying update operation xxx of 41,540" (this took 5 minutes)
- What is this doing.? Why I ask is that patching on this server happened back in July. And the server was rebooted then. Windows Update on the server is set Not to download updates. So it wasnt as if these were pending patches. I also checked the running prod VM had no Pending File Actions.
3. The server completed the update operations and then powered down and started back up again. Even more time..

While this is not a Veeam issue it is for awareness that timings set in the Application Group for a VM might need to be further increased when you get situations like this.
Further when you hit this with more than 1 VM in the App Group and this is the first VM the rest of the job fails. We have resorted to moving VM's into a 1 VM per App Group. As we also seem to suffer from slow VMware Tools service starting. No rhyme or reason or pattern.

I understand I can increase the boot time set for the VM, its more learning and awareness.

These are not DC or Exchange servers.

[hasnuman]

Hi,

I had the same problem I solved it in two steps;

First VM heartbeat was failing, I solved it via excluding VmWare folder from Symantec Endpoint Security. But ping attempts and scripts were still failing, I changed NIC of VmGuest from E1000 to VMXNET3. The guest was 2008 R2, on Wİndows 7 guests I did not experienced that kind of problem. Currently Surebackup jobs are working fine...

Greetings,

Hasan

[aeccles]

Hi,

I had the same problem I solved it in two steps;

First VM heartbeat was failing, I solved it via excluding VmWare folder from Symantec Endpoint Security. But ping attempts and scripts were still failing, I changed NIC of VmGuest from E1000 to VMXNET3. The guest was 2008 R2, on Wİndows 7 guests I did not experienced that kind of problem. Currently Surebackup jobs are working fine...

Greetings,

Hasan

Sorry for resurrecting an old thread, but I have a Symantec related question that I figured you had probably come across given the above post.
I recently upgraded Symantec Endpoint 11 to Endpoing 12.1.3 and since then, Veeam backups of Exchange extremely slow - processing rate of 5MB/s.
Any chance you ran into anything similar? Did you have make any exclusions for Veeam within Endpoint?

[foggy]

Is SEP installed in the Exchange VM? What the bottleneck stats for the job and how read and transferred data counters have changed since the SEP upgrade?

[aeccles]

This ended up being a network issue that was unrelated to Symantec and Veeam.

[foggy]

Thanks for taking your time to update the thread, much appreciated!

[cparker4486]

Hello,

I've created a SureBackup job to use as a test bed for an upgrade of a 3rd party component for Dynamics GP. I've got three VMs in the Application Group (DC, SQL, and Remote Desktop Services). I am able to login to the machines well before SureBackup recognizes it can move on from the Powering On stage. I have turned off all "role" tests for the machines.

I've had to increasingly give higher and higher timeouts for each VM and run the job almost 6 times before I got to a high enough amount to get past Powering On of the second machine (it's wildly different on each run; last time was just 14 minutes). At this point I have a 60 min timeout for each VM and, although I'm logged into the final machine, and it's working well, it has been waiting at Powering On for 50 minutes now.

I understand that performance is going to be significantly slower but I'm pretty sure it's powered on now.

[foggy]

Chris, please review this thread for some hints and feel free to open a case if nothing mentioned here really helps.

[cparker4486]

Looks like I am experiencing the same issue(s) discussed here. Mostly that VMtools is not coming online reliably. Thanks.