Exchange U-AIR Labs showing pending status

[gopher_49]

I tried to restore via the Exchange U-AIR wizard and my lab status gets stuck on the 'pending' status... I let it sit for about 30 mins and no luck... Any suggesetions? I'm running Exchange 2003 32 bit and ESXi v4 hosts...

[Gostev]

Did you approve the request for this lab in the Enterprise Manager?
Refer to U-AIR Workflow in sticky FAQ topic for more information.

[gopher_49]

ok.. I see how it flows now.. Here's the problem I'm having. I'm getting 1 error and a few warnings on the Sure Backup session. I'm assuming the below error is the one making the Sure Backup session fail. Is there a timeout setting I can adjust? Also, do I have to have my mounted (isolated) backend Exchange server mounted with a DC also at the same time? Or, can you extract the emails with just the backend server mounted in the virtual lab? Keep in mind. My virtual lab had only 1 VM in it. That being the Exchange backend server. I also have the production and isolated network being the same. This being a network in VLAN ID5 to void it conflicting with my production network.

Error when first creating Sure Backup
VMware Tools did not start in a timely manner

Errors found in log when trying to process Exchange application restore.
2010-10-27T23:36:26.2535672-05:00 Info: Getting virtual lab configuration
2010-10-27T23:36:56.8783712-05:00 Info: Starting virtual lab routing engine (Failed to add one or more routes)
2010-10-27T23:37:31.0656524-05:00 Info: backend1: Publishing
2010-10-27T23:37:34.8937529-05:00 Info: backend1: Updating configuration
2010-10-27T23:37:43.5968222-05:00 Info: backend1: Registering
2010-10-27T23:37:47.0030504-05:00 Info: backend1: Configuring networks (No networks)
2010-10-27T23:37:53.1905108-05:00 Info: backend1: Powering on (no VM networks)
2010-10-27T23:45:36.5469203-05:00 Info: backend1: Heartbeat test
2010-10-27T23:45:36.6094199-05:00 Info: backend1: Pinging (skipped: no VM networks)
2010-10-27T23:47:37.0617740-05:00 Info: backend1: Application initialization

[Alexey D.]

Hello Chris,

Also, do I have to have my mounted (isolated) backend Exchange server mounted with a DC also at the same time?

Yes, you need to have DC acting as Global Catalog, to handle credentials to restore Exchange server's items. Feel free to include DC in application group and ensure that DC starts prior to Exchange server.

VMware Tools did not start in a timely manner

Do you have VMware tools installed on this VM? Also please check VM startup timeout (can be found in application group's settings).

Configuring networks (No networks)

It looks like your virtual lab is not configured correctly.

[gopher_49]

I don't have a DC that's a VM yet. I'll add a DC and set my backend servers to poing to the new VM DC as it's DNS servers. In regards to my virtual lab... I have the production network and the isolated network as the same. This being a network that has a different VLAN ID. I did this to void conflicts of duplicate servers. I had to do this for when I tried an instant restore it placed my VM on my production network... Does the Sure Backup session place the VM's into the isolated network by default? I'm trying to void problems of duplicate servers on the same LAN.

[Gostev]

Sure it does put all VMs in the isolated network... check out sticky FAQ topic on how we setup virtual lab for you to ensure SureBackup/U-AIR/Sandbox functionality does not affect your production. Instant VM recovery does not use virtual lab, because it is designed for recovering VM into production environment.

[gopher_49]

ahh ok.. Sounds good.. I'll tweak out my virtual labs. I'll update this ticket as I get further into.

Thanks.

[gopher_49]

still no luck.. Now it hangs on the Sure Backup step 'starting virtual lab routing engine'. It eventually fails. I emailed tech support my logs and it seems that the virtual lab VM can't access itself and it's trying to access itself on port 443?! Not sure.. But stuck at this point.

[Gostev]

Before you can do U-AIR, you need to make sure your virtual lab is setup correctly by testing it with SureBackup job, which should finish correctly. Failing routing engine is usually a sign of virtual lab not being setup correctly. Let's see what our support has to say. Thanks!

[gopher_49]

I got my virtual labs and Sure Backups to run.. I couldn't get the proxy appliance to work.. So, I disabled that feature and solely rely on VLANs to segregate the networks... I have my DC and a backend server up and running in my Virtual Lab.. All services are started and db's mounted. I now started a U-Air Exchange restore session.. I approved it... It sees my Virtual Lab as 'Ready'. It now still sits on 'Preparing'... Now, I know I can launch my Virtual Labs and/or Sure Backup session by starting a U-Air recovery lab, however, I didn't think it mattered which order I did it in.. I started my Virtual Lab first and then started the U-Air recovery... Anyway... My virtual lab works perfectly fine.. I just can't get off the 'preparing' status in the U-Air Exchange recovery. I let it sit for about 20 mins...

[Gostev]

OK, at first - I do not know if you read FAQ, but you are missing a couple of very important points:
- Proxy appliance is essential because it provides SureBackup and U-AIR logic with access into isolated network, which is a must have for all advanced functionality.
- Whole request system is optional, you can just right-click VM in running SureBackup session and start Exchange restore (but again, this will not work with you current setup without proxy appliance).

Both full SureBackup functionality (test scripts), and U-AIR requires that you are able to ping your Exchange server located in isolated network by masquearade IP address. Until you can make this work, there is no point to do anything else.

I suggest that you setup a webex with our support and have them help you set everything up. You just need to do it once, as soon as your virtual lab is properly setup, you do not have to come back to this again.

[gopher_49]

I've had a WebEx session and no one can figure out why my proxy appliance won't respond.. The log files show requests being denied on port 443... I've read the FAQs but decided to move on and try different things for I've hit a wall.. SureBackup is testing my applications (DNS, Global Catalog, SMTP) and showing them to be successful.. I find that odd for my backup server is on VLAN ID 0 and my Virtual Lab is VLAN ID 05. It makes sense that the proxy appliance needs to be functioning for there needs to be a way for my backup server to access the isolated network...

I'm able to at least power on my DC, Exchange 2003 backend server, and a XP client and truly access my Exchange emails and see the backups work in reality which is truly amazing.. Not sure to why it's showing my applications as testing successful for there's no way my backup server can see these VM's in the isolated network..

I'll continue to work with the tech on getting my proxy appliance working.

Also,

Due to the VM's being powered on in non-persitent mode my DC boots into safe mode and then reboots itself. Once rebooted it works just fine. Also, my Exchange backend server does the same things except it boots in to normal mode. Once the info store service starts it reboots itself and from them on it works just fine. In my environment I've seen the non-persistent disk setting do this exact behaivor in the past.. It's not a big deal for once the VM reboots it works perfectly fine.. But.. Non-persistent mode does really odd things to DC's and Exchange 2003 backend servers.. I know for in my environment if I backup my VM via virtual appliance mode it crashes the VM, reboots it, and then boots the VM disks in non-persistent mode.. That's the reason I know how exactly non-persistent mode works in my environment.. No biggy.. I simply plan to adjust my timeouts to wait for the reboots before testing the applications.. Now I just need to get a script that truly tests the info store service..

Thanks

[Gostev]

Due to the VM's being powered on in non-persitent mode my DC boots into safe mode and then reboots itself. Once rebooted it works just fine. Also, my Exchange backend server does the same things except it boots in to normal mode. Once the info store service starts it reboots itself and from them on it works just fine. In my environment I've seen the non-persistent disk setting do this exact behaivor in the past.. It's not a big deal for once the VM reboots it works perfectly fine...

This is actually expected, and is a part of application-aware processing feature in Veeam Backup, and this will happen during any restore (including real, production restore). What you see is a proper, application specific restore procedures working. DC has to boot in safe mode with Active Directory not running, only then the VSS restore is performed, then DC rebooted and boots up normally. Exchange must be started with stores dismounted before the VSS restore is performed, then after restore it is also rebooted and boots up normally.

You will not see the same behavior if you disable application-aware processing feature in job settings (which is of course not recommended). But they will be booting straight without any additional operations. However, this will kill your AD (due to the USN rollback issue on restored DC), and Exchange may not behave correctly either due to not being restored using the procedure that Microsoft requires.

Anyway... what is your support case number? I will check on the status.

Please keep posted on test script you will come up with... I was thinking eseutil or isinteg (or even combination of both?) may be good, but I am not an Exchange admin so may be there are better options. We are also planning to add ability to launch test scripts locally on SureBackup VMs in future releases (not that you cannot do this right now with psexec, but this should make the setup more convenient).

[gopher_49]

My support ticket number is [ID#537098]. The tech has escalated it to the developers and has been very prompt on working on this and has done a great job. We're both at a point where we need higher level support.

Thanks.

[gopher_49]

FYI: I received a hot fix for this and everything works perfect. It took me awhile to get all of my timeout settings configured... It works incredibly well.

[Gostev]

Glad you have it sorted out and everything works well. Thank you for your help with nailing down this bug!