VAO DR server not seeing my Prod VLab

[perseusco]

Hi all,
I recently installed the 30 day trial of VAO. We have a remote data center that serves as our DR site where we replicate our production VM's.
I installed VAO server on my Prod site and VAO server on my DR site both sites are synced. I have on my Prod site an esxi cluster and another esxi cluster on my DR.
My issue is that the VAO server on the DR side does not see the VLab from production, it only sees the VLab on the DR side, I have setup the tags on both Vsphere servers and I can see the groups on the VAO DR Server just fine, but when I setup a Failover plan for my Prod VM's I don't have a VLab available to test with since its not seen by the VAO DR server.

[Alec King]

Hello,
Your virtual lab(s) should be created and configured only on the DR site. As the DR site is where the replica VMs are, this is also where the Virtual Lab must be.
The requirement is that the Veeam Backup server which is performing the replication is also the server where the lab should be connected. And this Veeam Backup server should be connected to the DR VAO server. Best practice says this Backup server should be on the DR site (so, 'pulling' replicas from Prod into DR)

Even though you create all Virtual Labs in the DR site - you can allocate a lab to Production Site users. Then Production site users can test their plans there. In VAO DR site UI, go to Configuration - Plan Components - Virtual Labs, and choose the Production site, and check the box to Enable the required lab. (note that each virtual lab can only be enabled for one site at a time)

Hope that helps, please let us know here how your VAO trial is going!

[perseusco]

Hi Alec,
I am not seeing the "check the box to Enable the required lab", all I see to the right is the VM Groups pulled from Production due to the tagging, do i need to tag the Prod VLab so it shows here?
My VBR in Production site is the one performing all the jobs, backup, disk backup, Replication, i have a VBR server on my DR site but its not doing anything, perhaps Proxy services, sounds like i dont have it configured right according to best practices. In my mind, i thought that when seting a failover plan after chosing the site, Prod or DR, the testing would happen on the correspondant VLab.
How easy would it be to migrate my VBR in Prod to DR (settings, jobs, etc)

[Alec King]

Hi Perseus,
The Enable Virtual Lab checkbox is on Virtual labs tab, in Configuration / Plan Components. Here is a screenshot
Note - even though the Virtual Labs are all connected on DR servers - you can 'enable' any lab for any production (or DR) site. That lets the users of that site access the lab.

You can migrate a Veeam Backup server using export and then import of the configuration. However you may have to recreate credentials, rescan repositories etc, on the new server.

I should point out, that a VAO Production server is NOT a hard requirement. VAO Production server may be required, e.g. for very slow inter-site links, complex networking, or need for strict multi-site role delegation. However (especially in a trial or POC) you can do everything from just the single VAO DR server. It is much simpler and the principles are the same.

The simplest solution for you might be to uninstall the VAO agent that you deployed on your Prod Backup server from VAO Prod server, and instead deploy a VAO agent from the VAO DR server. The requirement is not that your source Backup server must be IN the DR site - the requirement is that your source Backup server should be CONNECTED to the VAO DR Site server.
This use case doesn't work for all environments, but it can help you run a simple POC. Let us know if you have any questions!

Thanks,
Alec

[perseusco]

Hi Alec,
The only VLab I see is the one on the DR side.
I do have a VAO Agent on my DR. I think I got what I need, tons of questions though. On a similar note, with my current scenario (NO VAO), and working with failover plans, wouldn't it be better to have my VBR Server running my backups, backup copy's, Replication, etc. on my DR site? for the purpose of deploying a full site failover in case everything goes down on the prod side?

thanks

[Alec King]

Hi Perseus,
Yes, that's correct, in general the best practice would be for the VBR server to be on the DR site, so that it is still available in the case of total loss of the production site.
However VAO does have the feature that it will support replica failover even if a source (production site) VBR server is offline - in this case we can use the embedded VBR server in a VAO server for failover control. VAO will rescan the replicas using embedded VBR and can still perform that failover independent of the original VBR.

[perseusco]

Hi Alec,
I am still struggling here, I still cant have a VLab available for my failover. I have created a failover plan, selected my Production site, selected the grups (one VM), steps, the need of a VLab is because after running this fail over test, the replica being brought up is not keeping the re ip settings defined on the replication job, if I run a fail over for the same VM through VBR the re ip settings stay, so I opened a ticket and I was asked for VLab results.
Its intriguing that once I setup my failover and select my Production site, VAO knows where to go for a replica but does not know that there is a VLab on that same site that it can be used for testing?

[Alec King]

Hi Perseus,
OK so to confirm - the remaining problem is that re-IP does not work during the lab test, is that correct?

And you have deployed a VAO agent on a remote VBR server, where this lab is connected?

if you are using a remote VBR server (not embedded VBR server), there is a known issue where re-IP settings are not detected when the agent is installed. The workaround is to open VAO UI, go to Configuration - VAO Agents and run through the Edit wizard for the VAO agent. No need to change any settings, just click Next - Next - Finish through the wizard. It will reinitialize the agent and re-IP will then work in the lab.

BTW if you have opened a support ticket, please post the ticket number here. Thanks!

[perseusco]

Hi Alec, support ticket is 03214613.
Maybe I did not explain right, first of all here is my setup:

Production Site:
esxi cluster
Physical VBR running backup, backup copy and replication (replicating to DR), VLab-Main
Virtual VAO-Production

DR Site:
esxi cluster
Physical VBR (no jobs) VLab-Dr
Virtual VAO-DR

When i run a single VM failover through VBR on my production site, the replica on my DR site powers up just fine and uses the re ip settings defined on the replica job.
On VAO-DR i created a test failover for the same VM, the replica powers up but it keeps the same IP as the production site.

thanks

[perseusco]

This is what I get:

10:38:02 AM 10/2/2018 Info Step 'Process Replica VM' execution started. Plan mode = FailoverNow
10:38:03 AM 10/2/2018 Info 1 execution attempt
10:38:09 AM 10/2/2018 Warning Agent was not found. Performed rescan
10:38:16 AM 10/2/2018 Info Starting failover for VM Win7x64 to the state as of 4 days ago (2:26 PM Thursday 9/27/2018)
10:38:26 AM 10/2/2018 Info Reverting VM to the restore point snapshot
10:38:28 AM 10/2/2018 Warning Unable to use re-IP rules for Win7x64, because REP Win7x64 does not exist
10:38:39 AM 10/2/2018 Info Failover completed successfully
10:38:44 AM 10/2/2018 Info Starting VM...
10:38:55 AM 10/2/2018 Info Powering on VM Win7x64_replica on host VC-DR.JACOBSENCONSTRUCTION.COM
10:41:28 AM 10/2/2018 Info Waiting for VM to boot in 1105 sec
10:42:05 AM 10/2/2018 Info VM was started successfully
Info Step 'Process Replica VM' execution finished

[Alec King]

Hi Perseus,
Thanks for the details!
I believe the important line is here-

10:38:09 AM 10/2/2018 Warning Agent was not found. Performed rescan

This means that VAO is using the DR site VBR server for failover orchestration. We have performed a rescan of replica metadata and allowed the DR site VBR to actually perform failover. This is default behaviour because we assume the production site has some issues.

However even though we have rescanned the environment and located the replicas, we haven't picked up the re-IP rules, they are part of the job configuration on the other (Prod) VBR.

I suggest you remove the VAO agent from your Production VBR server. And then redeploy it, but connect the server to DR Site.
Here's the process -
1. In VAO UI, open Configuration - VAO Agents
2. Select your Production VBR and click Uninstall
3. Click Install
4. Select your DR Site, click next
5. Enter name for your Production VBR server
6. Complete the wizard

You will need to wait for collection and synchronization to happen. Once all the data has been collected then VAO will be able to directly orchestrate your production VBR. Then re-IP should work.

It is possible that your environment doesn't require the production-side VAO server at all. Everything can be accomplished with the VAO DR server only. We can discuss the requirements and architecture, but first I would like to get your failover working so please let us know if above helps!

Thanks,
Alec

[perseusco]

Hi Alec,
Re IP rules work now ! Thanks for that.
However i tried running a Lab test as well and i am getting the following:

3:05:24 PM 10/2/2018 Info Step 'Process Replica VM' execution started. Plan mode = StartTest
3:05:24 PM 10/2/2018 Info 1 execution attempt
3:06:25 PM 10/2/2018 Error Error while parsing node [PciSlotNumber] in [VmReconfigurationAuxData]
3:06:25 PM 10/2/2018 Info Step 'Process Replica VM' execution finished

any ideas?

thanks

Perseus

[Alec King]

Hi Perseus,

OK, great that failover with re-IP is working!

Can you confirm the configuration of VAO Site, VBR, and Virtual Lab now?
Are you using the virtual lab which is connected to the same VBR server that does replication jobs?

And, do you have one vCenter that includes both clusters in both your sites? Or each site has a separate vCenter?
If you have just one shared vCenter, then it's possible you don't need the VAO production server at all. It could help your testing if we simplify your configuration.

We'll investigate that specific error too. Can you create a log export from VAO and add it to your support case? Thanks!

[perseusco]

Hi Alec,
Production Site:
esxi cluster
Physical VBR running backup, backup copy and replication (replicating to DR), VLab-Main
Virtual VAO-Production

DR Site:
esxi cluster
Physical VBR (no jobs) VLab-Dr
Virtual VAO-DR

Each site has its own vcenter and my Lab test was done using the Vlab-Dr and i have it under the VAO-DR in the UI, i tried also switching the lab to the VAO-Production site in the UI but the same results. I also connected the Vlab-dr through VBR on my production site and it shows in VAO-Dr now, however the same result.

[Alec King]

Hi Perseus,

Do you have good network connection between sites? Because if you do, I'd recommend you just remove the VAO Production site. It's only required if you really need to optimise traffic across sites. Production VAO is not technically required for plan creation, failover, testing etc.

Regarding your virtual lab error, it may be that moving the lab between sites and between different VBRs has left some stale data in VAO configuration. It could be that VAO cannot find the lab, after all the moving and resyncing.

To simplify your testing I still recommend you remove the VAO Production Site. Please note if you remove the site, it will remove all plans you created there, and associated reports etc. If you only have one small plan for testing, then it's not so bad I hope!
To remove a site-
Go to VAO DR UI - Configuration - Sites, select your production site and remove it (there will be warnings)

Then you should connect your production vCenter to the VAO DR -
Go to VAO DR UI - Configuration - VMware Servers and add your production vCenter (it will then be directly connected to DR site)

When above is complete, you will be using only the VAO DR server. You can uninstall VAO from the production server, or just shut it down - it is not participating in the system any more.

Finally, it might be best to delete and recreate your virtual lab. This will ensure there is no bad cached config in our VAO database. It's important to create new lab with a new name - don't use the same name.
The lab should be attached to the server which performed replication (this is required for re-IP).

With above complete, you can create failover plans - you will only have one site, so by default we call it "DR Site", however you can create plans there with production VMs because we connected your production vCenter. You can perform lab tests, and failovers, re-IP and all other functionality should work.
Let us know how it goes!

Thanks
Alec