Failed to build restore location

[tmue]

Hi!

We currently have a problem in setting up our environment with VBR10 and VAO 3.0.
We have the production site in Datacenter1 and a restore site in Datacenter2 (geographically splitted).
Datacenter1:
- production cluster (VMware)
- Veeam VBR server with the primary backups

Datacenter2:
- Veeam VBR server hosting the backup copies (which get transferred with a backup copy job from VBR in DC1)
- Veeam VAO server
- Restore cluster with own storage (VMware).

I have plans configured through grouping with Veeam One and so on. Everything looks fine. I can configure the recovery location just fine (can find the storage, cluster and networks).

My problem now is that every time I make a readiness check it failes with this error for every VM -> "Failed to build restore location. Compute resources were not found or have duplicates in Veeam Backup & Replication." And I just can not find where this comes from. It recognizes the host just fine. And I have no duplicates at first sight (but I'm also not sure what "duplicates" in this case means - VM, Host,??).
Anyone expirienced a similar issue or knows where I should look next?

Case ID is 04836813

Any help is appreciated.

/edit: Screenshots
https://imgur.com/UZqZa9Q
https://imgur.com/7KbaaME

[Alec King]

Hello Thomas,

This error can mean that VAO cannot map all the resources. This can happen if for example one of your Veeam backup servers is connected to vCenter, but the other Veeam backup server is connected directly to ESXi hosts.
It's not possible to create the end-to-end resource mapping in such cases. We require all the components (Veeam B&R, ONE, VAO) to be connected to vCenter only (and using same naming, e.g. all using IP, or all using DNS)

We'll also review your logs from the support case. If in meantime you could check your connection configuration that could be the cause. Thanks!

[tmue]

Hi Alec!
Thank you for your answer.
Connections are fine as I can say.
I have 3 VBR servers - VBR production, VBR recovery location and the VBR on the VAO server itself. The VBR production can only see the production vCenter and its Cluster. The other two can see the production vCenter and also the DR vCenter on the recovery site. VAO and ONE can also see both vCenters. All connections are with IP. Could it be that the VBR host in production site also needs to see the vCenter in recovery site?? I do not want the production VBR to trigger the restores. This should be done with the VBR server in the recovery site (between the sites is only a VPN of 1GBit).

[Alec King]

Hey Thomas,

I'm not clear, do you have two backup servers? (three including the embedded VBR in VAO)
I assumed you had only a repository on the DR site, which was used for Backup Copy Jobs from the production VBR.
If you have a 2nd VBR on the DR site, are you planning to use it to recover from loss of the production VBR?
If this is your intention, you can use the embedded VBR of VAO for this. You will have to manually add the repository where the backup copies are to the embedded VBR. If you have lost production site, rescan this repository from the embedded VBR, it will map the backups, and you can then run your DR plan in VAO.

Alternatively, if your production VBR is not lost, then VAO will use that to orchestrate the restore, and there is no need to rescan repository.You will not need to worry about cross-site traffic, only orchestration commands will travel inter-site, while the restore traffic is all in DR site. You should configure the VAO Recovery Location with the "Prefer Backup Copies" option, then VAO will command VBR to restore only from the backup copy repository in DR.

Edit: Just to confirm, this does mean that the production VBR should be connected to both production vCenter and DR vCenter. VAO should also be connected to both.

Hope that helps, any questions let us know. Thanks!

[tmue]

Hi!

" If you have lost production site, rescan this repository from the embedded VBR, it will map the backups, and you can then run your DR plan in VAO."
This is exactly what I try to do. I will try do change the DR VBR to the embedded of VAO. Maybe that helps. "Prefer Backup Copies" is activated.
So if I switch the repository to the VAO embedded VBR you think that could be better? I can then simply delete the "second" DR VBR server in all configurations and only use the embedded one?
So if I want to also test it that way and so on I have to disconnect the production VBR server because it would be used every time for orchestration as long as its reachable from VAO? Same I think then for Readiness Checks?

[Alec King]

Hello!

Correct, if the only purpose of your 2nd VBR in the DR site was to "replace" the lost production VBR - then VAO can do this with embedded VBR. You can remove that 2nd VBR and simplify your configuration.
The embedded VBR is a fully-functional installation of VBR - however please remember to install your usual VBR license there! By default embedded VBR is not licensed.

And by default, if VAO can reach your production VBR - it will use that for orchestration of restores (and for readiness checks). However with Prefer Backup Copies option, the cross-site traffic will not be a problem.

And if you want to test the lost-production-VBR scenario, then yes - you would have to take that production VBR offline somehow so that VAO cannot reach it. Connect the DR repository to embedded VBR and rescan. Allow VAO to sync the data, and then your plan will work without any modification.

Hope that helps!

[tmue]

Ok, great! This is really helpful, thank you so much!

So, one additional question - if i disconnect the production VBR server then it seems like VeeamOne cannot "map" the backups of the protected VMs and I then have no restore points in VAO (for example again in the Readiness Check). Or do i miss something here? It seems like it cannot "connect" the Backups in the Copy Repository to the VMs in the production cluster.

[Alec King]

You are very welcome

When embedded VBR scans the repository it will take over management of those VM restore points. And it will understand what the source VMs are. The metadata on the repository has all the information required.
Then embedded VBR syncs with VAO and your plan is valid again, as VAO knows where the restore points for your VMs are. And Readiness Check should also work at this point!

[tmue]

Unfortunately this is not working as intended. As I thought - as soon as the production VBR server is offline, VAO cannot find the VM-Backups in its own repository. Also the "VMs" view on the repository in Veeam One is empty. Same behaivour as before with the additional VBR server.
Am I missing some mapping or something? It just cannot "connect/recognize" that the backups in copy repository are also "good for restore".

[Alec King]

So you have rescanned the repository in the console of (embedded) VBR?
After the rescan you should also wait (approx 10 minutes) to ensure that data has sync'd between VBR, VAO.
If it still isn't working as expected, please submit log bundle via support team and we will dive in. Thanks!

[tmue]

After a remote session with support yesterday it seems like VAO cannot work with Backup Copy backups. So as soon as the production backup is offline no restorepoints are found by VAO. Also the original problem is still existing.
My problem is that I cannot make a connection between the customer production VBR and our restore location vCenter (DC1 is cutomer DC, DC2 is our DC. Customers wants to work in our site on our hardware if their DC or the complete company building is damaged). But it seems like there HAS TO BE a connection between production VBR and DR vCenter. The embedded VAO VBR does nothing, even if the production VBR is missing.

[Alec King]

Hi Thomas,

I've just checked this solution in my lab and it works as expected there. Maybe there is one additional step you need to do, it's in bold below. Here is what I did -

- I have production VBR making a backup
- There is backup copy to a remote repo on a Windows server in DR site

To simulate loss of production site, I disconnected the NIC of the production VBR.
As expected the VBR is offline and the VAO readiness checks fail.

Now -
- Attach the remote server+repo to the embedded VBR of VAO. Run a rescan and import as part of this.
- Force removal of the offline production VBR agent from the VAO UI (this will remove duplicate restore point info from the VAO database)
- It is neccessary to wait for internal discovery and sync to complete on the VAO server (I returned after half an hour)
- Readiness Check now works. There is only one warning -[Backup Job] The backup job that created the selected restore point was not found on Veeam Backup & Replication server.
This is true of course, the embedded VBR sees the restore points but it does not contain the job that created them. This does not block recovery anyway.

Now I can run the restore plan, which will restore the VMs to the configured Recovery Location successfully.

It may be that you did not wait long enough for the internal sync to complete? Before this is completed the readiness check will show various errors such as 'backup not found'.

Hope this helps, let us know if you've any questions on the above.
Thanks!

[tmue]

Hi!

Thanks again for all your effort!

Tried this but as before it fails because it can not find restore points for the VMs in the plan. Somekind of "link" is missing here it seems.
Veeam One also does not recognize the VMs in Data Protection View in the remote repository as soon as I link it to another VBR installation and not the one which created it. It gets that this is a Veeam Repo and show me how many full and incremental backups are in it but can not link this to VMs. So the VMs tab in Veeam One is just empty.
You do your backup copy jobs in your lab also with the production VBR? And if yes, it is a simple Backup Copy Job to another server and its network shared path?

[Alec King]

Hi,

I've confirmed that it is essential to remove the agent for the failed production VBR from VAO. Only then will the plan work, so make sure you don't miss this step.

Also, when you refer to Veeam ONE, this is a standalone ONE server? Which you use for monitoring VBR and VMware? Let's leave it out of this scenario, because I am not sure how it would work. It is quite possible you don't see expected data in your ONE - it may show up later when a discovery runs - but anyway VAO does not require remote ONE to work.

In my lab I perform production backup job to local repository. Then I have a remote repository, windows server with local storage (however any kind of standard repository will work, e.g. network share)
I send backups to this repo using backup copy jobs.
Then if I take production VBR offline, I can recover as follows -
- Add the repository to embedded VBR and scan it (in VBR console you will see that backups are found)
- Remove the failed VAO Agent from VAO UI (you will need to use the 'force' option as it is offline)
- Wait for data to sync in VAO. As I said I waited 30 minutes (we can research how to make this faster)
- Soon you will see that Readiness Check works with no errors.
- Then you can run the Restore plan and it will succeed.

[tmue]

You, sir, are my hero!
It worked exactly as you described. Only thing I had to do in advance was to upgrade to VAO 4 and all VBR to v11. Your solution did not work for me with VAO 3 and VBR v10.
I already mentioned in my ticket that I think this is a feature that I would like to have - choose what VBR server should be used for ALL restore tasks.

And - now I have a problem that I cannot re-add the production VBR after I forced the uninstall in VAO (Agent failed communicating with server - but nothing changed in network, firewall and so on).

But we could do the planned restore test with our customer so that part is done. Thank you very much for your help!

[Alec King]

Great to hear that it's working for you now
Any more questions please let us know!