Replication errors on finalizing

[donikatz]

Only been using VBU for backup, but now looking to setup replication for the first time. Looks like it's working up until registering the replicated VM. I receive the following error:

"Finalizing target session Child vm for object <datacentername> (datacenter-2) was not found"

Note I've always often received similar errors on restoring backups at the registration stage, so I manually import the vmx. I'd assumed that was normal, but now I'm thinking it's not. And I'm assuming it's related.

Any ideas? Thanks

[Gostev]

Hello, no this error is not "normal" to see for either backup or replication. I am guessing may be the account you have used to add vCenter/ESX does not have sufficient privileges, hard to say more without full logs. Please contact our technical support directly for assistance with troubleshooting this issue.

[donikatz]

Thanks Anton, case opened.

[Gostev]

Great, it would be interesting to see what is causing this. Generally, no Veeam Backup operation (backup, restore, replication, failover) should require manual steps - everything is 100% automated.

[Gostev]

Wait a second, are you on 4.1 yet?

I have totally missed that this error was a known issue for 4.0, it is documented under Resolved Issues in the release notes for 4.1 (link). Funny since I was the one who made this document... totally forgot, what a fail

[Gostev]

OK, someone already tweeted me that you are on 4.1 - then sounds like the bug was fixed with restores, but not replication... if this appears to be so, good timing - we have an update coming soon and can include this fix as well. But let us investigate your logs first.

[donikatz]

Yep, running 4.1. Nice try, though.

[donikatz]

Also, I'm still having the issue in 4.1 with restores AND replication, so either my prob is a bit diff or there are still some more bugs to work out. Either way, I'll update the thread with the solution once I've resolved it with support. Thanks!

[donikatz]

Veeam developers have confirmed this as a bug. No easy patch -- it will require a significant recode -- so I'll have to wait several weeks for the next release.

Support has not been very forthcoming with details, but what I inferred from our WebEx and from the error in the OP is that the <datacentername> in my vCenter does not match the typical defaults to which Veeam queries, which is not something they had considered in their code. So I gather they will be recoding to account for values other than defaults. They claim to have seen this in only a handful of other installations, but would not speculate as to why. My own inference, however, is that it may be related to our historical upgrade path from VC 1.x onward, because the <datacentername> that appears in the web services object browser is a name that we haven't used since our original VMware project launch 4 years ago.

After some back & forth with trying to pull some suggestions for temporary workarounds out of support (they were reluctant to offer anything), the only possibilities seem to be rebuilding VC from scratch or re-adding hosts manually outside of VC, either one of which would require hours of reconfig on my part for VC and/or Veeam jobs. Or to just wait several weeks (no ETA) for the fix. Ouch.

I'm going to dig into SQL on my own and see if I can just change an entity or two to match the defaults expected by Veeam.

[donikatz]

Success!

Deconstructing what I'd seen Veeam dev examining and then analyzing further myself, it turns out my post above was close, but not quite. It was not the datacenter name that was mismatched, it was the name of the VM root folder (and Host root folder) within the datacenter. As I wrote above, they were named after an old datacenter we used back in VC 1.x and had changed years ago. Normally they should be called "vm" and "host" respectively, but mine were called "New York_vm" and "New York_host" I assume because of how things were done way back in VC 1.x. The problem is Veeam was coded to assume "vm" and not allow for variables, so when attempting to register the VM within the folder structure it could not find "vm" and failed.

To see what I mean:

1) Go to your vCenter web interface (https://<vcentername/ip>) and click on "Browse objects managed by vSphere." You're now in the Managed Object Browser (or MOB), which gives you a more under-the-hood look at the same infrastructure you see through VI Client.
2) Click "content"
3) Click the link value by "rootFolder" (in my case it's "group-d1")
4) Click your datacenter value under "childEntity" (in my case "datacenter-2" but yours will likely have a different number). If you have multiple datacenters like we, you will see multiple choices. You'll know you're in the right datacenter when you scroll down to the name string.
5) Click the link value by "vmFolder" (in my case "group-v4"). This will take you to the root of your VC's VM folders, what you see in VI Client under "VMs and Templates" view.
6) Under the name string, it will usually say "vm" but in my case it said "New York_vm". This is the problem.
("New York_host" vs "host" is in a similar path under host folder structure)

The fix:

Once the problem was clear, the workaround was simple. All I did was change the objects' entity names in the vCenter db (SQL) and everything worked perfectly. VMware doesn't use object names, rather IDs, so nothing else will be affected.

1) Stop the VMware VirtualCenter Server service
2) In your vCenter database, find the table "dbo.VPX_ENTITY" (you might have something other than dbo if your db owner has a different name, such as "vcuser" or "vcadmin" etc)
3) Change the incorrect fields, in my case "New York_vm" and "New York_host", to "vm" and "host" respectively.
4) Start the VMware VirtualCenter Server service
Problem solved!

So why the non-default name, in my case and the cases of a few other customers? Inconclusive, but best guess is it has to do with how objects were referenced back in VC 1.x versus how they are today, and either the upgrade path didn't account for that since VMware doesn't normally reference objects by name anyway, or there was a problem during one of our many upgrades over the years, based on certain shared conditions.

Bonus: It also fixed the problem we were having here: http://www.veeam.com/forums/viewtopic.php?f=2&t=2096 with which Veeam support was unable to help. When creating a new job, VMs & Template view didn't work. Obviously I now know this is another example of Veeam calling the folder root by the wrong name rather than ID, so the SQL change above fixed this problem as well.

Since Veeam support & dev knew the root of the problem, it would have been very easy for them to provide more info and we could have worked together to come up with this easy workaround without all the extra legwork. A bit disappointing I had to try to pull advice out of them and eventually was on my own to find a workaround. Still, problem solved, so I'm a happy camper.

Hope this helps someone else! I've emailed Veeam support with the fix so they can pass it on to other customers and/or make a tech doc out of it. Perhaps they can even save the trouble of a major recode, so can concentrate on more important things for the next version (restore job history, perhaps?)... Thanks

[Gostev]

Doni, thanks for posting such a thorough guide