The object has already been deleted

[VladV]

I have opened a case (Case # 00205914) for the following issue:

When failing over to a replica which was created using vCenter as a target I get this error: Power On virtual machine xxx_replica The object has already been deleted or has not been completely created
If I create a replica of the same VM but having the the host as a target (not going through vCenter) everything seems to be fine.

I have sent the logs and the sql DB from Veeam B&R server and the conclusion was:

it seems there is a conflict during failover involving the hostID of the vCenter inside the VBR Database which leads to a temporary duplicate and this error, therefore Veeam does not know where to send the poweron command. I suppose this is why we recommend using standalone hosts for anything related to virtual vCenter appliances - in that case we rely on fixed hostID of the target and source hosts.

Seeing that you did the replication over standalone hosts already i would close the case, if you have no further questions.

I understand that this is the best practice to replicate because there are high chances that there will be no vCenter when you need to failover but I have two questions which I raised also to support:

1. In the future we are looking to install a second linked vCenter at a remote site and replicate our VMs there. Will there be a similar issue then?
2. Is there a solution in the following case: I replicated some VMs having vCenter as a target and after a while my source went offline and I need to failover to those replicas. In this case I would get this error and I cannot move on. Being that this is not mentioned as a known issue or a warning anywhere, what is there to do to keep my business going and be able to use the functionality of VEEAM B&R. (poorly written on my behalf, what I meant to say is: given that this is not expected to happen - haven't been able to find any known issues or warnings for this situation - what are my options to power on the replicas and then failback/permanent failover during off hours)

[veremin]

In fact, there is a variety of our customers who successfully replicate from/to the ESX(i) that have been added through vCenter. So, there are no well-known issues related to it and support might be not 100% right in this particular case.

According to the previous experience, you can get the abovementioned error after ESX(i) host update, etc. Thus, I’m wondering whether recently there has been something like this.

As to your questions:

1. There shouldn’t be any issues.
2. Due to the way Veeam is storing replication restore points you will still be able to power on the corresponding VMs manually. However, it should be considered as a last solution.

Hope this helps.
Thanks.

[VladV]

1. There shouldn’t be any issues.
2. Due to the way Veeam is storing replication restore points you will still be able to power on the corresponding VMs manually. However, it should be considered as a last solution.

Thank you for your insight Vladimir.

According to the previous experience, you can get the abovementioned error after ESX(i) host update, etc. Thus, I’m wondering whether recently there has been something like this.

I believe I have installed Veeam right before updating our infrastructure to vSphere 5.1 U1 from 5.1. I then ran some backup tests of only one VM then proceeded to update vSphere. After updating I started adding the rest of the VMs.

I also tried to remove the test VM from inventory add it back again and then run a new replication job, but it gave the same error.

I will try to solve this with support to see what needs to be done.

Regards,
Vlad

[foggy]

You may want to check Veeam configuration database for duplicate host entries as it is used to be the case in other occurrences of this error. If this is the case, our support can assist you in removing them.

[VladV]

This is what I have in dbo.hosts:



Haven't found a way to properly format it for the forum.

So I have 3 esxi which are added through vCenter and esx3 is also added as a standalone host. Can't find any duplicates though.

Regards,
Vlad

[foggy]

Vlad, according to our QC, there are a few occasions when the wrong host record is picked up from the database in some particular cases that will be addressed in v7. So I suggest continue investigating with support to verify whether you experience on of the known issues.

Btw, which Veeam B&R version are you using?

[VladV]

Lastest version: 6.5 patch 3 - 6.5.0.144

[foggy]

Ok, thanks. It would be much appreciated if you could update this thread on the results of support investigation.

[VladV]

Sure, absolutely. If I could only get replies a little bit faster (still on trial though) .

Regards,
Vlad

[foggy]

At least you get replies, even being on a trial.

[VladV]

First of all please disregard the first conclusion. Because of a misunderstanding it was believed that I was trying to replicate vCenter through vCenter. We have clarified this through a webex session.

Yesterday we had another meeting with engineering to see what was going on and after some time of running tests and going through the DB the conclusion was (I asked him kindly to summarize the issue so I can share it on the forum):

"Same physical / bios ID's for the host added as standalone one and for the host added under the vCneter server resulted veeam to query the stand alone host to power on the VM with the VM reference of the vCenter server." He also mentioned that this is a known issue but wasn't sure when it scheduled to be fixed.

He then made some changes in veeam DB and I was able to successfully power on the machine that was replicated on the host through vCenter.

After the support session ended I continued testing and It seems that now the replicas that targeted the standalone hosts won't boot and show the following error:

5/23/2013 9:04:35 PM Starting failover to restore point '5/23/2013 8:49:06 PM'. Original VM: 'Apache'
5/23/2013 9:05:13 PM Fail Failed to perform failover to replica Error: A specified parameter was not correct.

I have tried to rescan the infrastructure, tried to delete the replica and job and recreate it from scratch, but they still won't start.

The only thing that worked was removing the standalone host and re-adding it back to infrastructure. Unfortunately this broke the replicas that were targeting vcenter. So we are back to square one.

[foggy]

Vlad, thanks for the update. Seems that DB cleanup caused the issue with standalone hosts. I hope our support will be able to perform the cleanup once more and in a more proper way so that you get both replications in a good state.

[VladV]

This was the latest response from support:

"You will be able to use a standalone host as a target IF it is not a part of the vCenter, in case it is a part of the vCenter server you should not add that host as a standalone one to veeam, it was confirmed that this behavior is a bug and is scheduled to be fixed in 7.0 The current workaround is to remove standalone hosts from the veeam console."

Currently this doesn't affect me because I don't have a second vCenter so I use the standalone host as a target. Only in the future, when we will have a linked vCenter at the DR site we will want to use vCenter as a target to get to the host and not directly. Until then I hope v7 will fix this issue.

[foggy]

Thanks for the update! Glad you are not affected by this issue currently.

[bskatz]

Hi
Right now I'm testing laboratory version 6.5.0.144 Veeam before installation for a client.
With the same result that conversation in this thread.
The object has already been deleted
This is a fresh installation on veem on server 2012, fresh installation vcenter apliance and two existing esxi version 5.1 and version 5.1 U1.
The problem occurs when processing order "FileOver now" into the veem console.
I assume there is no solution and to wait until the next version ?
Thanks

Bernardo Katz

[veremin]

Hi, Bernardo,

As stated above, under certain circumstances you might get the corresponding error when a host has been added as a standalone one and as a part of vCenter.

According to plan, this issue will be addressed in the upcoming release; for now you might want to delete standalone servers and see how it goes.

Anyway, I’d strongly recommend to contact our support team in order to understand whether you experience similar issue or completely different one.

Hope this helps.
Thanks.

[fcambot]

I had the same error message after i upgraded a esxi 5.0 host to 5.1 : jobs on the upgraded host failed with "Task failed error : The object has already been deleted or has not been completely created".
I just used "rescan" on the vcenter server in backup infrastructure and everyhting started as usual.
On another vbr server, i did not forgot to rescan and had no error. Each time i've upgraded my hosts (from 4.0 to 4.1, from 4.1 to 5,etc), a simple rescan helped me avoid troubles.

[foggy]

Florian, thanks for sharing your observations, much appreciated.

[Tommaso76]

Hi, I have this problem with the Veeam B&R v8, after upgrade the replication job has stopped working.
I have 2 hosts standalone with ESXi 5.5 despite all your solutions I keep getting this error
Thank's

[Vitaliy S.]

If hosts rescan or re-adding these VMs to the job (and then mapping to existing VM replicas) doesn't help, then please contact our technical team for further assistance.