The object has already been deleted

VMware specific discussions

The object has already been deleted

Veeam Logoby VladV » Thu May 16, 2013 6:53 pm

I have opened a case (Case # 00205914) for the following issue:

When failing over to a replica which was created using vCenter as a target I get this error: Power On virtual machine xxx_replica The object has already been deleted or has not been completely created
If I create a replica of the same VM but having the the host as a target (not going through vCenter) everything seems to be fine.

I have sent the logs and the sql DB from Veeam B&R server and the conclusion was:
it seems there is a conflict during failover involving the hostID of the vCenter inside the VBR Database which leads to a temporary duplicate and this error, therefore Veeam does not know where to send the poweron command. I suppose this is why we recommend using standalone hosts for anything related to virtual vCenter appliances - in that case we rely on fixed hostID of the target and source hosts.

Seeing that you did the replication over standalone hosts already i would close the case, if you have no further questions.


I understand that this is the best practice to replicate because there are high chances that there will be no vCenter when you need to failover but I have two questions which I raised also to support:

1. In the future we are looking to install a second linked vCenter at a remote site and replicate our VMs there. Will there be a similar issue then?
2. Is there a solution in the following case: I replicated some VMs having vCenter as a target and after a while my source went offline and I need to failover to those replicas. In this case I would get this error and I cannot move on. Being that this is not mentioned as a known issue or a warning anywhere, what is there to do to keep my business going and be able to use the functionality of VEEAM B&R. (poorly written on my behalf, what I meant to say is: given that this is not expected to happen - haven't been able to find any known issues or warnings for this situation - what are my options to power on the replicas and then failback/permanent failover during off hours)
VladV
Expert
 
Posts: 214
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: The object has already been deleted

Veeam Logoby v.Eremin » Fri May 17, 2013 9:36 am

In fact, there is a variety of our customers who successfully replicate from/to the ESX(i) that have been added through vCenter. So, there are no well-known issues related to it and support might be not 100% right in this particular case.

According to the previous experience, you can get the abovementioned error after ESX(i) host update, etc. Thus, I’m wondering whether recently there has been something like this.

As to your questions:

1.There shouldn’t be any issues.
2.Due to the way Veeam is storing replication restore points you will still be able to power on the corresponding VMs manually. However, it should be considered as a last solution.

Hope this helps.
Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: The object has already been deleted

Veeam Logoby VladV » Fri May 17, 2013 10:13 am

v.Eremin wrote:1.There shouldn’t be any issues.
2.Due to the way Veeam is storing replication restore points you will still be able to power on the corresponding VMs manually. However, it should be considered as a last solution.


Thank you for your insight Vladimir.

v.Eremin wrote:According to the previous experience, you can get the abovementioned error after ESX(i) host update, etc. Thus, I’m wondering whether recently there has been something like this.


I believe I have installed Veeam right before updating our infrastructure to vSphere 5.1 U1 from 5.1. I then ran some backup tests of only one VM then proceeded to update vSphere. After updating I started adding the rest of the VMs.

I also tried to remove the test VM from inventory add it back again and then run a new replication job, but it gave the same error.

I will try to solve this with support to see what needs to be done.

Regards,
Vlad
VladV
Expert
 
Posts: 214
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: The object has already been deleted

Veeam Logoby foggy » Fri May 17, 2013 10:32 am

You may want to check Veeam configuration database for duplicate host entries as it is used to be the case in other occurrences of this error. If this is the case, our support can assist you in removing them.
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: The object has already been deleted

Veeam Logoby VladV » Fri May 17, 2013 11:19 am

This is what I have in dbo.hosts:

Image

Haven't found a way to properly format it for the forum.

So I have 3 esxi which are added through vCenter and esx3 is also added as a standalone host. Can't find any duplicates though.

Regards,
Vlad
VladV
Expert
 
Posts: 214
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: The object has already been deleted

Veeam Logoby foggy » Fri May 17, 2013 12:54 pm

Vlad, according to our QC, there are a few occasions when the wrong host record is picked up from the database in some particular cases that will be addressed in v7. So I suggest continue investigating with support to verify whether you experience on of the known issues.

Btw, which Veeam B&R version are you using?
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: The object has already been deleted

Veeam Logoby VladV » Fri May 17, 2013 1:26 pm

Lastest version: 6.5 patch 3 - 6.5.0.144
VladV
Expert
 
Posts: 214
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: The object has already been deleted

Veeam Logoby foggy » Fri May 17, 2013 1:33 pm

Ok, thanks. It would be much appreciated if you could update this thread on the results of support investigation.
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: The object has already been deleted

Veeam Logoby VladV » Fri May 17, 2013 4:36 pm

Sure, absolutely. If I could only get replies a little bit faster (still on trial though) :) .

Regards,
Vlad
VladV
Expert
 
Posts: 214
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: The object has already been deleted

Veeam Logoby foggy » Mon May 20, 2013 1:36 pm

At least you get replies, even being on a trial. ;)
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: The object has already been deleted

Veeam Logoby VladV » Fri May 24, 2013 7:53 am

First of all please disregard the first conclusion. Because of a misunderstanding it was believed that I was trying to replicate vCenter through vCenter. We have clarified this through a webex session.

Yesterday we had another meeting with engineering to see what was going on and after some time of running tests and going through the DB the conclusion was (I asked him kindly to summarize the issue so I can share it on the forum):

"Same physical / bios ID's for the host added as standalone one and for the host added under the vCneter server resulted veeam to query the stand alone host to power on the VM with the VM reference of the vCenter server." He also mentioned that this is a known issue but wasn't sure when it scheduled to be fixed.

He then made some changes in veeam DB and I was able to successfully power on the machine that was replicated on the host through vCenter.

After the support session ended I continued testing and It seems that now the replicas that targeted the standalone hosts won't boot and show the following error:

5/23/2013 9:04:35 PM Starting failover to restore point '5/23/2013 8:49:06 PM'. Original VM: 'Apache'
5/23/2013 9:05:13 PM Fail Failed to perform failover to replica Error: A specified parameter was not correct.

I have tried to rescan the infrastructure, tried to delete the replica and job and recreate it from scratch, but they still won't start.

The only thing that worked was removing the standalone host and re-adding it back to infrastructure. Unfortunately this broke the replicas that were targeting vcenter. So we are back to square one.
VladV
Expert
 
Posts: 214
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: The object has already been deleted

Veeam Logoby foggy » Fri May 24, 2013 9:13 am

Vlad, thanks for the update. Seems that DB cleanup caused the issue with standalone hosts. I hope our support will be able to perform the cleanup once more and in a more proper way so that you get both replications in a good state.
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: The object has already been deleted

Veeam Logoby VladV » Fri May 24, 2013 10:25 am

This was the latest response from support:

"You will be able to use a standalone host as a target IF it is not a part of the vCenter, in case it is a part of the vCenter server you should not add that host as a standalone one to veeam, it was confirmed that this behavior is a bug and is scheduled to be fixed in 7.0 The current workaround is to remove standalone hosts from the veeam console."

Currently this doesn't affect me because I don't have a second vCenter so I use the standalone host as a target. Only in the future, when we will have a linked vCenter at the DR site we will want to use vCenter as a target to get to the host and not directly. Until then I hope v7 will fix this issue.
VladV
Expert
 
Posts: 214
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: The object has already been deleted

Veeam Logoby foggy » Fri May 24, 2013 11:03 am

Thanks for the update! Glad you are not affected by this issue currently.
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: The object has already been deleted

Veeam Logoby bskatz » Sat Jun 08, 2013 7:51 am

Hi
Right now I'm testing laboratory version 6.5.0.144 Veeam before installation for a client.
With the same result that conversation in this thread.
The object has already been deleted
This is a fresh installation on veem on server 2012, fresh installation vcenter apliance and two existing esxi version 5.1 and version 5.1 U1.
The problem occurs when processing order "FileOver now" into the veem console.
I assume there is no solution and to wait until the next version ?
Thanks

Bernardo Katz
bskatz
Lurker
 
Posts: 1
Liked: never
Joined: Sat Jun 08, 2013 7:36 am
Full Name: Bernardo Katz

Next

Return to VMware vSphere



Who is online

Users browsing this forum: albertwt, Google [Bot] and 12 guests