Features request: vmware UUID & MoREF improvements in ESX

[jmm]

[ ref:_00D30RWR._5000e1DSkLH:ref ]
[ID# 02205218]

The purpose of this feature request is to trouble shoot false positive VM identification in a vmware environment where veeam backup access ESX and not vSphere (cf : Why don't we present vSphere to Veeam ? below).
That's happen during Vmotion from one ESX to another that's error block Veeam Jobs Backup :
"Object was not found in hierarchy cache"

Moref, inventory identifier of a VM in vmware are unique inside the vSphere and inside an ESX host but not between ESX host ! :
So basically during a Vmotion :

vSphere MoREF is preserved (cf : http://www.virtuallyghetto.com/2011/11/ ... hange.html)
BUT ESX MoREF is NOT preserved ...

So after a lot of VMOTION events VEEAM lost the VMS and can no more attach the VM on his new ESX to his job.

So I would like a new feature in VEEAM : the ability to choose the way Veeam could retrieve VM that's disappear from his cache.
It could be a parameter that say : if VEEAM can no more identifie a VM on an ESX because his cached MoREF / UUID can no more be retrieve :
--> Veeam will search on other ESX with both : VM name and UUID.

I will be perfect in my environment, and after a little googloing on this question it could help some others users with that sort of problem.

The Full story :

Additional Details: Hi, We have done this WE a lots of Vmotion this WE in our Vsphere infrastructure. Then this morning i have noticed lots of message from VEEAM b&r : "Task failed Error: Object with reference '78' was not found in hierarchy cache" 1/ we got vSphere vs 6 but we manage our infrastructure in Veeam b&r directly by ESX (it is more fast to access ESX because our server with vSphere got a low WAN access to the main datacenters where resides ours main ESX and VEEAM b&r) : does this implementation could explain this bad behavior or may it be a bug on VEEAM b&r ? 2/ By powershell, Is it possible to get the "vmware id" that record Veeam for a VM in a job so that we could script when Vmotion happens : and so detect the new id and put in veeam job ? Thank you to help us resolve this problem.


So we wrote a first workaround. But this is not ok.

So we may change to use a request to correct directly in VEEAM SGBD MoRef
ID :
we will try to catch that vmotion events and then put in the table
"bobjects" , field "object_id". the new correct MoRef ID :

1/ We read that it may corrupt the CBT ? Could you help us to know wish
consequences of that action ?
Answer of the VEEAM Support : Such approach will eventually break the backups - MoREF is stored in the database and backup files metadata, so should any merges or restores happen, there will be a mismatch.
2/ Do we have to refresh " Veeam infrastructure Cache" and it is enough to
use that powershell :

Code: Select all

Rescan-VBREntity  -AllHosts

Answer of the VEEAM Support : SNot sure I understand you correctly, but if you mean "should we do a rescan after we edit bObjects", then - no, as it's better not to touch bObjects at all.


Our Conclusions so far :

a/ VEEAM don't manage VMOTION in an ESX backup Infrastructure. (It may work
on a vSphere backup infrastructure, but in our local situation (Wan to low
between the vSphere Server and the VEEAM server) --> This is a big problem
for us.

b/ If a VMOTION happen : no backup.

c/ If Vmotion is planned (hardware operation)
----> Veeam offer only manual operation on each job to backup again.
----> This manual operation could be automatized via Powershell (cf ps
script description below), but with the sames consequences :
(we use reverse incremental)
--> Nest execution of batch Job will be full (?)
--> Partial lost of the continuity of the VM :
In Veeam console : i have two VMs Jobs Disk /...
But physical i find only only one folder with something quite OK



*POWERSHELL script to detect and correct VMOTION MoRef ID changes.*

I have now a script to detect and correct that problem :

Code: Select all

#  do what  https://www.veeam.com/kb1299 say.
#------------init
# Need Powercli ok on your machine

#------------Paramètres des serveurs
$srvVeeam='VEEAMSERVER'

 
$VEEAMids=Invoke-Sqlcmd -Serverinstance "wda00001" -Database "VeeamBackup"
 "SELECT
right(Objectsinjobs.location,len(Objectsinjobs.location)-CHARINDEX('\',Objectsinjobs.location,-1))
as Fname,
Objectsinjobs.location, bobjects.object_id,BObjects.uuid  FROM BObjects
INNER JOIN ObjectsInJobs ON ObjectsInJobs.object_id = BObjects.id where
ObjectsInJobs.location is not null and ObjectsInJobs.location<>'';"

$Esx_filtre=@("01is.intra","02is.intra") #ESX to enumerate

foreach ($esx in $Esx_filtre) {
   $srvvSphere=$esx
   $viserver=Connect-VIServer -Server $srvvSphere -User "myrootuser"  -Password "mypasswoord"
   $srvvSphere


        Foreach ($VEEAMid in $VEEAMids) {
                $veauid=$VEEAMid.object_id
                $n++
                $Vvms=get-VM
                $viid= $Vvms | Where-Object {$_.Name -like $($VEEAMid.Fname)} | %{
                $_.ExtensionData.MoRef.value}
               # "$($VEEAMid.Fname) $veauid $viid"
                        if ( ($veauid -and $viid) -and ($veauid -ne $viid)  ) {
                                "NOT Ok $($VEEAMid.Fname) $veauid $viid"
               # we write corrections to the job
                $job=$($VEEAMid.Fname)
                        Get-VBRJob -Name $job | Get-VBRJobObject -Name $job |
                        Remove-VBRJobObject -Completely
                        Find-VBRViEntity -Name $job | Add-VBRViJobObject -Job $job

        }
}
Disconnect-VIServer -Confirm:$false
}

Why don't we present vSphere to Veeam ?

We have a first big issue when we start deploy VEEAM in our infra : " [ID# 02157314] Long time before copy backup data for somes VM "
We decide to present our ESX to VEEAM and not our vSphere.

We have changed, now we don't connect to vSphere but we mount esx directly into VEEAM.
Our proxies are in "transport Mode" => "Automatic selection". And everything seems ok : SAN is majority used, and fast like we wishes. (no more accident during backup)
--> Cause by the fact that our Main "VEEAM backup server" is on a vlan and a datacenter and our vSphere vcenter server on another vlan / datacenter with hundred kilometers distance.

Réf (thank you to the authors and Veeam support team for the help and good advises) :

https://www.interworks.com/en-gb/blog/i ... ror-vmware

https://www.virtual-allan.com/veeam-err ... not-found/

http://robvit.com/veeam/backup_replicat ... not-found/

http://www.virtual-guy.com/?p=16

https://tecbabble.wordpress.com/2013/08 ... b-id-1299/

[Andreas Neufert]

Thanks for your request.

I understand the technical facts of this situation.

Please know that the above script is not supported and Veeam support has it´s own scripts for this that they can share with customers.

If you want to protect VMs automatically at vmotion without vcenter usage, you can add the ESXi host at Backup Job VM scope and Veeam will protect all VMs automatically. Yes, vmotioned VMs are detected as new VMs and will consume extra backup space.

There is no good way to reliable detect VMs on ESXi host added infrastructures reliable as there is no central instance that takes care about numbers. I had lately some issues with a approach described by you that had deleted a production VM because the VM was detected the wrong way (correct number but other ESXi host had this number in usage already. If VMware do not change anything for us the ID change process will always be a task that a admin need to decide (see above mentioned support script. I would only do this if you had no other chance... for example if you lost your vcenter completely or migrated the wrong way with new morefs. To link existing backups to the "new" detected VM is dangeorus and I would not do this to protect my backup approach overal. At least if you do so, use SureBackup.

There are several ways out of this.
Create a vcenter User that has only the specific ESXi host of one site visible. That way Veeam will not scan the other ESXi hosts => reduced data between B&R and vcenter.
As well important, Backup & Replication caches all vcenter informations (scope them as above) and at backup time the data flow between B&R and vcenter is limited to only the action tasks. Rest is done by reading from infrastructure cache.
Infrastructue Cache was optimized several times => Use lates Version and Update level.

Another way is to place B&R next to vcenter and roll out Proxy, Repository and Guest Interaction Proxy on the ROBO site.

[jmm]

Thank you very much for yours explanations.
Our final goal will be to place B&R next to vcenter (middle-term action) but at short-term we will try your second solution too "Create a vcenter User that has only the specific ESXI host of one site visible", we have not imagined it at all.
In fact during our first POC with vcenter and B&R master role in two distant data-centers we don't noticed that's problem.
During our first implementation in production of B&R we encounter very long backup. The main cause was caused by the distant vcenter server vs B&R but it may be explained too by configurations changes in our WAN connection ... (and it is quite tricky to know what ;...) ... / ... so we finally used ESXI host against vcenter in B&R infrastructure to put in production.

[Andreas Neufert]

https://www.veeam.com/veeam_backup_9_0_ ... ons_pg.pdf
is an overview about needed permissions. See complete list at the end. Please be aware that some of the rights need to be given to the user on global level to work correctly.