Mitigating job failure on vCenter migration: How to map VM UID/MoRefID to VBR Job Object?

[cloudy]

Hello,
I'm writing a ps script to add/remove the vCenter appliance from various jobs to prevent the job failing with error -

Virtual Machine VMware vCenter Server Appliance is unavailable and will be skipped from processing. Nothing to process. All machines were excluded from task list.

This error occurs because the vCenter appliance has been migrated to another host and then perhaps migrated back to the same host. The size of the vm in the job gui is 0 bytes. VBR can no longer find the VM since the vm appliance's uid/morefid has changed between host migrations and does not match the historical record in the job. As this is the vCenter vm it is backed up via the host directly.

My question is, what information can I use to correlate the uid of the VM to the info in the backup job to compare for a mismatch? I'd prefer not to rely on the status of the job and the error itself, rather the MoRefID. Ideally this script would run before the scheduled job to prevent failures from occurring in the first place. Am I overlooking something here, or is there some other novel autonomous solution that I haven't considered?


Find-VBRViEntity output:

Code: Select all

ConnHostId             : c2759693-9a95-489a-b7db-779bdfbf9b4d
Type                   : Vm
Reference              : 16
Id                     : c2759693-9a95-489a-b7db-779bdfbf9b4d_16
Name                   : VMware vCenter Server Appliance
Path                   : 172.16.24.10\VMware vCenter Server Appliance
VmHostName             : HostyMcHostFace
VmFolderName           : vm
IsLegacyFaultTolerance : False
FaultToleranceType     : None
GuestInfo              : Veeam.Backup.Model.CGuestInfo
Uuid                   : 564d0dc7-8189-c559-e1ad-3044acd63689
PowerState             : PoweredOn
IsTemplate             : False
ProvisionedSize        : 311209832064
UsedSize               : 311209823808
Platform               : EVmware

Get-VBRJob -Name output:

Code: Select all

LogNameMainPart              : JobbyMcJobFace
FreeBackupImpl               : 
IsFree                       : False
UserCryptoKey                : 
Uid                          : 22d92d61-ae27-4846-afc9-f9f89f38dc19
Id                           : 22d92d61-ae27-4846-afc9-f9f89f38dc19
Info                         : Veeam.Backup.Model.CDbBackupJobInfo
JobType                      : Backup
SourceType                   : VDDK
JobTargetType                : Backup
TargetType                   : Other
TypeToString                 : VMware Backup
Description                  : 
Name                         : JobbyMcJobFace
BackupPlatform               : EVmware
TargetHostId                 : 00000000-0000-0000-0000-000000000000
TargetDir                    : ddboost://dd.example.com:Veeam_Backups@/
TargetFile                   : JobbyMcJobFace
Options                      : Veeam.Backup.Model.CJobOptions
IsContinuous                 : False
BackupPolicyTag              : 
NameWithDescription          : JobbyMcJobFace
SqlEnabled                   : False
OracleEnabled                : False
PostgreSqlEnabled            : False
ParentJobId                  : 
ParentScheduleId             : 
Usn                          : 1198321794
PolicyJobId                  : 00000000-0000-0000-0000-000000000000
IsCdpReplica                 : False
HvReplicaTargetOptions       : Veeam.Backup.Model.CDomHvReplicaTargetOptions
BackupStorageOptions         : Veeam.Backup.Model.CDomBackupStorageOptions
BackupTargetOptions          : Veeam.Backup.Model.CDomBackupTargetOptions
HvSourceOptions              : Veeam.Backup.Model.CDomHvSourceOptions
NotificationOptions          : Veeam.Backup.Model.CDomNotificationOptions
JobScriptCommand             : Veeam.Backup.Model.CDomJobScriptCommand
VcdReplicaOptions            : Veeam.Backup.Model.CDomVcdReplicaOptions
ViReplicaTargetOptions       : Veeam.Backup.Model.CDomViReplicaTargetOptions
CloudReplicaTargetOptions    : Veeam.Backup.Model.CDomCloudReplicaTargetOptions
ViSourceOptions              : Veeam.Backup.Model.CDomViSourceOptions
VssOptions                   : VssSnapshotOptions: [Enabled: [False], ApplicationProcessingEnabled: [True], IgnoreErrors: [False], IsCopyOnly: [False], UsePersistentGuestAgent: [False]], WinGuestFSIndexingOptions: [Type: 
                               [None]], LinGuestFSIndexingOptions: [Type: [None]], SqlBackupOptions: [TransactionLogsProcessing: [TruncateOnlyOnSuccessJob], BackupLogsFrequencyMin: [15], UseDbBackupRetention: [True], 
                               RetainDays: [15], ProxyAutoSelect: [True]FailJobOnDbAbsenceOrBackupImpossibility: [False]], ExchangeBackupOptions: [TransactionLogsProcessing: [TruncateOnlyOnSuccessJob]], WinCredsId: 
                               [00000000-0000-0000-0000-000000000000], LinCredsId: [00000000-0000-0000-0000-000000000000], Old Properties: [Enabled: [False], IgnoreErrors: [False], GuestFSIndexingType: [None], 
                               IncludedIndexingFolders: [], ExcludedIndexingFolders: [], LinGuestFSIndexingType: [None], LinIncludedIndexingFolders: [], LinExcludedIndexingFolders: [], IsFirstUsage: [True]]
ScheduleOptions              : Start time: [6/9/2022 02:00:00], Latest run time: [6/8/2022 09:46:46], Timeout for backup completion: [180min], Next run time: [], Daily options: [Enabled: True, DayNumberInMonth: Everyday, 
                               Days: Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday]
IsChildJob                   : False
IsRunning                    : False
IsIdle                       : False
IsScheduleEnabled            : True
SourceProxyAutoDetect        : False
IsDeleted                    : False
IsInitialReplica             : False
IsMappingReplica             : False
IsEpPolicy                   : False
IsBackupPolicy               : False
IsBackup                     : True
IsBackupSync                 : False
IsBackupCopy                 : False
IsTapeCopy                   : False
IsStorageCopy                : False
IsPluginBackupCopy           : False
IsBackupCopySimpleMode       : False
IsLogBackupCopy              : False
IsWorkerJob                  : False
IsLogBackupCopyWorkerJob     : False
IsChildWorkerJob             : False
IsPolicyJob                  : False
IsSnapshotReplica            : False
IsLegacyReplica              : False
IsReplica                    : False
IsCloudReplica               : False
IsVmCopy                     : False
IsTapeBackup                 : False
HasNoJobRecord               : False
IsFileTapeBackup             : False
IsVmTapeBackup               : False
IsNasBackup                  : False
IsNasBackupCopy              : False
IsAgentManagementChildJob    : False
IsAgentManagementParentJob   : False
IsAgentPolicyChildJob        : False
IsAgentStandaloneJob         : False
LinkedJobs                   : {}
LinkedJobIds                 : {}
LinkedRepositories           : {}
LinkedRepositoryIds          : {}
LinkedBackups                : {}
IsForeverIncremental         : False
IsBackupJob                  : True
IsEndpointJob                : False
IsLinuxEndpointJob           : False
IsMacEndpointJob             : False
SheduleEnabledTime           : 11/9/2021 00:12:46
IsBackupCopyGfsEnabled       : False
AutoScheduleOptions          : Veeam.Backup.Model.CAutoScheduleOptions
PreviousJobIdInScheduleChain : 
LatestRunLocal               : 6/8/2022 02:00:02

Get-VBRJobObject -Job output:

Code: Select all

Id                  : e467b904-dbe3-4ea1-9368-0d6a52a6a494
ObjectId            : 71b3b464-d6ea-47c3-bbbd-40792cd6b44b
Usn                 : 1198318632
Object              : Name: VMware vCenter Server Appliance
TypeDisplayName     : Virtual Machine
Filter              : 
Role                : EInclude
Name                : VMware vCenter Server Appliance
Type                : Include
Location            : 172.16.24.10\VMware vCenter Server Appliance
OrderNo             : 0
ApproxSizeString    : 279.7 GB
Info                : Veeam.Backup.Model.CDbObjectInJobInfo
JobId               : 22d92d61-ae27-4846-afc9-f9f89f38dc19
IsFolder            : False
IsIncluded          : True
IsVssChild          : False
IsExcludeDisksChild : False
IsExcluded          : False
IsExtended          : False
PolicyType          : None
SourceObjType       : BackupObject
VssOptions          : VssSnapshotOptions: [Enabled: [True], ApplicationProcessingEnabled: [True], IgnoreErrors: [False], IsCopyOnly: [False], UsePersistentGuestAgent: [False]], WinGuestFSIndexingOptions: [Type: 
                      [ExceptSpecifiedFolders], IncludedFolders: [], ExcludedFolders: [%windir%; %ProgramFiles%; %ProgramFiles(x86)%; %ProgramW6432%; %TEMP%]], LinGuestFSIndexingOptions: [Type: [ExceptSpecifiedFolders], 
                      IncludedFolders: [], ExcludedFolders: [/cdrom; /dev; /media; /mnt; /proc; /tmp; /lost+found]], SqlBackupOptions: [TransactionLogsProcessing: [TruncateOnlyOnSuccessJob], BackupLogsFrequencyMin: [15], 
                      UseDbBackupRetention: [True], RetainDays: [15], ProxyAutoSelect: [True]FailJobOnDbAbsenceOrBackupImpossibility: [False]], ExchangeBackupOptions: [TransactionLogsProcessing: [TruncateOnlyOnSuccessJob]], 
                      WinCredsId: [00000000-0000-0000-0000-000000000000], LinCredsId: [00000000-0000-0000-0000-000000000000], Old Properties: [Enabled: [True], IgnoreErrors: [False], GuestFSIndexingType: 
                      [ExceptSpecifiedFolders], IncludedIndexingFolders: [], ExcludedIndexingFolders: [%windir%; %ProgramFiles%; %ProgramFiles(x86)%; %ProgramW6432%; %TEMP%], LinGuestFSIndexingType: [ExceptSpecifiedFolders], 
                      LinIncludedIndexingFolders: [], LinExcludedIndexingFolders: [/cdrom; /dev; /media; /mnt; /proc; /tmp; /lost+found], IsFirstUsage: [True]]
ExtendedOptions     : Veeam.Backup.Model.COijExtendedOptions
DiskFilter          : Veeam.Backup.Core.CDiskFilter
DiskFilterInfo      : Veeam.Backup.Model.CDiskFilterInfo
UpdateConfig        : True

[david.domask]

Hi @cloudy,

I'm not sure I fully got you right, but I think I understand you want to fetch the VM MorefID from the last backup to compare to the current production MoRef, is that correct?

If so, this is hidden in OIB data and you need to use unsupported methods to get at it:

Code: Select all

PS C:\Users\Administrator> $backup.GetLastOibs().Auxdata


VmMoRef                  : vm-23625
VmRef                    : vm-23625
Platform                 : EVmware
...

So you'd fetch the last backup that includes the VM (name probably is okay, then sort by CreationTime -Descending and return the first element of the list), then you can parse with the above.

[cloudy]

@david.domask
Thanks for that. I came up with this script which in theory should remove/add the vcenter appliance to the job. However, I see is the appliance being added to the VM exclusions in the job and the object is added to the job but still shows 0 B in the UI. On the console I see different. I've tried several different variations in adding/removing, removing/adding and iterating through the job objects as well, but nothing seems to resolve the 0B issue, at least via powershell. Any ideas what the issue might be?

Script:

Code: Select all

#update vca appliance backup jobs with correct vcenter objects
[System.Collections.ArrayList]$jobs = @{}
[void]$jobs.Add([PSCustomObject]@{ JobName = "vCenter_VCA"; ObjectName = "vCenter"; HostSubNet = "172.16.24.*" })
[void]$jobs.Add([PSCustomObject]@{ JobName = "vCenter_LEARN"; ObjectName = "VMware vCenter Server Appliance"; HostSubNet = "172.16.25.*" })
[void]$jobs.Add([PSCustomObject]@{ JobName = "vCenter_VCAZ"; ObjectName = "VMware vCenter Server Appliance"; HostSubNet = "172.16.26.*" })

#get vcenter managed object reference id from host (MoRef)
$vmware_hosts = Get-VBRServer | Where-Object {$_.Type -eq "ESXi" -and $_.Description -ne ""}
#$vmware_vca = Get-VBRServer | Where-Object {$_.Type -eq "VC"}
foreach ($j in $jobs) #find host of each vcenter appliance
{
    $vmInfo = $vmware_hosts | ForEach-Object { Find-VBRViEntity -Name $j.ObjectName -Server $_.Name -VMsAndTemplates | Where-Object {$_.Path -like "$($j.HostSubnet)"}} #get vm object from vsphere connection
    $job = Get-VBRJob -Name $j.JobName #get backup job object
    $backup = Get-VBRBackup | Where-Object {$_.JobId -eq $job.Id} #get backup chains linked to job
    $vmBackup = ($backup.GetLastOibs().Auxdata | Where-Object {$_.HostName -like "$($j.HostSubnet)"} | Sort-Object -Descending -Property CreationTimeUtc)[0] #get most recent backup chain that matches the host subnet
    $vmMoRef = $vmBackup.VmMoRef.RawRef #VmMoRef id
    if ($vmInfo.Reference -ne $vmMoRef)
    {
        Write-Output "Vm `"$($j.ObjectName)`" does not match object ref in job `"$($j.JobName)`""
        Get-VBRJob -Name $j.JobName| Get-VBRJobObject | ForEach-Object { Remove-VBRJobObject $_ }   
        $vmInfo | Add-VBRViJobObject -Job $j.JobName #add vm object with new VmMoRef to job. job cannot have 0 vm objects.
    }
}

Post script check

Code: Select all

PS >Get-VBRJob -Name "vCenter_VCA"| Get-VBRJobObject 

Name                      Type       ApproxSize       Location                 
----                      ----       ----------       --------                 
vCenter                   Include    335.9 GB         172.16.24.10\vCenter     
vCenter                   Exclude    0 B              172.16.24.10\vCenter     
vCenter                   Exclude    0 B              ha-datacenter/pure_Lin...

[david.domask]

Hi @cloudy

Add the -Completely flag:

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

If set to True, the cmdlet will permanently remove the objects from the job.

If set to False, the cmdlet will exclude the objects from processing.

[cloudy]

Oof, yes, rookie mistake not reviewing the cmdlet parameters. Using [-Completely] param on Remove-VBRJobObject resolved the issue.

Thanks for your guidance @david.domask!

[david.domask]

Glad it worked out @cloudy! always welcome and good luck with scripting