Automated (non-vCD) vmWare vApp restoring from Veeam Backup and Replication

[ljambdadelta]

Hello

I'm in need of often backup and restore of certain vApps from vmWare vCenter. These vapps has several VMs with a set of NICs connected to a set of networks. And these nested VMs have to be powered on in strict order. And there is specific limit which blocks me from restore to original location option.

This works flawlessly in manual mode. I backup it up and restore it to specified place. But as IT specialist I want to have a nice automated way to do such as a thing. More to say, mapping networks manually is quite a pain. So I came up with idea of scripting it. First attempt was through API but I've found soon that it is quite raw at this moment so I can not restore my VMs via API, only back them up.

Next approach i tried was powershell script. It was much better but there is still a problem that makes it not so useful. There is no commandlet restoring vApp for vCenter without Cloud Director which is too complex solution for given objectives. But this route led me to working solution as restoring it as resource pool. Not fully satisfying but better than nothing.

Next time I tried to work with SureBackup's Virtual Labs and still got no look. Besides it is quite bold use of unrelated feature, it has its way with networking so I coundn't found a way to map networks to distributed switches' ones, though it permits to start up VMs in order I could set up.

Right now I'm thinking of writing a script grabbing vApp info from vCenter API, preserving it in database somewhere, backup. Than, in the restore time, i should run powershell script to restore it as Resource Pool and than grab info about this vApp from DB and power on VMs through vCenter API. This plan is a little complex and I suppose there should be a better way.

So, basically, I'm looking for a way to achieve my objective to restore vApp in vSphere environment from Veeam Backup and Replication 11, preserving startup order and having a way to map my networks. I would appreciate any clues to achieve it.

Thanks in advance.

[oleg.feoktistov]

Hello,

This works flawlessly in manual mode. I backup it up and restore it to specified place.

Could you please clarify how you got it working?

I don't have much of experience with vSphere vApps in terms of how VBR interacts with them, but from the tests I carried out in v11a lab it looks like we don't backup vApp metadata with VMware vSphere backup jobs. We just interact with vApp as with any other virtual container (folder, tag, datastore etc.), expanding a list of VMs contained within and backing them up directly. You can also see that, as soon as VMs within a vApp are backed up, restore wizard treats target vApp as a resource pool thus not offering vApp metadata restore.

The precise difference between vSphere and vCloud Director backup jobs can be noticed if you test it with an empty vApp. vSphere job exits with Warning stating that there is nothing to process while vCD job creates a restore point with vApp meta.

We also have the note here that mentions it:

You can use a regular backup job to process VMs that are part of vApps created in vCenter Server. To back up VMware vCloud Director vApps, you must use specifically developed vCD backup jobs. For more information, see Backup and Restore of vApps.

So I don't see how this is natively supported in our logic. Otherwise, we would certainly add or at least consider adding it to Powershell.

Thanks,
Oleg

[ljambdadelta]

Hello Oleg

First of all thank you for your clarification. And sorry for the late response as things were hard for me in this period of time.

I also thought this behavior was strange but it worked somehow. The only thing I can tell for sure that we had relied on this logic and we had no problem with start-ups of the restored VMs in the wrong order. I have an supposal that VMs are backed up in the start order and restored in the same one as they have been backed up. But this theory is too doubtful and relies on the VMware default behavior which is not reliable even if I'm correct.
I can't provide you the proof or test it by myself as I've got fired from this company and obviously I have no rights to access this infrastructure.

I can describe the process, though. Initially, we had VMs placed in the vApp. Than we made backup job for VMs of this vApp, run it and result in having the backed up container (we thought of it as vApp but you have mentioned that they are resource pools in-fact) with corresponding VMs.
To restore it we created vApp on the ESXi server and in the restoration job we targeted at this vApp residing on named server. And here we have the point where we started: I couldn't find the way to target PowerShell-based deployment to the vApp.

In fact I've got my time to write work-around to achieve bewished behavior. Of course it relies on saving part of metadata in external database and works with Veeam and VMware PowerShell extensions. Is it correct to share it in here or should I provide the link to external resource?

Best regars,
Vasiliy

[oleg.feoktistov]

Hi Vasiliy,

Thank you for the context. If you have a powershell script that implements this workaround using Veeam and VMware powershell modules, yes, you can share it here. Just make sure to omit or put in X's any named values like ids, ip/dns etc.

The way I see it is now the UI presents target vApp as a resource pool, but powershell doesn't allow it. So the challenge is to present a vApp as a resource pool and target it for restore through powershell. Here is how you could achieve it:

Code: Select all

# Querying entities through VBR and vSphere
$server = Find-VBRViEntity -Name 'esxi01.local' -Servers
$esxi = Get-VBRServer -Name 'esxi01.local'
$parentRP = Find-VBRViResourcePool -Name 'ResourcePool1' -Server $esxi
$creds = Get-Credential
Connect-VIServer -Server 'vcenter.server' -Credential $creds
$vapp = Get-VApp -Name 'vApp1'

# Presenting found vApp as a resource pool and using it in restores
$resPoolItem = [Veeam.Backup.Core.Infrastructure.CViResourcePoolItem]::new('vApp1', $server.ConnHostId, $vApp.ExtensionData.MoRef.Value, 'vApp1') 
$backup = Get-VBRBackup -Name 'Backup'
$rp = Get-VBRRestorePoint -Backup $backup -Name 'machine'
Start-VBRRestoreVM -RestorePoint $rp[0] -Server $esxi -ResourcePool $resPoolItem

Please be aware that the part with new() method is not supported. It's just the workaround to this scenario.
Also, as I mentioned earlier, we don't backup vApp metadata for vSphere, so there might be some wrong startup order issues etc. upon the restore.

I have tested the script above in my v11a lab. The restore to the existing target vApp has worked fine.

Best regards,
Oleg

[ljambdadelta]

Hello Oleg,

Quite simple and elegant solution yet requiring deep dive into unsupported features. In my approach I've used VMware API for the vApp management and organized it as 2-step solution: 1) deployed VMs in RP 2) Moved contents of RP to vApp and than switch order by saved in my DB

Backing up w/saving start order metadata

Code: Select all

# Args: 
# vcsa:
# 	VMware vSphere Appliance to work with
# vappname:
#	vApp that ought to be backed up
# veeamhost:
# 	Veeam Host address 
# jobname:
#	Name for newly created backup job
param($vcsa, $vappname, $veeamhost, $jobname)

Import-Module VMware.VimAutomation.Core
Import-Module SimplySql
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false

# vSphere communication
Connect-VIServer -Server $vcsa -Protocol https
$vmwvapp = Get-VApp -Name $vappname
$bootorderdict = @{}
foreach ($vmwvappvm in $vmwvapp.ExtensionData.VAppConfig.EntityConfig) {
	$bootorder = $vmwvappvm.StartOrder 
	if ( -Not ($bootorderdict.keys -contains $bootorder) ) {
		$bootorderdict[$bootorder] = New-Object Collections.Generic.List[string]
	}
	$bootorderdict[$bootorder].Add($vmwvappvm.Tag)
}

# Update Database 
Open-MySqlConnection -Server "127.0.0.1" -Database "vmware-backups" -UserName "YourUserHere" -Password "YourPWHere" 
$zip= ""
foreach ($key in $bootorderdict.keys) {
	$val = $bootorderdict[$key] -Join "^"
	$zip = $zip + $key + "~" + $val + "~"
}
$sqlquerry = "INSERT INTO startorder (vAppName, orderdata) VALUES (@vappname, @dict) ON DUPLICATE KEY UPDATE orderdata=@dict;"
Invoke-SqlQuery -Query $sqlquerry -Parameters @{vappname = $vappname; dict = $zip} 
Close-SqlConnection


# Veeam communication
Connect-VBRServer -server $veeamhost
$repository = Get-VBRBackupRepository
$vappentity = Find-VBRViEntity -Name $vappname
$job = Add-VBRViBackupJob -Name $jobname -Entity $vappentity -BackupRepository $repository
Start-VBRJob $job

Restoring N instances of vApps w/start order stored in DB w/separate networks

Code: Select all

# vcsa:
# 	VMware vSphere Appliance to work with
# vmwhost:
# 	VMware production host
# datastore:
# 	Connected to host datastore to place data
# veeamhost:
# 	Veeam Host address 
# ethname:
# 	Ethalon name (as stated in backup job)
# podsnumber:
# 	Pods quantity to be dispatched
# podstartindex:
# 	Starting Index of pods if it differs from 1

param($vcsa, $vmhost, $datastore, $veeamhost, $ethname, $podsnumber, $podstartindex)

Import-Module VMware.VimAutomation.Core
Import-Module SimplySql
Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false

function Convert-VeeamEthalonNICNameToLabPodvSwitchName {
    param(
        $nicname,
        $podname
    )

    # 31 is the vswitch name limit. 7 is ".Pod0X.". Minus 1 to target at the last char
    $freelen = 31 - 7 - $nicname.Length - 1
    $shortpodname = (($podname -split "\.")[0][0..$freelen] -join "") + "." + ($podname -split "\.")[-1]
    $deploynetname = $shortpodname + "." + $nicname

    return $deploynetname
}

function New-vmWareRestoreResourcePool {
	param(
		$deployserver,
        $vmwhost,
        $podname,
		$deploydatastore,
		$ethalon,
		$vm_restorepoint,
		$strpodindex
	)
    
    $location = Get-VMHost -Name $vmwhost
    New-ResourcePool -Location $location -Name $podname	
	$rpool = Find-VBRViResourcePool -Server $deployserver -name $podname
	$sourcenets = @()
	$deploynets = @()
	# $nics = $vm_restorepoint[0].GetBackup().GetOIBS().Auxdata.Nics
	$nics = $ethalon.GetOIBS().Auxdata.Nics
	foreach($nic in $nics) {
		$sourcenets += $nic.Network
        $deploynetname = Convert-VeeamEthalonNICNameToLabPodvSwitchName -nicname $nic.Network.NetworkName -podname $podname
		$deploynetobj = Get-VBRViServerNetworkInfo -Server $deployserver | Where-Object { $_.NetworkName -eq $deploynetname }
		# If network is not added we suppose it as common like 'Cisco-labs' etc
		if ( -Not $deploynetobj ) {
			$deploynetobj = $nic.Network
		}
		$deploynets += $deploynetobj
	}
	Start-VBRRestoreVM -RestorePoint $vm_restorepoint[0] -Server $deployserver -ResourcePool $rpool -Datastore $deploydatastore  -SourceNetwork $sourcenets -TargetNetwork $deploynets
} #-EnableNetworkMapping

function Get-DBStoredBootOrder{
	param(
		$vappname
	)
	
	# Update Database 
	Open-MySqlConnection -Server "127.0.0.1" -Database "vmware-backups" -UserName "YourUserHere" -Password "YourPWHere" 
	$sqlquerry = "SELECT orderdata FROM startorder WHERE vAppName=@vappname;"
	$fullstring = Invoke-SqlQuery -Query $sqlquerry -Parameters @{vappname = $vappname} 
	Close-SqlConnection


	# Parse SQL response
	$bootorder = @{}
    $prelastcharid = $fullstring[0].length - 2
	$fullstring = $fullstring[0][0..$prelastcharid] -join ""
	$iskey = $true
	$key = ""
	foreach ($str in ($fullstring -split "~")) {
		if($iskey) {
			$iskey = $false
			$key = $str
            if ( -Not $bootorder[$key] ) {
                $bootorder[$key] = @()
                }
		} else {
			$iskey = $true
			$str = $str -split "^"
            $bootorder[$key] += $str[1..$str.Length]
		}
	}
	return $bootorder
}

function Write-NetworkNameTemplates { 
    param (
        $ethalon
    )

    $nictemplatelist = @()
    $podnametemplate = $ethalon.Name + ".Pod00"
    foreach ($nic in $ethalon.GetOIBS().Auxdata.Nics) {
        $nictemplatelist += Convert-VeeamEthalonNICNameToLabPodvSwitchName -nicname $nic.Network.NetworkName -podname $podnametemplate
    }
    Write-Output $nictemplatelist
    Read-Host -Prompt "Press any key to continue"
}

function Convert-DBStoredBootOrderToSpecvmOrder {
	param(
		$bootorder
	)
	
	$vmOrder = New-Object System.Collections.ArrayList
	foreach ($key in $bootorder.keys) {
		$vmOrder.Add(@{
			Group = $key
			VM = $bootorder[$key]
		})
	}
	return $vmOrder
}

function Set-vmWarevAppBootOrder {
	param(
		$vapp,
		$bootorder
	)
	
	$vmOrder = Convert-DBStoredBootOrderToSpecvmOrder -bootorder $bootorder
	$spec = New-Object VMware.Vim.VAppConfigSpec
	$spec.EntityConfig = $vapp.ExtensionData.VAppConfig.EntityConfig
	foreach($group in $vmOrder){
		$spec.EntityConfig | where{$group.VM -contains $_.Tag} | %{
			$_.StartOrder = $group.Group
		}
	}
	$vapp.ExtensionData.UpdateVAppConfig($spec)
}

function Move-vmWareResourcePoolContentsTovApp {
	param(
		$podname,
        $vmwhost,
		$deployserver,
		$FolderContainer,
		$bootorder
	)
	$vappname = $podname + "-vApp"
    $location = Get-VMHost -Name $vmwhost
	New-VApp -Location $location -Name $vappname -InventoryLocation $FolderContainer
    # Yes, New-VApp return vApp instance. But if it fails for any reason it won't. Duplicate is case for fail
    $vapp = Get-VApp -Name $vappname
	$rp = Get-ResourcePool -Name $podname
	foreach ($vm in $rp.ExtensionData.Vm) {
        $vmobj = Get-VM -Id $vm 
		Move-VM -VM $vmobj -Destination $vapp
	}
    Remove-ResourcePool -ResourcePool $rp
	Set-vmWarevAppBootOrder -vapp $vapp -bootorder $bootorder
}

# https://www.veeam.com/kb1489
# Add-PSSnapin VeeamPSSnapin


Connect-VBRServer -server $veeamhost
While(-Not (Connect-VIServer -Server $vcsa -Protocol https)) {
    Write-Output "Error. Try again."
}

$podstartindex = $(If ($podstartindex) {$podstartindex} else {1})
#$podsnumer = 3

$deployserver = Get-VBRServer -Name $vmwhost
$deploydatastore = Find-VBRViDatastore -Server $deployserver -Name $datastore

$bootorder = Get-DBStoredBootOrder -vappname $vappname

$ethalon = Get-VBRBackup -Name $ethname
$FolderContainer = Get-Folder -Name "Labs"

Write-NetworkNameTemplates -ethalon $ethalon


for ( $podidx = $podstartindex; $podidx -le $podsnumber; $podidx++ ) {
	$strpodindex = $(If ($podidx -lt 10) {"0"} Else {""}) + $podidx.ToString()
    $podname = $ethalon.Name + ".Pod" + $strpodindex
	foreach($vm_restorepoint in Get-VBRRestorePoint -Backup $ethalon) {	
		New-vmWareRestoreResourcePool -deployserver $deployserver -vmwhost $vmwhost -deploydatastore $deploydatastore -ethalon $ethalon -vm_restorepoint $vm_restorepoint -podname $podname -strpodindex $vm_restorepoint
		}
    Move-vmWareResourcePoolContentsTovApp -podname $podname -vmwhost $vmwhost -deployserver $deployserver -FolderContainer $FolderContainer -bootorder $bootorder
}

[ljambdadelta]

There is a flaw in this scriptlet that it doesn't use asynchronous powershell jobs because they lose context (described functions included) and I've come to know about it just after I've finished synchronous version. Well, there can be made modifications to improve that but I've got no more time on this project so it has been done as is.

[oleg.feoktistov]

Hi Vasiliy,

Thanks for sharing your script. Hope it helps somebody with a similar case.

Best regards,
Oleg