Veeam 7 and vCD Catalog Templates

[jgreenback]

Is anyone using Veeam7 with a private vCloud Director environment?
How do you handle backing up the Organization Catalog templates with Veeam? Veeam doesn't have a best practice for this case.

Thanks.

[Vitaliy S.]

Hi Jason,

Currently there is no out-of-the-box functionality to backup catalog templates, however these features are planned for future releases. As a workaround for now, you may want to backup VM templates directly from the vSphere environment.

Thanks!

[jgreenback]

Thanks. Can I assume this is on the roadmap for Veeam?

[Vitaliy S.]

Yes.

[the-d]

Hi,

will it be possible to backup ( & restore ) vCD catalogs or vApp Templates in one of the next releases?
This would be very helpfull, i don´t need the original vApp in the Organization or in a Backup.

Thanks&regards
Axel

[Vitaliy S.]

Hi Axel, yes, we are looking into adding this functionality. Thanks!

[jgreenback]

Hi Axel, yes, we are looking into adding this functionality. Thanks!

@Vitaliy S.
How soon can we expect to see the this functionality? Is there a timeline?
We are running in a Test/Dev environment and the catalogs are where all the time is invested. Running vApps only live for a few hours.

Thx,
jgreenback

[foggy]

How soon can we expect to see the this functionality? Is there a timeline?

Jason, unfortunately, we do not publish our roadmap.

[Dan Feeley]

Greetings,

I am also posting to make sure this feature is not lost. vCloud backup and restoration of catalogs is vital. it actually has a bigger impact than one might think. Without catalog backup/restoration it indirectly breaks other features that we also rely on greatly, such as linked clones.

First and most obvious, in the event of a total disaster it is not possible to restore your entire vCloud environment back to its original condition. There is no way to do it. It is impossible. Please someone prove me wrong.

Backing up the vApp catalogs through ESX instead of vCloud is not a realistic solution either. While you get a copy of the VM you can't really do much with them to restore their original state within vCloud, other than importing again as a net-new vApp. This is not a backup. This is a desperate last resort.

We have 900+ systems within our catalogs. We can have an easy 20+ vApps built from a single original vApp, with all of these as linked clones. If the catalog cannot be restored each of these vApps will be restored as unlinked clones. Currently we have 50+TB of linked clones using only 6+TB of actual space. Without the catalog you cannot restore the linked clone in its linked format. We do not have the capacity to restore 50TB of vApps.

My case on this issue is #00505677.

[haslund]

This is also something I am hoping to see added in v8.
We have many service provider customers who will only use Veeam if they have this functionality.

Please consider adding this in v8!

[Gostev]

Hey, Rasmus. It's way too late to consider anything at all for v8 at this time unfortunately, because we are in the final testing cycle already. The feature is not forgotten, but there has not been enough requests to justify adding it into v8. Thanks for your feedback though!

[haslund]

Fair enough, I hope it will make it into perhaps either an update or v9

[jgreenback]

We've been waiting for this feature since V7 was announced. Looks like it won't make it to V8 either. That's a shame. I hope more vCD/Veeam users come forward and ask for this feature as it's a major need, as I see it.
Until then, i hope this script i'm posting helps someone out there. If you have any questions or improvements on the script, please post them here.
Thanks.

This script establishes a remote PowerShell connection to your VeeamBR Server and GETs all Orgs, Catalogs, Templates, and then converts them to VM folder locations. They are then added/deleted on pre-existing backup jobs in Veeam BRS.
*PLEASE READ THROUGH THE SCRIPT BEFORE RUNNING IT*

Pre-Reqs
*Backup jobs need to be defined before running this script. Empty jobs that the script populates and maintains with VM folder locations.
*You need PowerShell v3 or higher on the machine that runs the PS script.
*Veeam BRS needs PowerShell v3 or higher PLUS PowerCLI 5.5 installed.

Code: Select all

####################################################################
#  Compare current vCD Org Catalog Templates (converted to vSphere #
#  Folders) to current Veeam backup job vSphere folder objects     #
#  then add new catalog templates and remove old ones from BACKUP  #
####################################################################

#Connect to VeeamBRS and Add Snapins

###################################################################
$Session = New-PSSession -Computername pa-veeam-brs
Enter-PSSession $Session

Invoke-Command -Session $Session -ScriptBlock {
   Add-PsSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue
   Add-PsSnapin -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue
   Add-PsSnapin -Name VMware.VimAutomation.Cloud -ErrorAction SilentlyContinue
} 

###################################################################

#Connect to vSphere and vCloud
#Write-Host "Connecing to vSphere"
#Connect-VIServer -Server "vcenter.lab.local"

Write-Host "Connecing to vCloud"
Connect-CIServer -Server "vcdcell.lab.local"

###################################################################

#Gloabl Variables and arrays
#
$vCenterServer = Get-VBRServer | Where {$_.Type -eq “VC”}


###################################################################

#vCloud Director - Get all vApp Tempaltes in all catalogs per $Org
$Orgs = Get-Org
#

foreach ($Org in $Orgs)
    { 
        Write-Host "Processing $Org........."
        $VeeamBackupJob = Get-VBRJob -Name "Backup Job - $Org - DR4100 - TEMPLATES"
		if ($VeeamBackupJob -eq $NULL)
		{
			Write-Host -WarningAction Continue "No Backup Backup Job exists for $Org.   Please create one if backups are required"
		    #CREATE JOB or continue
			continue
		}
                                                       ########### Filter out ARCHIVE and TEST catalogs   ############
		$AllOrgTemplates = Get-Org $Org | Get-Catalog | ? { $_.Name -notlike "Test*" -and $_.Name -notlike "*Staging" -and $_.Name -notlike "Archive" } | Get-CIVAppTemplate
		if ($AllOrgTemplates -eq $NULL)
		{
			Write-Host -WarningAction Continue "$Org has no Templates to backup."
		    #CREATE JOB or continue
			continue
		}

        $vCDTemplatesList = @()


        foreach ($Template in $AllOrgTemplates)
        { 
            $TemplateFullName = $Template.Name + " (" + $Template.Id.Substring(24) + ")"
            $vCDTemplatesList += $TemplateFullName
        }

    ###################################################################


    #####################################################################################
    #  Issues with the below portion of the script.  If a template is renamed it will not be able 
    #  to remove the old entry of that template from the defined backup job.
    #####################################################################################

        #Veeam - Get all objects in  - Template Backup job
        $VeeamBackupVMFoldersInfo = $VeeamBackupJob.GetObjectsInJob() | ? {$_.Role -eq "EInclude"}
		$VeeamBackupVMFolderNameList = @()
		if ($VeeamBackupVMFoldersInfo -eq $NULL)
		{
			Write-Host -WarningAction Continue "$VeeamBackupJob has no VMFolders defined."		    
		}
		else
		{
			foreach ($VMFolderPath in $VeeamBackupVMFoldersInfo)
			{ 
				if ($VMFolderName = $VMFolderPath.Location.Split("\")[5] -eq $NULL)
					{
						$VMFolderName = $VMFolderPath.Location.Split("\")[1]
                    
					} 
					else 
					{
						$VMFolderName = $VMFolderPath.Location.Split("\")[5]
					}
					$VeeamBackupVMFolderNameList += $VMFolderName
			}
		}
        ###################################################################

        #Compare the 2 arrays and add New vCD templates that are not present in Veeam BU Job


        $VeeamBackupVMFolderListDifferences = Compare-Object $vCDTemplatesList $VeeamBackupVMFolderNameList
        $VeeamBackupVMFolderToAdd = @()
        $VeeamBackupVMFolderToRemove = @()

        if($VeeamBackupVMFolderListDifferences -eq $NULL) 
            {
                Write-Host -WarningAction Continue "Nothing New Here. No need to continue.  Processing next Org."
                continue
            }
        
        foreach ($DifferentVMFolder in $VeeamBackupVMFolderListDifferences)
            {
            
                if($DifferentVMFolder.SideIndicator -eq "=>") 
                    {
                        $VeeamBackupVMFolderToRemove += $DifferentVMFolder.InputObject
                        Write-Host "$DifferentVMFolder needs to be DELETED from job"
                    }
                else 
                    {
                
                        $VeeamBackupVMFolderToAdd += $DifferentVMFolder.InputObject
                        Write-Host "$DifferentVMFolder needs to be ADDED to job"
                    }
            }

        ###################################################################

        #Load vCenterServer/FoldersandVMs view to find Template folder and add it to the Template backup job
        $LoadvCenterFoldersAndVMs = ((New-Object -typename Veeam.Backup.Core.InfrastructureTree.HostsAndClustersLoader).LoadHierarchy($vCenterServer.Info)).FoldersAndVms()

        #Add New vCDTemplates (VMFolders) to Veeam Backup Job
        $AddvCenterVMFoldersToVeeamBackupList = @()


        foreach ($vCenterVMFolder in $VeeamBackupVMFolderToAdd)
        {
            $VMFolder = Find-VBRViEntity -Server $vCenterServer -VMsAndTemplates -Name $vCenterVMFolder
			if($VMFolder -eq $NULL) 
            {
				 Write-Host "$VMFolder is NULL... continuing"
				 continue
			}
            $AddvCenterVMFoldersToVeeamBackupList += $LoadvCenterFoldersAndVMs.GetChild($VMFolder.Reference, "Folder", $true)
        }

        if($AddvCenterVMFoldersToVeeamBackupList -ne $NULL)
        {
	        Add-VBRJobObject -Job $VeeamBackupJob -Server $vCenterServer -Entities $AddvCenterVMFoldersToVeeamBackupList
        }

        #Delete VMFolders From Veeam Backup Job        
        foreach ($VeeamVMFolder in $VeeamBackupVMFolderToRemove)
        {
            $VBRVMFolder = Get-VBRJobObject -Job $VeeamBackupJob -Name $VeeamVMFolder
            $VBRVMFolder | Remove-VBRJobObject
            $VBRVMFolder.Delete()
        }

    }
###################################################################

#Exit and delete PS Session
Exit
Get-PSSession | Remove-PSSession

###################################################################

[jgreenback]

Question: Does anyone know if vCloud Director Catalog backup functionality made it into Veeam V9? We've not upgraded yet and I did not see it mentioned in the release notes.

[Vitaliy S.]

Hi Jason, this feature is not part of v9 release.

[jgreenback]

Since we're on the subject of backing up vCloud Catalogs I thought I should update my script post with my changes.
I now handle (sort of) mis-matched VM Folder names in vSphere caused by special characters used in naming a catalog template in vCloud.
Also changed the order in which I add and delete VM folders as this was causing some issues because the veeam cmdlets would only update the NAME of the object in the job but not the location of that object. Not sure why. I'm thinking it has to do with the VM Folder MoREf ID in vSphere being the unique identifier that the cmdlets use in the property Reference.
Finally, I've added logging and a final check for an error log. If it finds an error log was generated it emails the contents to our admins.
Hope this helps someone out there. PM me is you have questions or improvements.
Thanks.

UPDATES:
- Added logging via Start-Transcript / Stop-Transcript in PowerShell
* Output is a daily single file that is appended. New logs each day.
* Change the -Path to where you want the log to go.
- Determine if the Template name can be matched to a vSphere (VMs and Templates) folder. If not, then kick out an error file (dated) with the details.
* When using special characters the folder name in vSphere is shortend thus causing the mis-match and the error.
Not sure how else to handle this. We just email the owner and ask them to rename their template without special characters.
- Email section that checks for and error file and send the admins the contents of the error file.
- Removed Remote PS Session. No longer running remotely due to security hardening here.
- Changed order in which I add/delete VM folders from the backup jobs. Delete first, Add last.
- Now using v8 cmdlets.

*PLEASE READ THROUGH THE SCRIPT BEFORE RUNNING IT* *USE AT YOUR OWN RISK*

Code: Select all

####################################################################
#  Compare current vCD Org Catalog Templates (converted to vSphere #
#  Folders) to current Veeam backup job vSphere folder objects     #
####################################################################
#Start Logging
Start-Transcript -Path "C:\Veeam\Catalog Backup Script Log\vCDTemplateBackup$(get-date -format MM-dd-yy).log" -Force -Append -NoClobber

#Email and Error log variables
$MailServer = "{SMTP_SERVER or IP goes here}"
$ErrorFile = "C:\Veeam\Catalog Backup Script Log\ERROR_$(get-date -format MM-dd-yy).log"

###################################################################
#Load all snapins
Add-PsSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue
Add-PsSnapin -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue
Add-PsSnapin -Name VMware.VimAutomation.Cloud -ErrorAction SilentlyContinue

###################################################################
#Connect to vCloud instance
Write-Output "Connecing to vCloud"
Connect-CIServer -Server "{VCLOUDSERVERNAME or IP goes here}"
Write-Output "`r`n"
###################################################################

#Get vCenter Server from Veeam
$vCenterServer = Get-VBRServer | Where {$_.Type -eq “VC”}


###################################################################

#vCloud Director - Get all vApp Tempaltes in all catalogs per $Org
$Orgs = Get-Org

foreach ($Org in $Orgs)
    { 
        Write-Output "####################################################`r`n"
        Write-Output "Processing $Org.........`r`n"
        $VeeamBackupJob = Get-VBRJob -Name "Backup Job - $Org - TEMPLATES"
if ($VeeamBackupJob -eq $NULL)
{
Write-Output -WarningAction Continue "No Backup Backup Job exists for $Org.   Please create one if backups are required"
            Write-Output " "
    #CREATE JOB or continue
continue
}
        ### FILTER OUT unwanted catalogs here: ARCHIVE and TEST catalogs ###
$AllOrgTemplates = Get-Org $Org | Get-Catalog | ? { $_.Name -notlike "Test*" -and $_.Name -notlike "*Staging" -and $_.Name -notlike "Archive" } | Get-CIVAppTemplate
if ($AllOrgTemplates -eq $NULL)
{
Write-Output -WarningAction Continue "$Org has no Templates to backup."
            Write-Output " "
    #CREATE JOB or continue
continue
}

        $vCDTemplatesList = @()


        foreach ($Template in $AllOrgTemplates)
        { 
            $TemplateFullName = $Template.Name + " (" + $Template.Id.Substring(24) + ")"
            $vCDTemplatesList += $TemplateFullName
        }

    ###################################################################


    ###################################################################

        #Veeam - Get all objects in  - Template Backup job
        $VeeamBackupVMFoldersInfo = $VeeamBackupJob.GetObjectsInJob() | ? {$_.Role -eq "EInclude"}
$VeeamBackupVMFolderNameList = @()
if ($VeeamBackupVMFoldersInfo -eq $NULL)
{
Write-Output -WarningAction Continue "$VeeamBackupJob has no VMFolders defined."    
}
else
{
foreach ($VMFolderPath in $VeeamBackupVMFoldersInfo)
{ 
                $VMFolderName = $VMFolderPath.Location.Split("\")[-1]
$VeeamBackupVMFolderNameList += $VMFolderName
}
}
      #######################################################################
        
        #Compare the 2 arrays and add New vCD templates that are not present in Veeam BU Job


        $VeeamBackupVMFolderListDifferences = Compare-Object $vCDTemplatesList $VeeamBackupVMFolderNameList
        $VeeamBackupVMFolderToAdd = @()
        $VeeamBackupVMFolderToRemove = @()

        if($VeeamBackupVMFolderListDifferences -eq $NULL) 
            {
                Write-Output -WarningAction Continue "Nothing New Here. No need to continue.  Processing next Org."
                Write-Output " "
                continue
            }
        
        foreach ($DifferentVMFolder in $VeeamBackupVMFolderListDifferences)
            {
            
                if($DifferentVMFolder.SideIndicator -eq "=>") 
                    {
                        $VeeamBackupVMFolderToRemove += $DifferentVMFolder.InputObject
                        Write-Output "$DifferentVMFolder needs to be DELETED from job"
                    }
                else 
                    {
                
                        $VeeamBackupVMFolderToAdd += $DifferentVMFolder.InputObject
                        Write-Output "$DifferentVMFolder needs to be ADDED to job"
                    }
            }

        ###################################################################
        
        #Load vCenterServer/FoldersandVMs view to find Template folder and add it to the Template backup job
        $LoadvCenterFoldersAndVMs = ((New-Object -typename Veeam.Backup.Core.InfrastructureTree.HostsAndClustersLoader).LoadHierarchy($vCenterServer.Info)).FoldersAndVms()

        #Add New vCDTemplates (VMFolders) to Veeam Backup Job
        $AddvCenterVMFoldersToVeeamBackupList = @()

        #Add space between delete/add list
        Write-Output " "

        #Delete VMFolders From Veeam Backup Job.  This must be done first since renaming catalog templates only changes the name and location but not the "Reference:" property
        #thus deleting newly added templates when ran second.
        
        foreach ($VeeamVMFolder in $VeeamBackupVMFolderToRemove)
        {
            Write-Output "## Deleting $VeeamVMFolder from Backup Job - $Org - TEMPLATES ##"
            Get-VBRJobObject -Job $VeeamBackupJob -Name $VeeamVMFolder | Remove-VBRJobObject -Completely -Verbose
            Write-Output "#########################################################################"
            Write-Output " "
            Write-Output " "
        }

        #Add has to be done second since we are adding a different folder name with the same "Reference:" property

        foreach ($vCenterVMFolder in $VeeamBackupVMFolderToAdd)
        {
            if($vCenterVMFolder -eq $NULL)
            {
 Write-Output "$vCenterVMFolder is NULL... continuing"
 Return
            }
            Write-Output "## Adding $vCenterVMFolder to Backup Job - $Org - TEMPLATES ##"
            $AddVMFolder = Find-VBRViEntity -Server $vCenterServer -VMsAndTemplates -Name $vCenterVMFolder

                if($AddVMFolder -eq $NULL)
                {                    
                    Write-Host "**ERROR**`r`n$vCenterVMFolder does not exist in vSphere.`r`nVerify that the template name matches the corresponding vSphere folder.`r`nTo check this go to the catalog of the $Org Organization >> Find, then right-click on $vCenterVMFolder >> choose Open in vSphere Web Client.`r`nIf the 2 values do not mathc contact the Org Admins of $Org and ask them to rename the template.`r`n" -foregroundcolor "Red"
                    Write-Output "$Org Org Template ::$vCenterVMFolder:: was not found.`r`nvSphere folder name does not match the vCloud template name."  | Out-File "C:\Veeam\Catalog Backup Script Log\ERROR_$(get-date -format MM-dd-yy).log"
                    Continue
                }
                else
                {
                    $AddVMFolder | Add-VBRViJobObject -Job $VeeamBackupJob -Verbose
                    Write-Output "#######################################################################"
                    Write-Output " "
                }
        }
    Write-Output "Finsihed with $Org`r`n"
    Write-Output "####################################################`r`n"
    }
#Checkif there is an error file, if so, email contents to Admins
if (Test-Path $ErrorFile)
{   #Send content of error file to DevOps Admins
    Send-MailMessage -From "veeam_service@yourwork.com" -To "vCloudAdmins@yourwork.com" -Subject "Error On Template Backup Script" -body (Get-Content "C:\Veeam\Catalog Backup Script Log\ERROR_$(get-date -format MM-dd-yy).log" | Out-String) -dno onSuccess, onFailure -smtpServer $MailServer
}
else
{
    Write-Output "No errors found... this time`r`n"
}

#End Logging
Stop-Transcript

[jmarlhioud]

Is this feature planned for a future release?

[Vitaliy S.]

Yes, it is still logged for our next updates, just checked its status in the tracking system.

[jgreenback]

I've been looking forward to this feature for a while. Thanks for the update Vitaliy.

[jmarlhioud]

Yes, it is still logged for our next updates, just checked its status in the tracking system.

Looks like it didn't make it into 9.5, can you give us an update as when this could be available?

[Vitaliy S.]

As far as I see, it is still pending for the next major releases. Do not have an ETA yet.

[mattbrown80]

Hi

We have been waiting for this feature for quite a while - is it still on the roadmap?
Using 'vSphere' backups for vCloud Director Catalogs isn't an acceptable workaround long term.

Thanks

[shawn-wwt]

Is backing up vApp templates supported? We are running vCD 9.7. Thanks!

[jgreenback]

Hi @shawn-wwt,
We’re running vCD 9.1 and testing with 9.7.
I can confirm that backing up templates is still not supported in Veeam 9.5u4b.
I’m hoping we’ll see this in Veeam 10.

[foggy]

This is not a part of v10 release. Thanks!

[wfturner_merit]

Catalog support was also not available in B&R V11. Any chance we'll see it in V12, or is it already too late for that release?

[Vitaliy S.]

I don't believe anything has changed (even in v12), but If I find out otherwise I will update the thread with more info.

[mattbrown80]

Any update on when this will be available? It has been 10 years since the feature was requested

[Gostev]

It's not currently planned, too few requests in 10 years vs. other features.