Reset CBT on VMs during full backups - Pre-Backup-Script

[the_mentor]

Hi Everyone,
Due to the whole issues with CBT recently i would like to see if i can find a way to run a Pre backup script that will check if the running job is running in active full mode.
and if it does it will do a CBT reset to all the VMs in the job.

Does anyone have any idea on how to accomplish such a thing?

Thank you!

[veremin]

You can get job source objects, using this cmdlet, and then reset CBT for those objects prior to active full backup. Thanks.

[the_mentor]

Thank you v.Eremin,
Any ideas on how to check if the running backup job a full job?

[the_mentor]

Ok I've made some progress on the script so far
So far it seems that the i manage to detect that the running job is running in full mode.
The issue i'm facing now is it seems that "Get-VBRJobObject" doesnt get me the whole list of VMs since in some of the jobs I use folders and it seems to list them in there rather than a full list of VMs.

Code: Select all

$BackupJobName = "TheNameOftheVeeamJob"
$VBRJobObj = Get-VBRBackup -Name "$BackupJobName"
if($VBRJobObj){
    Write-Host "Veeam Job Detected - '$BackupJobName'"
    Write-Host "Detecting if Running Job is running in FullMode..."
    $VBRRunningJobBackupSession = Get-VBRBackupSession | Where-Object {$_.State -like "Working" -and $_.Name -like "*$BackupJobName*"} 
    if($VBRRunningJobBackupSession.IsFullMode){
        Write-Host "Full/Active Full Backup Mode Detected for '$BackupJobName'."
        Write-Host "Getting VM Objects Inside Veeam Job - '$BackupJobName'"
        $VeeamJobVMs = Get-VBRJobObject -Job "$BackupJobName" 
        
        #Looping through VMs in Veeam Backup Job
        foreach ($VM in $VeeamJobVMs) {
            $VMName = $VM.name
            Write-Host $VMName
            #Code to Reset CBT
            #Reset-CBT -VM "$VMName" -vCenter $vCenterServer
        }
    }
}
else {Write-Host "Veeam Job Not Found"}

[tsightler]

So I really see two options (there may be more, but this is what comes to mind):

1) You can grab the folder objects as you're doing now and then attempt to parse out the VMs in that folder in your code. I have some example code that does this that I'll share below. This will work and is not that difficult, but if you start getting into exclusions and nested objects it can get very complex quite quickly.

2) You can assume that the VM objects haven't changed since the last backup job and just using something like the following:

Code: Select all

$VeeamJobVMs = $job.FindLastBackup().GetLastOibs()

This simply pulls the list of individual VM objects from the last backup run. This has some small risk such as a VM that failed on the most recent run wouldn't be listed, or if a VM was added between the incremental and the full run so it didn't actually exist in the job in the previous backup, but I'd think this would likely be corner cases that wouldn't happen very often, and would just get caught on the next full cycle.

Below is the code I use to list VMs that are in a folder. It's a little slow because of the call to Find-VBRViEntity which loads the entire hierarchy into a variable, but this makes it much faster for reuse later if you need to loop through different folders in different jobs. Easier to pay the price for the call to vCenter once rather than mulitple times. It also assumes only a single folder object in the job, but wouldn't be very difficult to add logic to parse out multiple folders in a job, it just wasn't required for the use cases I've used this code for to this point.

The code currently just looks at a job, grabs the folder object, and quickly parses out what VMs are in that folder hierarchy and prints out their names but it should be easy to adapt to other use cases.

Code: Select all

asnp "VeeamPSSnapIn" -ErrorAction SilentlyContinue

$jobname = "<Job_Name>"

# Get entire vCenter VM hierarchy using VM and Template view for object [ath
$vmsandtemplates = Find-VBRViEntity -VMsAndTemplates
$vmfoldertree = $vmsandtemplates |? {$_.Type -eq "Vm"}
$vmfolders = $vmsandtemplates |? {$_.Type -eq "Folder"}

# Get Backup Job
$job = Get-VBRJob -Name $jobname | Sort -Property Name

write-Host $job.Name
# Get all included objects in job (assumes single folders)
$jobobjs = $job.GetObjectsInJob() | ?{$_.Type -eq "Include"}
# Get path for folder object
$jobobjid = $jobobjs.GetObject().Info.HostId.ToString() + "_" + $jobobjs.GetObject().Info.ObjectId
$jobobjpath = ($vmfolders | ?{$_.Id -eq "$jobobjid"}).Path
write-host $jobobjpath
# Get subset of VMs that are in the folder
$vmsinfolder= $vmfoldertree |?{$_.Path -like "$jobobjpath*"} | Sort -Property Name

ForEach ($vm in $vmsinfolder) {
    write-host "   " $vm.Name
}

[the_mentor]

tsightler,
The below is fantastic i think I will go with objects based the last backup since most of the times we dont add or remove new objects to a job and even if we do and they are missed they will be a part of the next full cycle.

I'm only left with one question before I can consider my script perfect.
Is there a way to determine what is the name of the vCenter server of a processed VM?
I would like to have the script figure that out rather than have a hard coded name.
Also this is important in case we have a job that has VMs in 2 vCenters.

Thank you very much for all the assistanace i will post the script here once I'm done with it so other people can also utilize it.

[tsightler]

In your code above you should be able to get the vCenter name for each VM with something simple like:

Code: Select all

$VM.AuxData.HostName

Let me know if that doesn't work for some reason. There are several other ways you can get the parent host information for the objects in the backup.

[the_mentor]

tsightler,
That code worked like a charm thank you for all the help.
I'm doing some final testing and then I'll post the script.

[the_mentor]

Here is the Full Script I've Created.
Let me know if you have any questions/feedback/ideas/issues etc...

Code: Select all

param (
[parameter(Mandatory = $true)]
[string]$BackupJobName
)
cls
<#
.Author 
    Name: Avri Roth 
    Email: Avri@AvriTech.com
    Please review the script I test it in my environment without issue but you should always check someone's else's code and execute it at your own risk.
.SYNOPSIS
Resets the Change Block Tracking (CBT) file on Veeam Full Backups.
.DESCRIPTION
The Script will detect if the Veeam Backup job runs in Full Backup mode and than reset the Change Block Tracking (CBT) file for  Virtual Machine within the job.
.Credit for the CBT Reset Function/Script
Written by Chris Wahl for community usage
Twitter: @ChrisWahl
GitHub: chriswahl
.LINK
.Note 
Edit the Veeam Job and go to Storage>Advanced>Scripts> Run the following script before the job
Enter Powershell C:\Patchtoscript\VeeamCBTResetOnFullBackup.ps1 -BackupJobName 'Name of the Backup Job' 
The name of the backup job should be the name of the job you are adding this to (i havent found a dynamic way to detect the job name yet.
The Script will detect the current running folder and save a log file with the following format MM/DD/yyyy-BackupJobName
.EXAMPLE
.\VeeamCBTResetOnFullBackup.ps1 -BackupJobName 'BackupJob Name' 
.VERSION
1.0 - Initial Script
#>

#### CONFIG BEGIN ####
#Import Required Addins
if ( (Get-PSSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue) -eq $null )
{
    Add-PsSnapin VeeamPSSnapIn
}
if ( (Get-PSSnapin -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue) -eq $null )
{
Add-PsSnapin VMware.VimAutomation.Core
}
$invocation = (Get-Variable MyInvocation).Value
$CurrentDirectoryPath = Split-Path $invocation.MyCommand.Path
$LogPrefix = $(get-date -Format "yyyy-MM-dd")
$LogFile = "$CurrentDirectoryPath\$LogPrefix-$BackupJobName.log"
#### CONFIG END ####

#### FUNCTIONS BEGIN ####
Function AddToLogFile($logText,$logcolor){
    $dateLog = $null
    $dateLog = Get-Date -format "yyyy/MM/dd hh:mm tt:"
    $logText = "$dateLog $logText"
    If ($logcolor -ne $null) {Write-Host $logText -foreground "$logcolor"}else{Write-Host $logText} #check for output color and set the output color
    Add-Content $LogFile $logText
}

#Requires -Version 2
Function Reset-CBT{
    <#  
            .SYNOPSIS
            Resets the Change Block Tracking (CBT) file for affected Virtual Machines
            .DESCRIPTION
            The Reset-CBT cmdlet will reset the Change Block Tracking (CBT) file for a Virtual Machine affected by issues or corruption with the CBT file.
            .NOTES
            Written by Chris Wahl for community usage
            Twitter: @ChrisWahl
            GitHub: chriswahl
            .LINK
            https://github.com/WahlNetwork/powershell-scripts
            https://github.com/WahlNetwork/powershell-scripts/blob/master/VMware%20vSphere/Reset-CBT.ps1
            http://wahlnetwork.com/2015/12/01/change-block-tracking-cbt-powercli/
            .EXAMPLE
            Reset-CBT -VM 'WAHLNETWORK' -vCenter VCENTER.DOMAIN.LOCAL
            Disables CBT for a VM named WAHLNETWORK, then creates and consolidates (remove) a snapshot to flush the CBT file. The assumption here is that your backup software will then re-enable CBT during the next backup job.
            .EXAMPLE
            Reset-CBT -VM 'WAHLNETWORK' -vCenter VCENTER.DOMAIN.LOCAL -NoSnapshots
            Disables CBT for a VM named WAHLNETWORK but will not use a snapshot to flush the CBT file. This is useful for environments where you simply want to disable CBT and do not have backup software that will go back and re-enable CBT.
            .EXAMPLE
            Reset-CBT -VM $VMlist -vCenter VCENTER.DOMAIN.LOCAL
            Disables CBT for all VMs in the list $VMlist, which can be useful for more targeted lists of virtual machines that don't easily match a regular expression.
            Here are some methods to build $VMlist
            $VMlist = Get-VM -Location (Get-Folder 'Test Servers')
            $VMlist = Get-VM -Location (Get-DataCenter 'Austin')
            .EXAMPLE
            Get-VM -Location (Get-Folder 'Test Servers') | Reset-CBT -vCenter VCENTER.DOMAIN.LOCAL
            Similar to the previous example, except that it uses a pipeline for the list of virtual machines.
            .EXAMPLE
            Reset-CBT -VM 'WAHLNETWORK' -vCenter VCENTER.DOMAIN.LOCAL -EnableCBT
            Enables CBT for a VM named WAHLNETWORK. No other activities are performed. This is useful for when you want to enable CBT for one or more virtual machines.
    #>

    [CmdletBinding()]
    Param(
        [Parameter(Mandatory = $true,Position = 0,HelpMessage = 'Virtual Machine',ValueFromPipeline = $true)]
        [Alias('Name')]
        [ValidateNotNullorEmpty()]
        $VM,
        [Parameter(Mandatory = $true,Position = 1,HelpMessage = 'vCenter FQDN or IP address')]
        [ValidateNotNullorEmpty()]
        [String]$vCenter,
        [Parameter(Mandatory = $false,Position = 2,HelpMessage = 'Enables CBT for any VMs found with it disabled')]
        [ValidateNotNullorEmpty()]
        [Switch]$EnableCBT,
        [Parameter(Mandatory = $false,Position = 3,HelpMessage = 'Prevents usings snapshots from flushing the CBT file')]
        [ValidateNotNullorEmpty()]
        [Switch]$NoSnapshots
    )

    Process {

        Write-Verbose -Message 'Importing required modules and snapins'
        $powercli = Get-PSSnapin -Name VMware.VimAutomation.Core -Registered
        try 
        {
            switch ($powercli.Version.Major) {
                {
                    $_ -ge 6
                }
                {
                    Import-Module -Name VMware.VimAutomation.Core -ErrorAction Stop
                    Write-Verbose -Message 'PowerCLI 6+ module imported'
                }
                5
                {
                    Add-PSSnapin -Name VMware.VimAutomation.Core -ErrorAction Stop
                    Write-Warning -Message 'PowerCLI 5 snapin added; recommend upgrading your PowerCLI version'
                }
                default 
                {
                    throw 'This script requires PowerCLI version 5 or later'
                }
            }
        }
        catch 
        {
            throw $_
        }


        Write-Verbose -Message 'Ignoring self-signed SSL certificates for vCenter Server (optional)'
        $null = Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -DisplayDeprecationWarnings:$false -Scope User -Confirm:$false

        Write-Verbose -Message 'Connecting to vCenter'
        try 
        {
            $null = Connect-VIServer -Server $vCenter -ErrorAction Stop -Session ($global:DefaultVIServers | Where-Object -FilterScript {
                    $_.name -eq $vCenter
            }).sessionId
        }
        catch 
        {
            throw 'Could not connect to vCenter'
        }

        Write-Verbose -Message 'Gathering data on VM inventory'
        $fixvm = Get-VM $VM
        [array]$notfixedvm = $null

        Write-Verbose -Message 'Creating configuration specification'
        $vmconfigspec = New-Object -TypeName VMware.Vim.VirtualMachineConfigSpec
        
        Write-Verbose -Message 'Walking through VM inventory'
        foreach($_ in $fixvm)
        {
            if ($EnableCBT -ne $true -and $_.ExtensionData.Config.ChangeTrackingEnabled -eq $true -and $_.PowerState -eq 'PoweredOn' -and $_.ExtensionData.Snapshot -eq $null)
            {
                try 
                {
                    Write-Verbose -Message "Reconfiguring $($_.name) to disable CBT" -Verbose
                    $vmconfigspec.ChangeTrackingEnabled = $false
                    $_.ExtensionData.ReconfigVM($vmconfigspec)

                    if ($NoSnapshots -ne $true)
                    {
                        Write-Verbose -Message "Creating a snapshot on $($_.name) to clear CBT file" -Verbose
                        $null = New-Snapshot -VM $_ -Name 'CBT Cleanup'

                        Write-Verbose -Message "Removing snapshot on $($_.name)" -Verbose
                        $null = $_ |
                        Get-Snapshot |
                        Remove-Snapshot -RemoveChildren -Confirm:$false
                    }
                }
                catch 
                {
                    throw $_
                }
            }
            elseif ($EnableCBT -and $_.ExtensionData.Config.ChangeTrackingEnabled -eq $false)
            {
                Write-Verbose -Message "Reconfiguring $($_.name) to enable CBT" -Verbose
                $vmconfigspec.ChangeTrackingEnabled = $true
                $_.ExtensionData.ReconfigVM($vmconfigspec)
            }
            else 
            {
                if ($_.PowerState -ne 'PoweredOn' -and $EnableCBT -ne $true) 
                {
                    Write-Warning -Message "Skipping $_ - Not powered on"
                    $notfixedvm += $_
                }
                if ($_.ExtensionData.Snapshot -ne $null -and $EnableCBT -ne $true) 
                {
                    Write-Warning -Message "Skipping $_ - Snapshots found"
                    $notfixedvm += $_
                }
            }
        }

        if ($notfixedvm -ne $null)
        {
            Write-Warning -Message 'The following VMs were not altered'
            $notfixedvm | Format-Table -AutoSize
        }

    } # End of process
} # End of function
#### FUNCTIONS END ####
AddToLogFile;AddToLogFile;AddToLogFile;cls #Add Space in the log

$VBRJobObj =$null
$VBRRunningJobBackupSession = $null

AddToLogFile "Checking if Veeam Job Exists ($BackupJobName)"

$VBRJobObj = Get-VBRJob -Name "$BackupJobName"
if($VBRJobObj){
    AddToLogFile "Veeam Job Detected - '$BackupJobName'"
    AddToLogFile "Detecting if Running Job is running in FullMode..."
    $VBRRunningJobBackupSession = Get-VBRBackupSession | Where-Object {$_.State -like "Working" -and $_.Name -like "*$BackupJobName*" -and $_.IsRetryMode -eq $false} #| select Name,State,IsFullMode,IsRetryMode
    if($VBRRunningJobBackupSession){
        if($VBRRunningJobBackupSession.IsFullMode){
            AddToLogFile "Full/Active Full Backup Mode Detected for '$BackupJobName'."
            AddToLogFile "Getting VM Objects Inside Veeam Job - '$BackupJobName'"
            
            #Get VMs based on the VMs that were processed in the last backup.
            $VeeamJobVMs = $VBRJobObj.FindLastBackup().GetLastOibs()
            
            AddToLogFile "Starting CBT Reset..."
            #Looping through VMs in Veeam Backup Job
            foreach ($VM in $VeeamJobVMs) {
                $VMName = $VM.VmName #get VM Name From Veeam Job
                $VMvCenterNameFromVeeamJob = $VM.AuxData.HostName  #get vCenter Server From Veeam Job
                AddToLogFile "Resetting CBT for: $VMName (vCenter:$VMvCenterNameFromVeeamJob)"

                #Code to Reset CBT
                $CBTResetCMD = "Reset-CBT -VM $VMName -vCenter $VMvCenterNameFromVeeamJob"
                AddToLogFile $CBTResetCMD
                #Run the CBT Reset Command
                Invoke-Expression -Command $CBTResetCMD
            }
            AddToLogFile "CBT Reset Was Completed!"
        }
        else {
            AddToLogFile "Incremental/Reverse Incremental Backup Mode Detected for '$BackupJobName'."
            AddToLogFile "CBT will NOT be reset!."
        }
    }
    else{
        AddToLogFile "Running Backup Job Session was not found for '$BackupJobName'."
    }
}
else {AddToLogFile "Veeam Job Not Found"}

[tsightler]

Nice work! Thanks for sharing.

The name of the backup job should be the name of the job you are adding this to (i havent found a dynamic way to detect the job name yet).

So I saw this part and thought I'd share with you the following little magical code I wrote some time ago for exactly this use case. OK, so it's not that magical, it's actually more like a crude workaround, but it's been proven to work reliably in the real word in a number of environments I've worked with. I consider it a life-saver when you need to run pre/post-job tasks that require the job name and you have a bunch of jobs (when I originally wrote this the client I was working with had about 50 jobs).

I usually still allow passing in the job name with a parameter, since that makes troubleshooting the script from the cmdline a lot easier, so that's why the auto detection code is wrapped in conditional logic.

Code: Select all

if (-not($jobName)) {
    # Determine job name from calling Veeam.Backup.Manager process
    $parentPid = (Get-WmiObject Win32_Process -Filter "processid='$pid'").parentprocessid.ToString()
    $parentCmd = (Get-WmiObject Win32_Process -Filter "processid='$parentPid'").CommandLine
    $cmdArgs = $parentCmd.Replace('" "','","').Replace('"','').Split(',')
    $jobName = (Get-VBRJob | ? {$cmdArgs[4] -eq $_.Id.ToString()}).Name
}

if(-not($jobName)) { Throw "No -jobName parameter and could not be automatically determined." }

This code works by determining the parent process of the running Powershell environment which, when run as a pre/post-script, will be the Veeam.Backup.Manager process. This process includes the job UUID as part of the command line so this script grabs that command line, splits the arguments into an array, and then uses the extracted job UUID to find the matching job name. You could obviously just as easily grab the entire job object at that point as well.

[the_mentor]

tsightler,
thank you very much for the JobName detection code snippet I will work on incorporating it into my script.
I can think of a few different scenarios in which it can be handy.
It will definitely be a nice thing if Veeam will inject a environment variables into the powershell session that it generates.
These environment variables can have stuff like JobName,Repo,vCenter server and so on and so on.
VMware SRM does that for for the powershell scripts that it can execute.

[tsightler]

It will definitely be a nice thing if Veeam will inject a environment variables into the powershell session that it generates.
These environment variables can have stuff like JobName,Repo,vCenter server and so on and so on.
VMware SRM does that for for the powershell scripts that it can execute.

Agree, I've asked for similar things myself. Actually I just wanted the option to be able to pass those things as variables, but I agree that simply including them in the Powershell environment is an even better idea.

[veremin]

Yep, we've already tracked this request - ability to pass different variables to pre/post job activities. Chances are, it will be implemented in one of the next product releases. Thanks.