How to find stale/orphaned backups

[Mgamerz]

I am using VAW (this is not specific to VAW) to backup my laptop/workstation endpoints. I have it linked to my OU in Active Directory so it auto deploys the agents to those systems. I have some users that I have to make temporary systems for - if their hardware dies and they need a temporary loaner, if they are going international and are not permitted to take their main laptop, etc. Since I have a protection group linked to my OU in the VBR console, these new test systems are automatically added and create backups. Once the need for the system disappears these systems are wiped and obviously no more backups are taken. Sometimes I have to take a backup of a system after re-formatting it just to have it die again (our full disk encryption software we are forced to use is terrible), so it makes a new full backup of the same hostname.

From my understanding, VAW backups are stored by hostname-guid folder structure, so a fresh install of windows with same hostname will essentially create an entirely new backup chain. Is there a way to easily find these orphaned old chains? I am running fairly low on disk space in the repo my backups are going to.

[Shestakov]

Hello,
Could you clarify the criteria of the backup chains you are looking for?
Is that backup of computer that is not in the protection group anymore or backup chain that is outdated because the computer has a new backup chain?
Thanks!

[Mgamerz]

They are chains that are essentially no longer used.

Scenario: I link protection group to my OU which has Computer1 in it.

OU
|- Computer1

This system backs up. Eventually something goes terribly wrong and it dies. I have to reformat it, or I purchase a new computer for the user (I am not restoring an image from veeam to it).

I name the new installation Computer1. However, as this is a new instance of windows, veeam assigns it a new GUID. (edit: It seems it's not a new GUID, but the chain is still different. Not sure if GUID is per hostname or per motherboard/chassis itself)

I now have backup chains for Computer1 (pre reformat) and Computer1 (post reformat). I can tell this by looking in my repository folders and seeing workstation backups being created there.

How do I find the old backups for Computer1 (pre reformat)? They do not show up in the UI.

Here is an example. I have an orphaned backup chain that is using 200+GB that that I assume is not included in retention anymore as it's on a separate chain. I understand keeping them around might be useful for retention so you don't want to auto delete them but it would be nice to have a way to find these so they don't just take up space if thy're not actually useful anymore.

In UI:


On disk:


I was able to manually find this one because I know this system specifically as it was a huge PITA to deal with when it died multiple times within a few days of each other. For my other less anger inducing machines I feel it will be very easy to forget about them.

[Shestakov]

Thanks for the explanation.

We have a Veeam ONE report automating such a search for VMs, but not for computers backed up by agents.
So technically it should notify you if some server or workstation was backed periodically, but now is not in any of backup jobs/policies?
What if you started to protect laptop not with central management, but locally?

As a workaround for your case I would consider Veeam ONE reports Agent Backup Job and Policy History and Computer Backup Status. They can show when amount of job/policy objects changed and tell which computers stopped being backed up.

[Vek17]

So you are looking for backups that exist in the filesystem, but are for whatever reason not known to Veeam? If that is the case I have a powershell command I run for it. It makes some assumptions regarding how it connects to windows repositories and is not something I'd consider general purpose enough to use in any environment, but it is a decent base.

Code: Select all

using namespace System.Security.Cryptography
using namespace System.Text

function Invoke-VeeamSQL{
    [CmdletBinding()]
    param(
        [Parameter(
            Mandatory=$True, 
            ValueFromPipeline=$True,
            ValueFromPipelineByPropertyName=$True
        )]
        [String]$SQLQuery
    )
    BEGIN{
        $VeeamSQLServer   = Get-ItemPropertyValue -Path 'HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication\' -Name 'SqlServerName'
        $VeeamSQLDatabase = Get-ItemPropertyValue -Path 'HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication\' -Name 'SqlDatabaseName'
        $VeeamSQLInstace  = Get-ItemPropertyValue -Path 'HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication\' -Name 'SqlInstanceName'
        $Result = [System.Collections.ArrayList]::new()
        Write-Verbose "SqlServerName: $($VeeamSQLServer)"
        Write-Verbose "SqlDatabaseName: $($VeeamSQLDatabase)"
        Write-Verbose "SqlInstanceName: $($VeeamSQLInstace)"
    }
    PROCESS{
        Write-Verbose $SQLQuery
        $SQLConn = [System.Data.SqlClient.SqlConnection]::New()
        $SQLConn.ConnectionString = "Server=$VeeamSQLServer\$VeeamSQLInstace;Database=$VeeamSQLDatabase;Integrated Security=True"
        $SQLConn.Open()
        $SQLCmd = [System.Data.SqlClient.SqlCommand]::New()
        $SQLCmd.Connection  = $SQLConn
        $SQLCmd.CommandText = $SQLQuery
        $SQLReader  = $SQLCmd.ExecuteReader()
        $SQLColumns = $SQLReader.GetSchemaTable().ColumnName
        while($SQLReader.Read()){ 
            $Temp = [Ordered]@{}
            foreach($Column in $SQLColumns){
                $Temp.add($Column,$SQLReader[$Column])
            }
            $Result.Add([PSCustomObject]$Temp) > $null
        }
        $SQLConn.Close()
    }
    END{
        return $Result.ToArray()
    }
}

function Unprotect-VeeamCredential {
    [CmdletBinding()]
    param(
        [Parameter(Position=0, ValueFromPipeline=$true, Mandatory=$True)]
        [Veeam.Backup.PowerShell.Infos.CInternalCredentials]$Credential
    )
    BEGIN{
        $Credentials = @()
        function UnProtect-String {
            param (
                [Parameter(Position=0, ValueFromPipeline=$true, Mandatory=$True)]
                [ValidateNotNullOrEmpty()]
                [String] $encrypted
            )
            $byteData = [System.Convert]::FromBase64String($encrypted)
            return [System.Text.Encoding]::UTF8.GetString([ProtectedData]::Unprotect($byteData, $null, [DataProtectionScope]::LocalMachine))
        }
    }
    PROCESS{
        foreach($Cred in $Credential){
            $SQLResult = Invoke-VeeamSQL -SQLQuery "SELECT id,user_name,password,description FROM [Credentials] WHERE id = '$($Cred.id)'"
            $Credentials += [PSCustomObject]@{
                id              = $SQLResult.id
                UserName        = $SQLResult.user_name
                Password        = [RaderVeeam.Security.Credentials]::DecryptVeeamPassword($SQLResult.Password)
                Description     = $SQLResult.description
            }
        }
    }
    END{
        return $Credentials
    }
}

function Find-VeeamUntrackedFiles {
    [CmdletBinding()]
    Param(
    )
    BEGIN{
        $allRepos    = @(
            Get-VBRBackupRepository
            (Get-VBRBackupRepository -ScaleOut).Extent.Repository
        )
        $StorageMap = @{}

        foreach($Backup in [Veeam.Backup.Core.CBackup]::GetAll()){
            if(!$StorageMap.ContainsKey([String]$Backup.MetaFileName)){
                $StorageMap.Add([String]$Backup.MetaFileName,$Backup)
            }
            foreach ($Storage in $Backup.GetAllStorages()){
                if(!$StorageMap.ContainsKey($Storage.PartialPath.Elements[0])){
                    $StorageMap.Add($Storage.PartialPath.Elements[0],$Storage)
                }
            }
        }
    }
    PROCESS{
        $targetRepo     = $allRepos | Out-GridView -OutputMode Multiple
        $hosts          = Get-VBRServer
        $ErrorActionPreference = "Stop"
        foreach($Repo in $targetRepo){
            try{
                switch($Repo.Type){
                    WinLocal {
                        $vbrhost = $hosts | Where-Object id -eq $Repo.hostid
                        Get-ChildItem -Path "\\$($vbrhost.Info.HostInstanceId)\$($Repo.Path.ToString() -replace ':','$')" -Recurse -File -Include "*.vib","*.vrb","*.vbk","*.vbm" -Exclude "*.erm*" | Where-Object {!$StorageMap.ContainsKey($_.Name)}
                    }
                    CifsShare {
                        $Cred = Get-VBRCredentials | Where-Object {$_.id -eq $Repo.ShareCredsId}
                        $Args = @{
                            Name = $Repo.id
                            PSProvider = "FileSystem"
                            Root = $Repo.Path.ToString()
                        }
                        if($Cred){
                            $Cred = $Cred | Unprotect-VeeamCredential
                            $Args.Add("Credential",[PSCredential]::new($Cred.UserName, $($Cred.Password | ConvertTo-SecureString -AsPlainText -force)))
                        }
                        New-PSDrive @Args > $Null
                        Get-ChildItem -Path "$($Repo.id):" -Recurse -File -Include "*.vib","*.vrb","*.vbk","*.vbm" -Exclude "*.erm*","index_data.vbk","*.meta.vbk" | Where-Object {!$StorageMap.ContainsKey($_.Name)}  
                    }
                    default {
                        Write-Warning "This repo type is not supported: $($Repo.Type)"
                    }
                }
            }catch{
                Write-Warning "Failed to access repo: $($Repo.Name)"
            }
        }
    }
    END{
    }
}

If you are looking for Veeam backups that have no job associated but ARE known to Veeam then that is much easier with something as simple as: Get-VBRBackup | ? jobID -eq ([GUID]::Empty)

Those typically show up as imported however.

[Shestakov]

I guess it's neither of two cases.
Backups are known and associated with policy, but the computer doesn't exist anymore.

[Mgamerz]

The backup is "associated" with a job... kind of. It is assigned the same guid and has the same hostname. However it is a separate backup chain and the UI doesn't seem to know about it. I am OK with it being a separate backup chain (it is a "different" system, from a technical perspective).

This is very easy to reproduce.

1) Backup a system using veeam agent in unmanaged mode (i have mine set up from the VBR console to push agent and default policy).
2) Let it backup several times
3) Reformat the computer that is being backed up. Give it the same hostname (I do not want to deal with the huge mess that would be DNS otherwise).
4) Let Veeam back it up. Mine does this automatically through it's policy - it scans all hosts, sees this one without backup agent, pushes it, pushes default backup policy, system backs up.
5) If you look at my data above, only backups that are made with chain from step 4) are show nin the UI. Backups from step 1-2 don't show up and are effectively hidden from the user in the backup UI. I can understand some technical limitations but tools to find these orphaned chains would be helpful when reclaiming unused space.

I am not sure if this affects VM/host style backups. I don't often reformat my VMs (:

[Shestakov]

Got it.
You need a method to find backups of removed agents. I'll think about a creation of corresponding Veeam ONE report.
By the way, have you use Retention Policy for Outdated Backups?
Thanks