How to find orphaned backup files in cloud connect SOBR?

[EdgarRicharte]

Hello I've recently found some old backup files that have been replaced with more recent backup set within our SOBR. But they are still in our SOBR taking up storage. Anyone have experience finding those old backups to remove them and free up some space. Below is how my SOBR shows our stats. As you can see it is off and doesn't add up. I could go through each tenant and find the old backups but we are an MSP and that would take a long time since we have a lot of tenants. I was hoping someone would have a better solution. Thank you for any advice in advanced!


----Capacity-----Free------Used Space
1.--63.5TB-------6TB--------51.6TB
2.--22.5TB-------4.7TB------17.7TB
3.--63.5TB-------4.6TB------50.4TB
4.--22.5TB-------1.2TB------14.7TB
5.--63.5TB-------3.7TB------53.7TB

PS: Our backup policy is set to do 3 restore points. With 7 days of insider protection. Also all repos are formatted with ReFS on Windows Server 2019 v1809 fully updated.

[HannesK]

Hello,
the main question is, why they became orphaned. I'm trying to investigate that topic since months, but I cannot find any customer where it happened because of a bug. Everyone so far did manual "optimizations"...

Would it be possible that you can check with support and try to find out why you have orphaned files?

Thanks,
Hannes

[EdgarRicharte]

We did actually do a bunch of backup evacuations. Since we had to shift a lot of data around. And consolidated from 10 repos down to 5 within our SOBR. I just attributed it to that. The consolidation literally took months. I just attributed the orphaned backups to that.

[HannesK]

hmm okay. I don't know, whether such script exists in public, but you could read all existing backup files from the SQL database and compare that with the file names on your repositories.

If you like, I can move the thread to the public PowerShell forum.

[EdgarRicharte]

That would be great HannesK! Please move the thread. tyvm!

[david.domask]

Hey Edgar,

Hannes poked me on this and I took a crack at it. Works in my lab, but I'm sure there are some rough-edges I didn't consider, but it should give you a list of the backups Veeam is aware of on the provider's side.

Few notes and disclaimers though:

[*]100% Unsupported (especially since I had to cheat and use reflections to get the backups). I'll do my best to respond on 'best-effort', but let's see if it works out for you
[*]This is meant for Providers only due to how the Provider DBs/data is stored. Existing methods on the forums will work for normal VBR installations.
[*]Right now it's oriented exclusively at SOBR.
[*]Likely, any Dedup Appliance (Storeonce, DataDomain, etc) won't give "good" data. It should give the file path, but I'm not sure how pretty it will look
[*]If you want to add a property for just the Backup Name, for the line $StoragePathsandBlocksize += New-Object in the -Property, add to the end StorageName=$Storage.filepath. This will add a property you can also sort by in $allStorages
[*]For $backupsonSobr, the method has a few flags it's looking for. Most you can ignore, but bool skipImported is the first "True" value. I'm not as familiar with Provider Installations and if you can have Imported Backups or not, but if you can, you might want to change that to "False" and I suppose it should grab these backups.

Hope it helps!

Code: Select all

function Get-StoragesPathsAndTentantPathFromBackup {
    param(
        [Parameter(Mandatory=$true, Position=0)]
        [Object[]]$Backup
    )
    $Storages = $Backup[0].GetallChildrenStorages() | Sort-Object -Property PartialPath, CreationTime -Descending
    $BackupPaths = $Backup[0].GetPartialPathWithTenantFolder()
    $Repository = $Backup.FindRepository()[0]
    $StoragePathsandBlocksize  = @()
    if($Repository.Type -eq "ExtendableRepository"){
        foreach($Storage in $Storages){
            $Extent = $Repository.FindExtentRepo($Storage.Id)
            $StoragePathsandBlocksize += New-Object -TypeName psobject -Property @{Tenant=$BackupPaths.Elements[0];Extent=$Extent.Name;Path=$($Extent.Path.ToString(),$BackupPaths.Elements[0],$BackupPaths.Elements[1],$Storage.filepath -join "\");CreationTime=$Storage.CreationTime}
            }
    } else {
        $StoragePathsandBlocksize += $Storages | Sort-Object -Property PartialPath, CreationTime -Descending | Select-Object -Property PartialPath,BlockAlignmentSize
    }
    return $StoragePathsandBlocksize
}

$repo = Get-VBRBackupRepository -Scaleout -Name 'Name of SOBR'
$backupsOnSobr = [Veeam.backup.Core.CBackup]::GetInRepository($($repo.id),"True","True","False","True")
$allStoragesOnSobr = @()
Foreach($backup in $backupsOnSobr){
$allStoragesOnSobr += Get-StoragesPathsAndTentantPathFromBackup -backup $backup
}
$allStoragesOnSobr | select-Object -Property Tenant, Path

[EdgarRicharte]

Hey David,
So the only thing I changed from the script was "Name of SOBR'. When I ran it I mostly received errors. But did have a handful of tenants show up. Is there anything else I should fill out? Also the files that got listed are those files that are in the SOBR but veeam does not recognize them?

[david.domask]

Hi Edgar,

Interesting.

Can you PM me the output? Let's continue the discussion there.

[dave.moreland]

Hi David,
Sorry I'm a bit late to this discussion but this did help me and I wanted to share what I found and the change I had to make.

We have had some tenants notice that their usage keeps climbing which prompted some file audits of our SOBR by hand. Audits of 3 tenants found that 2 had orphaned files so we wanted a way to audit the SOBRs programmatically to see how deep the rabbit hole is. Some searching landed me here.

When I first ran the script against my v11 Cloud Connect I was getting the following error:
Method Invocation failed because [Veeam.Backup.Core.CExtendableRepository] does not contain a method named 'FindExtentRepo'

I ran a get-member -inputobject $Repository and was not able to find a method named FindExtentRepo, However I did find a method named FindRepositoryForExistingStorage which accepts a storageid guid. This could be because at the time of this posting its more likely that v10 was in use. After changing the method name with the one I found, I was able to get the expected output from the script.

Based on my findings I would like to propose the following change to the script:
original lime from script:

Code: Select all

$Extent = $Repository.FindExtentRepo($Storage.Id)

proposed change:
$Extent = $Repository.FindRepositoryForExistingStorage($Storage.Id)

Thank you very much David for all your time and effort, it is much appreciated!

Regards,
Dave

[SasoB]

Hi,

If anyone else will try it.
In my case variable $StoragePathsandBlocksize was not created as it should, because PATH had more than two Elements ($BackupPaths.Elements[0], $BackupPaths.Elements[1],...)

Code: Select all

$StoragePathsandBlocksize += New-Object -TypeName psobject -Property @{Tenant=$BackupPaths.Elements[0];Extent=$Extent.Name;Path=$($Extent.Path.ToString(),$BackupPaths.Elements[0],$BackupPaths.Elements[1],$Storage.filepath -join "\");CreationTime=$Storage.CreationTime}

so I changed it to

Code: Select all

$StoragePathsandBlocksize += New-Object -TypeName psobject -Property @{Tenant=$BackupPaths.Elements[0];Extent=$Extent.Name;Path=$($Extent.Path.ToString() +""+ ($BackupPaths.Elements -join "\") +"\"+$Storage.FilePath);CreationTime=$Storage.CreationTime}

But script helped a lot indeed.

Thank you