KB4835

[jasonede]

I've put this together and trying to work through my repositories. So far it seems to be working. It checks if a job is running then waits for it to finish before doing integrity check and repair. The job is then run and the check at the end carried out. It creates a log file for each job and a transcript in case it crashes.

Code: Select all

#Requires -Version 5.1
<#
.SYNOPSIS
    Identifies and repairs corrupted files in a Veeam Backup for Microsoft 365
    object-storage repository, following the procedure in KB4874.

.PARAMETER JobName
    Name of the Veeam backup job whose repository will be repaired.
    If omitted, the script prompts for it interactively.

.PARAMETER StartFromStep
    Step number to resume from (1-6). Default is 1 (full run).
    Use this to re-run from a specific step after a failure:
      1 - Full run (integrity check through final verification)
      2 - Re-parse integrity check results (re-uses last Test-VBORepository output)
      3 - Skip integrity check; go straight to repair session
      4 - Skip repair session; just re-enable the job
      5 - Skip repair session and re-enable; just run the backup job
      6 - Skip to final verification only
    Note: When starting from Step 3 or lower, the job is disabled first.
    When starting from Step 4 or higher, the job is assumed to be already enabled.

.NOTES
    Applies to: Veeam Backup for Microsoft 365 v8.5
    Must be run from an elevated (Run as Administrator) PowerShell window.
#>

[CmdletBinding()]
param(
    [Parameter(Position = 0)]
    [string]$JobName,

    [Parameter()]
    [ValidateRange(1, 6)]
    [int]$StartFromStep = 1
)

# ==============================================================================
# Configuration -- adjust these as needed
# ==============================================================================
$TopSitesCount          = 10    # How many top-affected sites to display after Step 2
$PollIntervalSeconds    = 30    # Polling interval for repair session and backup job
$PostEnableDelaySeconds = 60    # Seconds to wait after re-enabling the job before starting it
$ModulePath             = "C:\Program Files\Veeam\Backup365\Veeam.Archiver.PowerShell.dll"


# ==============================================================================
# Helper functions
# ==============================================================================

function Test-IsAdmin {
    $id = [Security.Principal.WindowsIdentity]::GetCurrent()
    $p  = New-Object Security.Principal.WindowsPrincipal($id)
    return $p.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
}

function Get-SafeFileName ([string]$Name) {
    foreach ($c in [IO.Path]::GetInvalidFileNameChars()) {
        $Name = $Name.Replace([string]$c, '_')
    }
    return $Name
}

function Write-Log {
    param(
        [string]$Message,
        [ValidateSet("INFO","WARN","ERROR")]
        [string]$Level = "INFO"
    )
    $ts   = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
    $line = "[$ts] [$Level] $Message"
    Write-Host $line
    Add-Content -Path $script:LogFile -Value $line
}

function Write-Section ([string]$Title) {
    $bar   = "=" * 80
    $block = @("", $bar, "  $Title", $bar)
    foreach ($line in $block) {
        Write-Host $line
        Add-Content -Path $script:LogFile -Value $line
    }
}

function Write-TableToScreenAndLog ([object[]]$Data, [string[]]$Properties = @()) {
    $table = if ($Properties.Count -gt 0) {
        $Data | Format-Table $Properties -AutoSize -Wrap | Out-String -Width 600
    } else {
        $Data | Format-Table -AutoSize -Wrap | Out-String -Width 600
    }
    Write-Host $table
    Add-Content -Path $script:LogFile -Value $table
}

# Called on all exit paths -- re-enables the job if this script disabled it
function Exit-Script ([int]$Code = 0) {
    if ($null -ne $script:DisabledJob) {
        Write-Log "Re-enabling job '$($script:DisabledJob.Name)' before exit..."
        try {
            Enable-VBOJob -Job $script:DisabledJob -ErrorAction SilentlyContinue | Out-Null
            Write-Log "Job re-enabled."
        } catch {
            Write-Log "Could not re-enable job automatically. Please re-enable manually in the Veeam console: $($script:DisabledJob.Name)" -Level "WARN"
        }
        $script:DisabledJob = $null
    }
    Write-Log "Script exiting with code $Code."
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit $Code
}

$script:DisabledJob = $null
$script:LogFile     = $null   # set below once we have the job name


# ==============================================================================
# Job name -- parameter or interactive prompt
# ==============================================================================
if ([string]::IsNullOrWhiteSpace($JobName)) {
    $JobName = Read-Host "Enter the Veeam backup job name"
}
if ([string]::IsNullOrWhiteSpace($JobName)) {
    Write-Host "[ERROR] No job name provided. Exiting."
    exit 1
}


# ==============================================================================
# Log and transcript setup
# ==============================================================================
$SafeName        = Get-SafeFileName $JobName
$CurrentDir      = (Get-Location).Path
$script:LogFile  = Join-Path $CurrentDir "${SafeName}.log"
$TranscriptFile  = Join-Path $CurrentDir "Transcript_${SafeName}.log"

# Transcript is overwritten on each run
Start-Transcript -Path $TranscriptFile -Force | Out-Null

# Append a run-start separator to the log
Add-Content -Path $script:LogFile -Value ""
Add-Content -Path $script:LogFile -Value ("*" * 80)
Add-Content -Path $script:LogFile -Value ("*  NEW RUN -- $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')  StartFromStep=$StartFromStep")
Add-Content -Path $script:LogFile -Value ("*" * 80)

Write-Section "Veeam VBO365 Repository Repair (KB4874) -- $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"
Write-Log "Job name:              $JobName"
Write-Log "Starting from step:    $StartFromStep"
Write-Log "Log file:              $($script:LogFile)"
Write-Log "Transcript:            $TranscriptFile"
Write-Log "Running as admin:      $(Test-IsAdmin)"
Write-Log "Top sites to display:  $TopSitesCount"
Write-Log "Post-enable delay:     ${PostEnableDelaySeconds}s"


# ==============================================================================
# Prerequisites -- module
# ==============================================================================
Write-Section "Checking Prerequisites"

if (-not (Test-Path $ModulePath)) {
    Write-Log "Veeam PowerShell module not found at: $ModulePath" -Level "ERROR"
    Write-Log "Ensure Veeam Backup for Microsoft 365 v8.5 is installed." -Level "ERROR"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 1
}

try {
    Import-Module $ModulePath -ErrorAction Stop
    Write-Log "Veeam PowerShell module imported successfully."
} catch {
    Write-Log "Failed to import Veeam module: $($_.Exception.Message)" -Level "ERROR"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 1
}


# ==============================================================================
# Locate the job
# ==============================================================================
Write-Section "Locating Backup Job"

$job = $null
try {
    $job = Get-VBOJob -Name $JobName -ErrorAction Stop
} catch {
    Write-Log "Could not find a job named '$JobName': $($_.Exception.Message)" -Level "ERROR"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 1
}
Write-Log "Found job: $($job.Name)"


# ==============================================================================
# Check job scope -- must include OneDrive, SharePoint, or Teams
# ==============================================================================
Write-Section "Checking Job Scope"

$items = $null
try {
    $items = @(Get-VBOBackupItem -Job $job -ErrorAction Stop)
} catch {
    Write-Log "Could not retrieve backup items for job '$JobName': $($_.Exception.Message)" -Level "ERROR"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 1
}

$hasOneDrive   = ($items | Where-Object { $_.OneDrive } | Measure-Object).Count -gt 0
$hasSharePoint = ($items | Where-Object { $_.Sites }    | Measure-Object).Count -gt 0
$hasTeams      = ($items | Where-Object { $_.Teams }    | Measure-Object).Count -gt 0

Write-Log "Job scope -- OneDrive: $hasOneDrive  |  SharePoint: $hasSharePoint  |  Teams: $hasTeams"

if (-not ($hasOneDrive -or $hasSharePoint -or $hasTeams)) {
    Write-Log "Job '$JobName' does not back up OneDrive, SharePoint, or Teams workloads." -Level "WARN"
    Write-Log "KB4874 remediation applies only to those workloads. No action taken." -Level "WARN"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 0
}
Write-Log "Job covers at least one applicable workload. Proceeding."


# ==============================================================================
# Get repository from job
# ==============================================================================
$repository = $job.Repository
if ($null -eq $repository) {
    Write-Log "Could not determine the repository from the job object. Verify the job has an assigned repository." -Level "ERROR"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 1
}
Write-Log "Repository: $($repository.Name)"


# ==============================================================================
# Pre-Step: Disable the job (only when running Steps 1-3)
# ==============================================================================
if ($StartFromStep -le 3) {
    Write-Section "Pre-Step: Disabling Backup Job"

    # Wait for any currently-running session to finish before disabling
    $activeSession = Get-VBOJobSession -Job $job -Last
    $jobRunningStatuses = @("Running", "Queued", "Updating")

    if ($null -ne $activeSession -and $activeSession.Status -in $jobRunningStatuses) {
        Write-Log "Job '$($job.Name)' is currently running (Status=$($activeSession.Status)). Waiting for it to finish before disabling..."
        Write-Host ""

        $firstPoll     = $true
        $pollLineCount = 0

        while ($true) {
            $activeSession = Get-VBOJobSession -Job $job -Last
            $timestamp     = Get-Date -Format "yyyy-MM-dd HH:mm:ss"

            $displayLines = @(
                "Timestamp  : $timestamp",
                "Job        : $($activeSession.JobName)",
                "Status     : $($activeSession.Status)",
                "Sync Type  : $($activeSession.JobSessionConfigType)"
            )

            if (-not $firstPoll -and $pollLineCount -gt 0) {
                try {
                    [System.Console]::SetCursorPosition(0, [System.Console]::CursorTop - $pollLineCount)
                } catch { }
            }
            foreach ($line in $displayLines) { Write-Host $line }
            $pollLineCount = $displayLines.Count
            $firstPoll     = $false

            Add-Content -Path $script:LogFile -Value "[$timestamp] [POLL] Waiting for running job -- Status=$($activeSession.Status)"

            if ($activeSession.Status -notin $jobRunningStatuses) {
                Write-Host ""
                Write-Log "Job session finished with status '$($activeSession.Status)'. Proceeding to disable."
                break
            }

            Start-Sleep -Seconds $PollIntervalSeconds
        }
    } else {
        Write-Log "Job '$($job.Name)' is not currently running. Proceeding to disable."
    }

    Write-Log "Disabling job '$($job.Name)' to prevent scheduled runs during repair..."

    try {
        Disable-VBOJob -Job $job -ErrorAction Stop | Out-Null
        $script:DisabledJob = $job
        Write-Log "Job disabled."
    } catch {
        Write-Log "Failed to disable job: $($_.Exception.Message)" -Level "ERROR"
        Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
        exit 1
    }
} else {
    Write-Log "Pre-Step: Job disable skipped (StartFromStep=$StartFromStep -- job assumed already enabled)."
}


# ==============================================================================
# Steps 1 and 2: Integrity check and review results
# ==============================================================================
if ($StartFromStep -le 2) {

    # ------------------------------------------------------------------
    # Step 1: Integrity check
    # ------------------------------------------------------------------
    Write-Section "Step 1: Running Integrity Check (Test-VBORepository)"
    Write-Log "This may take several hours depending on repository size."

    if (-not (Test-IsAdmin)) {
        Write-Log "WARNING: PowerShell is NOT running as administrator. If Test-VBORepository fails, close this session and re-run the script from an elevated (Run as Administrator) PowerShell window." -Level "WARN"
    }

    Write-Log "Running Test-VBORepository against '$($repository.Name)'..."

    $integrityErrors = @()
    try {
        Test-VBORepository -Repository $repository -ErrorVariable integrityErrors -ErrorAction SilentlyContinue | Out-Null
    } catch {
        $integrityErrors += $_
    }

    # Check whether the failure looks privilege-related
    if ($integrityErrors.Count -gt 0) {
        $combinedMessages = ($integrityErrors | ForEach-Object { "$_" }) -join " "
        if ((-not (Test-IsAdmin)) -and ($combinedMessages -match 'access.denied|denied|privilege|administrator|elevation|unauthorized')) {
            Write-Log "Test-VBORepository appears to have failed because the session is not elevated." -Level "ERROR"
            Write-Log "Please close this window, re-open PowerShell using 'Run as Administrator', and run the script again." -Level "ERROR"
            Exit-Script 1
        }
    }

    Write-Log "Integrity check complete. Reviewing results..."

    # ------------------------------------------------------------------
    # Step 2: Parse and review results
    # ------------------------------------------------------------------
    Write-Section "Step 2: Reviewing Integrity Check Results"

    $corruptedSites = [System.Collections.Generic.List[PSCustomObject]]::new()
    $corruptedLists = [System.Collections.Generic.List[PSCustomObject]]::new()

    foreach ($errObj in $integrityErrors) {
        $msg = "$errObj"

        if ($msg -match "The following site has corrupted file data:\s*(.+?)\s*\(.*?count:\s*(\d+)\)") {
            $siteUrl   = $Matches[1].Trim()
            $siteCount = [int]$Matches[2]
            $corruptedSites.Add([PSCustomObject]@{ Type = "Site"; Path = $siteUrl; Count = $siteCount })
        }

        if ($msg -match "The following list contains items referencing corrupted files:\s*(.+?)\s*\(.*?count:\s*(\d+)\)") {
            $listPath  = $Matches[1].Trim()
            $listCount = [int]$Matches[2]
            $corruptedLists.Add([PSCustomObject]@{ Type = "List"; Path = $listPath; Count = $listCount })
        }
    }

    $corruptionFound = ($corruptedSites.Count -gt 0) -or ($corruptedLists.Count -gt 0)

    if (-not $corruptionFound) {
        Write-Log "No corruption detected in repository '$($repository.Name)'."
        Write-Log "No further action is required."
        Exit-Script 0
    }

    $totalSiteFiles = ($corruptedSites | Measure-Object -Property Count -Sum).Sum
    $totalListItems = ($corruptedLists | Measure-Object -Property Count -Sum).Sum
    $totalAffected  = $totalSiteFiles + $totalListItems

    Write-Log "Corruption detected."
    Write-Log "  Corrupted sites:      $($corruptedSites.Count) site(s) -- $totalSiteFiles affected file(s)"
    Write-Log "  Corrupted lists:      $($corruptedLists.Count) list(s) -- $totalListItems affected item(s)"
    Write-Log "  Total affected items: $totalAffected"

    $allAffected = [System.Collections.Generic.List[PSCustomObject]]::new()
    foreach ($s in $corruptedSites) { $allAffected.Add($s) }
    foreach ($l in $corruptedLists) { $allAffected.Add($l) }

    $topAffected = @($allAffected | Sort-Object Count -Descending | Select-Object -First $TopSitesCount)

    $topHeader = "Top $TopSitesCount most-affected sites/locations (by corrupted item count):"
    Write-Host ""
    Write-Host $topHeader
    Add-Content -Path $script:LogFile -Value ""
    Add-Content -Path $script:LogFile -Value $topHeader

    Write-TableToScreenAndLog $topAffected -Properties @("Type","Count","Path")

} else {
    Write-Log "Steps 1-2: Integrity check skipped (StartFromStep=$StartFromStep)."
}


# ==============================================================================
# Step 3: Start repair session
# ==============================================================================
if ($StartFromStep -le 3) {

    Write-Section "Step 3: Scanning Repository and Flagging Affected Items"
    Write-Log "Starting full repair session on repository '$($repository.Name)'..."
    Write-Log "Note: The repository is locked for all backups and restores during this step."
    Write-Log "      Ensure all backup and restore sessions for this repository are closed."

    $repairSession = $null
    try {
        $repairSession = Start-VBORepositoryRepairSession `
            -Repository $repository `
            -Type CleanMissingFilesData `
            -Mode Full `
            -Scope Full `
            -ErrorAction Stop
        Write-Log "Repair session started. Session ID: $($repairSession.Id)"
    } catch {
        Write-Log "Failed to start repair session: $($_.Exception.Message)" -Level "ERROR"
        Exit-Script 1
    }

    Write-Log "Polling repair session every $PollIntervalSeconds seconds. Waiting for completion..."
    Write-Host ""

    # Non-terminal states -- session is still starting up or running
    $repairNonTerminal = @("Running", "Preparing", "Queued")

    $firstPoll     = $true
    $pollLineCount = 0

    while ($true) {
        $sessionState = Get-VBORepositoryRepairSession -Id $repairSession.Id
        $timestamp    = Get-Date -Format "yyyy-MM-dd HH:mm:ss"

        $displayLines = @(
            "Timestamp  : $timestamp",
            "Id         : $($sessionState.Id)",
            "Type       : $($sessionState.Type)",
            "Repository : $($sessionState.RepositoryId)",
            "State      : $($sessionState.State)",
            "Error      : $($sessionState.ErrorMessage)"
        )

        if (-not $firstPoll -and $pollLineCount -gt 0) {
            try {
                [System.Console]::SetCursorPosition(0, [System.Console]::CursorTop - $pollLineCount)
            } catch { }
        }
        foreach ($line in $displayLines) { Write-Host $line }
        $pollLineCount = $displayLines.Count
        $firstPoll     = $false

        Add-Content -Path $script:LogFile -Value "[$timestamp] [POLL] Repair -- State=$($sessionState.State)  Error=$($sessionState.ErrorMessage)"

        if ($sessionState.State -notin $repairNonTerminal) {
            Write-Host ""
            if ($sessionState.State -eq "Finished" -and [string]::IsNullOrEmpty($sessionState.ErrorMessage)) {
                Write-Log "Repair session completed successfully."
            } else {
                Write-Log "Repair session ended with state '$($sessionState.State)'." -Level "WARN"
                if (-not [string]::IsNullOrEmpty($sessionState.ErrorMessage)) {
                    Write-Log "Session error: $($sessionState.ErrorMessage)" -Level "ERROR"
                    Write-Log "Stopping -- repair session reported an error. Review the error above before retrying." -Level "ERROR"
                    Exit-Script 1
                }
            }
            break
        }

        Start-Sleep -Seconds $PollIntervalSeconds
    }

} else {
    Write-Log "Step 3: Repair session skipped (StartFromStep=$StartFromStep)."
}


# ==============================================================================
# Step 4: Re-enable the job
# ==============================================================================
Write-Section "Step 4: Re-enabling Backup Job"

# Decide whether the job needs enabling. Re-enable if this run disabled it, or if
# the job is currently found disabled (e.g. left disabled by an aborted prior run).
$disabledByThisRun = ($null -ne $script:DisabledJob)
$needsEnable       = $disabledByThisRun

if (-not $disabledByThisRun) {
    # Re-query current state -- IsEnabled on the object fetched at start may be stale
    try {
        $currentJobState = Get-VBOJob -Name $JobName -ErrorAction Stop
        if (-not $currentJobState.IsEnabled) {
            $needsEnable = $true
            Write-Log "Job '$($job.Name)' is currently disabled (not by this run). It will be re-enabled."
        }
    } catch {
        Write-Log "Could not re-query job state to check if enabled: $($_.Exception.Message)" -Level "WARN"
    }
}

if ($needsEnable) {
    Write-Log "Re-enabling job '$($job.Name)'..."
    try {
        Enable-VBOJob -Job $job -ErrorAction Stop | Out-Null
        $script:DisabledJob = $null
        Write-Log "Job re-enabled successfully."
    } catch {
        Write-Log "Failed to re-enable job: $($_.Exception.Message)" -Level "ERROR"
        Exit-Script 1
    }

    Write-Log "Waiting $PostEnableDelaySeconds seconds for repository to exit maintenance mode before starting backup..."
    $waitEnd = (Get-Date).AddSeconds($PostEnableDelaySeconds)
    while ((Get-Date) -lt $waitEnd) {
        $remaining = [int]($waitEnd - (Get-Date)).TotalSeconds
        Write-Host "`r  $remaining seconds remaining...    " -NoNewline
        Start-Sleep -Seconds 1
    }
    Write-Host ""
    Write-Log "Wait complete. Proceeding to Step 5."

} else {
    Write-Log "Step 4: Job is already enabled -- nothing to do."
}


# ==============================================================================
# Step 5: Run the backup job
# ==============================================================================
if ($StartFromStep -le 5) {

    Write-Section "Step 5: Running Backup Job"
    Write-Log "Starting backup job '$($job.Name)' to redownload affected files..."

    try {
        Start-VBOJob -Job $job -ErrorAction Stop | Out-Null
        Write-Log "Backup job started."
    } catch {
        Write-Log "Failed to start backup job: $($_.Exception.Message)" -Level "ERROR"
        Exit-Script 1
    }

    # Brief pause to allow the session record to be created before first poll
    Start-Sleep -Seconds 5

    Write-Log "Polling backup job status every $PollIntervalSeconds seconds. Waiting for completion..."
    Write-Host ""

    $firstPoll     = $true
    $pollLineCount = 0

    # Non-terminal: Running, Queued, Updating -- anything else means the job has finished
    $backupTerminal = @("Success", "Warning", "Failed", "Stopped", "NotConfigured")

    while ($true) {
        $currentSession = Get-VBOJobSession -Job $job -Last
        $timestamp      = Get-Date -Format "yyyy-MM-dd HH:mm:ss"

        if ($null -eq $currentSession) {
            Write-Host "[$timestamp] Waiting for backup session to initialise..."
            Start-Sleep -Seconds $PollIntervalSeconds
            continue
        }

        $displayLines = @(
            "Timestamp  : $timestamp",
            "Job        : $($currentSession.JobName)",
            "Status     : $($currentSession.Status)",
            "Sync Type  : $($currentSession.JobSessionConfigType)"
        )

        if (-not $firstPoll -and $pollLineCount -gt 0) {
            try {
                [System.Console]::SetCursorPosition(0, [System.Console]::CursorTop - $pollLineCount)
            } catch { }
        }
        foreach ($line in $displayLines) { Write-Host $line }
        $pollLineCount = $displayLines.Count
        $firstPoll     = $false

        Add-Content -Path $script:LogFile -Value "[$timestamp] [POLL] Backup -- Status=$($currentSession.Status)  SyncType=$($currentSession.JobSessionConfigType)"

        if ($currentSession.Status -in $backupTerminal) {
            Write-Host ""
            Write-Log "Backup job finished. Final status: $($currentSession.Status)"
            if ($currentSession.Status -eq "Failed") {
                Write-Log "Backup job reported a failure. Check the Veeam console for details. Proceeding to Step 6 for verification." -Level "WARN"
            }
            break
        }

        Start-Sleep -Seconds $PollIntervalSeconds
    }

} else {
    Write-Log "Step 5: Backup job run skipped (StartFromStep=$StartFromStep)."
}


# ==============================================================================
# Step 6: Final verification and identify unrecoverable files
# ==============================================================================
Write-Section "Step 6: Final Verification (Test-VBORepository -DetailedOutput)"
Write-Log "Running detailed integrity check. This downloads file metadata from M365 and may take time."

# Record time before running so we can identify the log file this run produces
$verificationStartTime = Get-Date

try {
    Test-VBORepository -Repository $repository -DetailedOutput -ErrorAction SilentlyContinue 2>&1 | Out-Null
    Write-Log "Detailed verification complete."
} catch {
    Write-Log "Test-VBORepository (detailed) noted: $($_.Exception.Message)" -Level "WARN"
    Write-Log "The detailed log file should still have been written to disk." -Level "WARN"
}

# Locate the integrity log written during this run
$integrityLogBase = Join-Path $env:ProgramData "Veeam\Backup365\Logs\IntegrityVerification"
$integrityLogDir  = Join-Path $integrityLogBase $repository.Name

Write-Log "Looking for integrity verification logs in: $integrityLogDir"

if (-not (Test-Path $integrityLogDir)) {
    Write-Log "Integrity log directory not found: $integrityLogDir" -Level "WARN"
    Write-Log "Check manually under: $integrityLogBase" -Level "WARN"
    Exit-Script 0
}

# Only consider log files written after this run's Test-VBORepository call started
$latestLog = Get-ChildItem -Path $integrityLogDir -Filter "IntegrityVerification_$($repository.Name)_*.log" -ErrorAction SilentlyContinue |
    Where-Object { $_.LastWriteTime -ge $verificationStartTime } |
    Sort-Object LastWriteTime -Descending |
    Select-Object -First 1

if ($null -eq $latestLog) {
    Write-Log "No integrity verification log for repository '$($repository.Name)' written since $($verificationStartTime.ToString('yyyy-MM-dd HH:mm:ss')) found in: $integrityLogDir" -Level "WARN"
    Write-Log "The verification may not have produced output, or the log path differs. Check manually under: $integrityLogBase" -Level "WARN"
    Exit-Script 0
}

Write-Log "Scanning log file: $($latestLog.FullName)"

# Stream the log with Select-String (avoids loading the whole file into memory).
# Match on the rarer marker first, then confirm the second on the same line.
$unrecoverableLines = @(
    Select-String -Path $latestLog.FullName -Pattern 'RepairHistory: FileWasReset' -ErrorAction Stop |
        Where-Object { $_.Line -match 'Backed-up file is empty' } |
        ForEach-Object { $_.Line }
)

if ($unrecoverableLines.Count -eq 0) {
    Write-Log "No unrecoverable files found. All affected files were successfully redownloaded."
    Exit-Script 0
}

Write-Section "Unrecoverable Files -- $($unrecoverableLines.Count) file(s) could not be recovered"
Write-Log "$($unrecoverableLines.Count) file(s) remain empty in the backup. These files no longer exist in Microsoft 365 and cannot be restored."

$parsedFiles = foreach ($line in $unrecoverableLines) {
    $orgId    = 'N/A'
    $siteId   = 'N/A'
    $webId    = 'N/A'
    $intWebId = 'N/A'
    $fileId   = 'N/A'
    $version  = 'N/A'

    if ($line -match 'OrganizationId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})') { $orgId    = $Matches[1] }
    if ($line -match 'SiteId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})')         { $siteId   = $Matches[1] }
    if ($line -match '(?<!Internal)WebId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})') { $webId = $Matches[1] }
    if ($line -match 'InternalWebId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})')  { $intWebId = $Matches[1] }
    if ($line -match 'file: Id:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})')       { $fileId   = $Matches[1] }
    if ($line -match 'Version:\s*(\d+)')                                                                                    { $version  = $Matches[1] }

    [PSCustomObject]@{
        OrganizationId = $orgId
        SiteId         = $siteId
        WebId          = $webId
        InternalWebId  = $intWebId
        FileId         = $fileId
        Version        = $version
    }
}

Write-TableToScreenAndLog $parsedFiles

Write-Log "For further assistance, raise a Veeam Support case: https://www.veeam.com/kb1771"


# ==============================================================================
# Done
# ==============================================================================
Write-Section "Script Complete -- $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"
Write-Log "All steps completed. Review the full log at: $($script:LogFile)"

Stop-Transcript -ErrorAction SilentlyContinue | Out-Null

[raftbone]

@polina

Any estimations, how long step 3 "usually" takes? Our OneDrive objects storage has around 200 TB data and the step 3 is meanwhile running since more than 3 days.

[Bubblebop]

@raftbone My biggest repository is around 6 TB. Step 3 finished in 1–3 hours?, but step 4 took more than 3 days to complete.

I do get alot of "Duplicate web restore points found in object storage." on Step 5. Anyone knows what it means?

[Bubblebop]

@jasonede I haven't tested your script, but it looks like it's missing fetching logs from the proxies

[raftbone]

@raftbone My biggest repository is around 6 TB. Step 3 finished in 1–3 hours?, but step 4 took more than 3 days to complete.

I do get alot of "Duplicate web restore points found in object storage." on Step 5. Anyone knows what it means?

Thanks for sharing your result. Then our 200 TB object storage will hopefully soon finish with step3

[jasonede]

@jasonede I haven't tested your script, but it looks like it's missing fetching logs from the proxies

From what I read in the Veeam KB I don't think that's needed as it's all run from the VBO server and the logs generated are on the server by the script running even if the proxy will be leveraged in the background. Hopefully someone from Veeam will confirm if this is correct at some point.

[jasonede]

From working through our other VBO server where we do use proxies it seems as though the logs are stored there. I've modified main script to request credentials if they're not supplied and test before starting so it can run through unattended. I've also added a second script that enumerates all proxies and requests and tests credentials to them before storing securely. The repair script knows about the format for these files and so if found it will just use them.
Repair-VBORepository.ps1

Code: Select all

#Requires -Version 5.1
<#
.SYNOPSIS
    Identifies and repairs corrupted files in a Veeam Backup for Microsoft 365
    object-storage repository, following the procedure in KB4874.

.PARAMETER JobName
    Name of the Veeam backup job whose repository will be repaired.
    If omitted, the script prompts for it interactively.

.PARAMETER StartFromStep
    Step number to resume from (1-6). Default is 1 (full run).
    Use this to re-run from a specific step after a failure:
      1 - Full run (integrity check through final verification)
      2 - Re-parse integrity check results (re-uses last Test-VBORepository output)
      3 - Skip integrity check; go straight to repair session
      4 - Skip repair session; just re-enable the job
      5 - Skip repair session and re-enable; just run the backup job
      6 - Skip to final verification only
    Note: When starting from Step 3 or lower, the job is disabled first.
    When starting from Step 4 or higher, the job is assumed to be already enabled.

.PARAMETER ProxyCredential
    Credentials used to access VBO proxy servers via UNC in Step 6.
    Lookup priority:
      1. Per-proxy credential file saved by Save-VBOProxyCredentials.ps1
         (<ScriptFolder>\Credentials\<Hostname>.xml)
      2. This parameter (applies to all proxies)
      3. Interactive Get-Credential prompt (reused for remaining proxies)
    Run Save-VBOProxyCredentials.ps1 once to pre-store credentials so this
    script can run without prompting.

.NOTES
    Applies to: Veeam Backup for Microsoft 365 v8.5
    Must be run from an elevated (Run as Administrator) PowerShell window.
#>

[CmdletBinding()]
param(
    [Parameter(Position = 0)]
    [string]$JobName,

    [Parameter()]
    [ValidateRange(1, 6)]
    [int]$StartFromStep = 1,

    [Parameter()]
    [System.Management.Automation.PSCredential]
    $ProxyCredential
)

# ==============================================================================
# Configuration -- adjust these as needed
# ==============================================================================
$TopSitesCount          = 10    # How many top-affected sites to display after Step 2
$PollIntervalSeconds    = 30    # Polling interval for repair session and backup job
$PostEnableDelaySeconds = 60    # Seconds to wait after re-enabling the job before starting it
$ModulePath             = "C:\Program Files\Veeam\Backup365\Veeam.Archiver.PowerShell.dll"
$ScriptDir              = if ($PSScriptRoot) { $PSScriptRoot } else { (Get-Location).Path }
$CredentialDir          = Join-Path $ScriptDir "Credentials"


# ==============================================================================
# Helper functions
# ==============================================================================

function Test-IsAdmin {
    $id = [Security.Principal.WindowsIdentity]::GetCurrent()
    $p  = New-Object Security.Principal.WindowsPrincipal($id)
    return $p.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
}

function Get-SafeFileName ([string]$Name) {
    foreach ($c in [IO.Path]::GetInvalidFileNameChars()) {
        $Name = $Name.Replace([string]$c, '_')
    }
    return $Name
}

function Write-Log {
    param(
        [string]$Message,
        [ValidateSet("INFO","WARN","ERROR")]
        [string]$Level = "INFO"
    )
    $ts   = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
    $line = "[$ts] [$Level] $Message"
    Write-Host $line
    Add-Content -Path $script:LogFile -Value $line
}

function Write-Section ([string]$Title) {
    $bar   = "=" * 80
    $block = @("", $bar, "  $Title", $bar)
    foreach ($line in $block) {
        Write-Host $line
        Add-Content -Path $script:LogFile -Value $line
    }
}

function Write-TableToScreenAndLog ([object[]]$Data, [string[]]$Properties = @()) {
    $table = if ($Properties.Count -gt 0) {
        $Data | Format-Table $Properties -AutoSize -Wrap | Out-String -Width 600
    } else {
        $Data | Format-Table -AutoSize -Wrap | Out-String -Width 600
    }
    Write-Host $table
    Add-Content -Path $script:LogFile -Value $table
}

# Returns a PSCredential for a proxy. Priority:
#   1. Credential file saved by Save-VBOProxyCredentials.ps1 (<CredentialDir>\<hostname>.xml)
#   2. -ProxyCredential parameter (stored in $script:ProxyCredential)
#   3. Interactive Get-Credential prompt (reused for subsequent proxies)
function Get-ProxyCredential ([string]$Hostname) {
    $credFile = Join-Path $CredentialDir "$Hostname.xml"
    if (Test-Path $credFile) {
        try {
            $cred = Import-Clixml -Path $credFile -ErrorAction Stop
            Write-Log "Loaded stored credentials for '$Hostname'."
            return $cred
        } catch {
            Write-Log "Could not load credential file for '$Hostname' ($credFile): $($_.Exception.Message)" -Level "WARN"
        }
    }

    if ($null -ne $script:ProxyCredential) {
        Write-Log "Using -ProxyCredential parameter for '$Hostname'."
        return $script:ProxyCredential
    }

    Write-Log "No stored credentials for '$Hostname'. Prompting..."
    $prompted = Get-Credential -Message "Enter credentials for proxy '$Hostname' (used for UNC log retrieval)"
    if ($null -ne $prompted) {
        $script:ProxyCredential = $prompted
    }
    return $prompted
}

# Called on all exit paths -- re-enables the job if this script disabled it
function Exit-Script ([int]$Code = 0) {
    if ($null -ne $script:DisabledJob) {
        Write-Log "Re-enabling job '$($script:DisabledJob.Name)' before exit..."
        try {
            Enable-VBOJob -Job $script:DisabledJob -ErrorAction SilentlyContinue | Out-Null
            Write-Log "Job re-enabled."
        } catch {
            Write-Log "Could not re-enable job automatically. Please re-enable manually in the Veeam console: $($script:DisabledJob.Name)" -Level "WARN"
        }
        $script:DisabledJob = $null
    }
    Write-Log "Script exiting with code $Code."
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit $Code
}

$script:DisabledJob       = $null
$script:LogFile           = $null   # set below once we have the job name
$script:ProxyCredential   = $ProxyCredential
$script:EligibleProxies   = @()


# ==============================================================================
# Job name -- parameter or interactive prompt
# ==============================================================================
if ([string]::IsNullOrWhiteSpace($JobName)) {
    $JobName = Read-Host "Enter the Veeam backup job name"
}
if ([string]::IsNullOrWhiteSpace($JobName)) {
    Write-Host "[ERROR] No job name provided. Exiting."
    exit 1
}


# ==============================================================================
# Log and transcript setup
# ==============================================================================
$SafeName  = Get-SafeFileName $JobName
$LogDir    = Join-Path (Get-Location).Path "Logs"

if (-not (Test-Path $LogDir)) {
    New-Item -Path $LogDir -ItemType Directory -Force | Out-Null
}

$script:LogFile = Join-Path $LogDir "${SafeName}.log"
$TranscriptFile = Join-Path $LogDir "Transcript_${SafeName}.log"

# Transcript is overwritten on each run
Start-Transcript -Path $TranscriptFile -Force | Out-Null

# Append a run-start separator to the log
Add-Content -Path $script:LogFile -Value ""
Add-Content -Path $script:LogFile -Value ("*" * 80)
Add-Content -Path $script:LogFile -Value ("*  NEW RUN -- $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')  StartFromStep=$StartFromStep")
Add-Content -Path $script:LogFile -Value ("*" * 80)

Write-Section "Veeam VBO365 Repository Repair (KB4874) -- $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"
Write-Log "Job name:              $JobName"
Write-Log "Starting from step:    $StartFromStep"
Write-Log "Log file:              $($script:LogFile)"
Write-Log "Transcript:            $TranscriptFile"
Write-Log "Running as admin:      $(Test-IsAdmin)"
Write-Log "Top sites to display:  $TopSitesCount"
Write-Log "Post-enable delay:     ${PostEnableDelaySeconds}s"


# ==============================================================================
# Prerequisites -- module
# ==============================================================================
Write-Section "Checking Prerequisites"

if (-not (Test-Path $ModulePath)) {
    Write-Log "Veeam PowerShell module not found at: $ModulePath" -Level "ERROR"
    Write-Log "Ensure Veeam Backup for Microsoft 365 v8.5 is installed." -Level "ERROR"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 1
}

try {
    Import-Module $ModulePath -ErrorAction Stop
    Write-Log "Veeam PowerShell module imported successfully."
} catch {
    Write-Log "Failed to import Veeam module: $($_.Exception.Message)" -Level "ERROR"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 1
}


# ==============================================================================
# Locate the job
# ==============================================================================
Write-Section "Locating Backup Job"

$job = $null
try {
    $job = Get-VBOJob -Name $JobName -ErrorAction Stop
} catch {
    Write-Log "Could not find a job named '$JobName': $($_.Exception.Message)" -Level "ERROR"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 1
}
Write-Log "Found job: $($job.Name)"


# ==============================================================================
# Check job scope -- must include OneDrive, SharePoint, or Teams
# ==============================================================================
Write-Section "Checking Job Scope"

$items = $null
try {
    $items = @(Get-VBOBackupItem -Job $job -ErrorAction Stop)
} catch {
    Write-Log "Could not retrieve backup items for job '$JobName': $($_.Exception.Message)" -Level "ERROR"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 1
}

$hasOneDrive   = ($items | Where-Object { $_.OneDrive } | Measure-Object).Count -gt 0
$hasSharePoint = ($items | Where-Object { $_.Sites }    | Measure-Object).Count -gt 0
$hasTeams      = ($items | Where-Object { $_.Teams }    | Measure-Object).Count -gt 0

Write-Log "Job scope -- OneDrive: $hasOneDrive  |  SharePoint: $hasSharePoint  |  Teams: $hasTeams"

if (-not ($hasOneDrive -or $hasSharePoint -or $hasTeams)) {
    Write-Log "Job '$JobName' does not back up OneDrive, SharePoint, or Teams workloads." -Level "WARN"
    Write-Log "KB4874 remediation applies only to those workloads. No action taken." -Level "WARN"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 0
}
Write-Log "Job covers at least one applicable workload. Proceeding."


# ==============================================================================
# Get repository from job
# ==============================================================================
$repository = $job.Repository
if ($null -eq $repository) {
    Write-Log "Could not determine the repository from the job object. Verify the job has an assigned repository." -Level "ERROR"
    Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
    exit 1
}
Write-Log "Repository: $($repository.Name)"


# ==============================================================================
# Proxy detection and credential check
# ==============================================================================
Write-Section "Proxy Detection"

try {
    $allProxies = @(Get-VBOProxy -ErrorAction Stop)
    $script:EligibleProxies = @($allProxies | Where-Object {
        $_.OperatingSystemKind -eq 'Windows' -and
        $_.State               -eq 'Online'  -and
        $_.MaintenanceModeState -eq 'Disabled'
    })
    Write-Log "$($allProxies.Count) proxy/proxies configured; $($script:EligibleProxies.Count) eligible (Windows, online, not in maintenance)."
} catch {
    Write-Log "Could not enumerate proxies: $($_.Exception.Message)" -Level "WARN"
}

if ($script:EligibleProxies.Count -gt 0) {
    Write-Log "Credential source per proxy (checked at Step 6 log retrieval):"
    foreach ($proxy in $script:EligibleProxies) {
        $credFile = Join-Path $CredentialDir "$($proxy.Hostname).xml"
        $status   = if (Test-Path $credFile) {
            "credential file found"
        } elseif ($null -ne $script:ProxyCredential) {
            "will use -ProxyCredential parameter"
        } else {
            "no stored credentials -- will prompt if needed in Step 6"
        }
        Write-Log "  $($proxy.Hostname): $status"
    }
    Write-Log "To store proxy credentials in advance, run Save-VBOProxyCredentials.ps1."
} else {
    Write-Log "No eligible proxies found. Integrity logs will be read locally."
}


# ==============================================================================
# Pre-Step: Disable the job (only when running Steps 1-3)
# ==============================================================================
if ($StartFromStep -le 3) {
    Write-Section "Pre-Step: Disabling Backup Job"

    # Wait for any currently-running session to finish before disabling
    $activeSession = Get-VBOJobSession -Job $job -Last
    $jobRunningStatuses = @("Running", "Queued", "Updating")

    if ($null -ne $activeSession -and $activeSession.Status -in $jobRunningStatuses) {
        Write-Log "Job '$($job.Name)' is currently running (Status=$($activeSession.Status)). Waiting for it to finish before disabling..."
        Write-Host ""

        $firstPoll     = $true
        $pollLineCount = 0

        while ($true) {
            $activeSession = Get-VBOJobSession -Job $job -Last
            $timestamp     = Get-Date -Format "yyyy-MM-dd HH:mm:ss"

            $displayLines = @(
                "Timestamp  : $timestamp",
                "Job        : $($activeSession.JobName)",
                "Status     : $($activeSession.Status)",
                "Sync Type  : $($activeSession.JobSessionConfigType)"
            )

            if (-not $firstPoll -and $pollLineCount -gt 0) {
                try {
                    [System.Console]::SetCursorPosition(0, [System.Console]::CursorTop - $pollLineCount)
                } catch { }
            }
            foreach ($line in $displayLines) { Write-Host $line }
            $pollLineCount = $displayLines.Count
            $firstPoll     = $false

            Add-Content -Path $script:LogFile -Value "[$timestamp] [POLL] Waiting for running job -- Status=$($activeSession.Status)"

            if ($activeSession.Status -notin $jobRunningStatuses) {
                Write-Host ""
                Write-Log "Job session finished with status '$($activeSession.Status)'. Proceeding to disable."
                break
            }

            Start-Sleep -Seconds $PollIntervalSeconds
        }
    } else {
        Write-Log "Job '$($job.Name)' is not currently running. Proceeding to disable."
    }

    Write-Log "Disabling job '$($job.Name)' to prevent scheduled runs during repair..."

    try {
        Disable-VBOJob -Job $job -ErrorAction Stop | Out-Null
        $script:DisabledJob = $job
        Write-Log "Job disabled."
    } catch {
        Write-Log "Failed to disable job: $($_.Exception.Message)" -Level "ERROR"
        Stop-Transcript -ErrorAction SilentlyContinue | Out-Null
        exit 1
    }
} else {
    Write-Log "Pre-Step: Job disable skipped (StartFromStep=$StartFromStep -- job assumed already enabled)."
}


# ==============================================================================
# Steps 1 and 2: Integrity check and review results
# ==============================================================================
if ($StartFromStep -le 2) {

    # ------------------------------------------------------------------
    # Step 1: Integrity check
    # ------------------------------------------------------------------
    Write-Section "Step 1: Running Integrity Check (Test-VBORepository)"
    Write-Log "This may take several hours depending on repository size."

    if (-not (Test-IsAdmin)) {
        Write-Log "WARNING: PowerShell is NOT running as administrator. If Test-VBORepository fails, close this session and re-run the script from an elevated (Run as Administrator) PowerShell window." -Level "WARN"
    }

    Write-Log "Running Test-VBORepository against '$($repository.Name)'..."

    $integrityErrors = @()
    try {
        Test-VBORepository -Repository $repository -ErrorVariable integrityErrors -ErrorAction SilentlyContinue | Out-Null
    } catch {
        $integrityErrors += $_
    }

    # Check whether the failure looks privilege-related
    if ($integrityErrors.Count -gt 0) {
        $combinedMessages = ($integrityErrors | ForEach-Object { "$_" }) -join " "
        if ((-not (Test-IsAdmin)) -and ($combinedMessages -match 'access.denied|denied|privilege|administrator|elevation|unauthorized')) {
            Write-Log "Test-VBORepository appears to have failed because the session is not elevated." -Level "ERROR"
            Write-Log "Please close this window, re-open PowerShell using 'Run as Administrator', and run the script again." -Level "ERROR"
            Exit-Script 1
        }
    }

    Write-Log "Integrity check complete. Reviewing results..."

    # ------------------------------------------------------------------
    # Step 2: Parse and review results
    # ------------------------------------------------------------------
    Write-Section "Step 2: Reviewing Integrity Check Results"

    $corruptedSites = [System.Collections.Generic.List[PSCustomObject]]::new()
    $corruptedLists = [System.Collections.Generic.List[PSCustomObject]]::new()

    foreach ($errObj in $integrityErrors) {
        $msg = "$errObj"

        if ($msg -match "The following site has corrupted file data:\s*(.+?)\s*\(.*?count:\s*(\d+)\)") {
            $siteUrl   = $Matches[1].Trim()
            $siteCount = [int]$Matches[2]
            $corruptedSites.Add([PSCustomObject]@{ Type = "Site"; Path = $siteUrl; Count = $siteCount })
        }

        if ($msg -match "The following list contains items referencing corrupted files:\s*(.+?)\s*\(.*?count:\s*(\d+)\)") {
            $listPath  = $Matches[1].Trim()
            $listCount = [int]$Matches[2]
            $corruptedLists.Add([PSCustomObject]@{ Type = "List"; Path = $listPath; Count = $listCount })
        }
    }

    $corruptionFound = ($corruptedSites.Count -gt 0) -or ($corruptedLists.Count -gt 0)

    if (-not $corruptionFound) {
        Write-Log "No corruption detected in repository '$($repository.Name)'."
        Write-Log "No further action is required."
        Exit-Script 0
    }

    $totalSiteFiles = ($corruptedSites | Measure-Object -Property Count -Sum).Sum
    $totalListItems = ($corruptedLists | Measure-Object -Property Count -Sum).Sum
    $totalAffected  = $totalSiteFiles + $totalListItems

    Write-Log "Corruption detected."
    Write-Log "  Corrupted sites:      $($corruptedSites.Count) site(s) -- $totalSiteFiles affected file(s)"
    Write-Log "  Corrupted lists:      $($corruptedLists.Count) list(s) -- $totalListItems affected item(s)"
    Write-Log "  Total affected items: $totalAffected"

    $allAffected = [System.Collections.Generic.List[PSCustomObject]]::new()
    foreach ($s in $corruptedSites) { $allAffected.Add($s) }
    foreach ($l in $corruptedLists) { $allAffected.Add($l) }

    $topAffected = @($allAffected | Sort-Object Count -Descending | Select-Object -First $TopSitesCount)

    $topHeader = "Top $TopSitesCount most-affected sites/locations (by corrupted item count):"
    Write-Host ""
    Write-Host $topHeader
    Add-Content -Path $script:LogFile -Value ""
    Add-Content -Path $script:LogFile -Value $topHeader

    Write-TableToScreenAndLog $topAffected -Properties @("Type","Count","Path")

} else {
    Write-Log "Steps 1-2: Integrity check skipped (StartFromStep=$StartFromStep)."
}


# ==============================================================================
# Step 3: Start repair session
# ==============================================================================
if ($StartFromStep -le 3) {

    Write-Section "Step 3: Scanning Repository and Flagging Affected Items"
    Write-Log "Starting full repair session on repository '$($repository.Name)'..."
    Write-Log "Note: The repository is locked for all backups and restores during this step."
    Write-Log "      Ensure all backup and restore sessions for this repository are closed."

    $repairSession = $null
    try {
        $repairSession = Start-VBORepositoryRepairSession `
            -Repository $repository `
            -Type CleanMissingFilesData `
            -Mode Full `
            -Scope Full `
            -ErrorAction Stop
        Write-Log "Repair session started. Session ID: $($repairSession.Id)"
    } catch {
        Write-Log "Failed to start repair session: $($_.Exception.Message)" -Level "ERROR"
        Exit-Script 1
    }

    Write-Log "Polling repair session every $PollIntervalSeconds seconds. Waiting for completion..."
    Write-Host ""

    # Non-terminal states -- session is still starting up or running
    $repairNonTerminal = @("Running", "Preparing", "Queued")

    $firstPoll     = $true
    $pollLineCount = 0

    while ($true) {
        $sessionState = Get-VBORepositoryRepairSession -Id $repairSession.Id
        $timestamp    = Get-Date -Format "yyyy-MM-dd HH:mm:ss"

        $displayLines = @(
            "Timestamp  : $timestamp",
            "Id         : $($sessionState.Id)",
            "Type       : $($sessionState.Type)",
            "Repository : $($sessionState.RepositoryId)",
            "State      : $($sessionState.State)",
            "Error      : $($sessionState.ErrorMessage)"
        )

        if (-not $firstPoll -and $pollLineCount -gt 0) {
            try {
                [System.Console]::SetCursorPosition(0, [System.Console]::CursorTop - $pollLineCount)
            } catch { }
        }
        foreach ($line in $displayLines) { Write-Host $line }
        $pollLineCount = $displayLines.Count
        $firstPoll     = $false

        Add-Content -Path $script:LogFile -Value "[$timestamp] [POLL] Repair -- State=$($sessionState.State)  Error=$($sessionState.ErrorMessage)"

        if ($sessionState.State -notin $repairNonTerminal) {
            Write-Host ""
            if ($sessionState.State -eq "Finished" -and [string]::IsNullOrEmpty($sessionState.ErrorMessage)) {
                Write-Log "Repair session completed successfully."
            } else {
                Write-Log "Repair session ended with state '$($sessionState.State)'." -Level "WARN"
                if (-not [string]::IsNullOrEmpty($sessionState.ErrorMessage)) {
                    Write-Log "Session error: $($sessionState.ErrorMessage)" -Level "ERROR"
                    Write-Log "Stopping -- repair session reported an error. Review the error above before retrying." -Level "ERROR"
                    Exit-Script 1
                }
            }
            break
        }

        Start-Sleep -Seconds $PollIntervalSeconds
    }

} else {
    Write-Log "Step 3: Repair session skipped (StartFromStep=$StartFromStep)."
}


# ==============================================================================
# Step 4: Re-enable the job
# ==============================================================================
Write-Section "Step 4: Re-enabling Backup Job"

# Decide whether the job needs enabling. Re-enable if this run disabled it, or if
# the job is currently found disabled (e.g. left disabled by an aborted prior run).
$disabledByThisRun = ($null -ne $script:DisabledJob)
$needsEnable       = $disabledByThisRun

if (-not $disabledByThisRun) {
    # Re-query current state -- IsEnabled on the object fetched at start may be stale
    try {
        $currentJobState = Get-VBOJob -Name $JobName -ErrorAction Stop
        if (-not $currentJobState.IsEnabled) {
            $needsEnable = $true
            Write-Log "Job '$($job.Name)' is currently disabled (not by this run). It will be re-enabled."
        }
    } catch {
        Write-Log "Could not re-query job state to check if enabled: $($_.Exception.Message)" -Level "WARN"
    }
}

if ($needsEnable) {
    Write-Log "Re-enabling job '$($job.Name)'..."
    try {
        Enable-VBOJob -Job $job -ErrorAction Stop | Out-Null
        $script:DisabledJob = $null
        Write-Log "Job re-enabled successfully."
    } catch {
        Write-Log "Failed to re-enable job: $($_.Exception.Message)" -Level "ERROR"
        Exit-Script 1
    }

    Write-Log "Waiting $PostEnableDelaySeconds seconds for repository to exit maintenance mode before starting backup..."
    $waitEnd = (Get-Date).AddSeconds($PostEnableDelaySeconds)
    while ((Get-Date) -lt $waitEnd) {
        $remaining = [int]($waitEnd - (Get-Date)).TotalSeconds
        Write-Host "`r  $remaining seconds remaining...    " -NoNewline
        Start-Sleep -Seconds 1
    }
    Write-Host ""
    Write-Log "Wait complete. Proceeding to Step 5."

} else {
    Write-Log "Step 4: Job is already enabled -- nothing to do."
}


# ==============================================================================
# Step 5: Run the backup job
# ==============================================================================
if ($StartFromStep -le 5) {

    Write-Section "Step 5: Running Backup Job"
    Write-Log "Starting backup job '$($job.Name)' to redownload affected files..."

    try {
        Start-VBOJob -Job $job -ErrorAction Stop | Out-Null
        Write-Log "Backup job started."
    } catch {
        Write-Log "Failed to start backup job: $($_.Exception.Message)" -Level "ERROR"
        Exit-Script 1
    }

    # Brief pause to allow the session record to be created before first poll
    Start-Sleep -Seconds 5

    Write-Log "Polling backup job status every $PollIntervalSeconds seconds. Waiting for completion..."
    Write-Host ""

    $firstPoll     = $true
    $pollLineCount = 0

    # Non-terminal: Running, Queued, Updating -- anything else means the job has finished
    $backupTerminal = @("Success", "Warning", "Failed", "Stopped", "NotConfigured")

    while ($true) {
        $currentSession = Get-VBOJobSession -Job $job -Last
        $timestamp      = Get-Date -Format "yyyy-MM-dd HH:mm:ss"

        if ($null -eq $currentSession) {
            Write-Host "[$timestamp] Waiting for backup session to initialise..."
            Start-Sleep -Seconds $PollIntervalSeconds
            continue
        }

        $displayLines = @(
            "Timestamp  : $timestamp",
            "Job        : $($currentSession.JobName)",
            "Status     : $($currentSession.Status)",
            "Sync Type  : $($currentSession.JobSessionConfigType)"
        )

        if (-not $firstPoll -and $pollLineCount -gt 0) {
            try {
                [System.Console]::SetCursorPosition(0, [System.Console]::CursorTop - $pollLineCount)
            } catch { }
        }
        foreach ($line in $displayLines) { Write-Host $line }
        $pollLineCount = $displayLines.Count
        $firstPoll     = $false

        Add-Content -Path $script:LogFile -Value "[$timestamp] [POLL] Backup -- Status=$($currentSession.Status)  SyncType=$($currentSession.JobSessionConfigType)"

        if ($currentSession.Status -in $backupTerminal) {
            Write-Host ""
            Write-Log "Backup job finished. Final status: $($currentSession.Status)"
            if ($currentSession.Status -eq "Failed") {
                Write-Log "Backup job reported a failure. Check the Veeam console for details. Proceeding to Step 6 for verification." -Level "WARN"
            }
            break
        }

        Start-Sleep -Seconds $PollIntervalSeconds
    }

} else {
    Write-Log "Step 5: Backup job run skipped (StartFromStep=$StartFromStep)."
}


# ==============================================================================
# Step 6: Final verification and identify unrecoverable files
# ==============================================================================
Write-Section "Step 6: Final Verification (Test-VBORepository -DetailedOutput)"
Write-Log "Running detailed integrity check. This downloads file metadata from M365 and may take time."

# Record time before running so we can identify the log file this run produces
$verificationStartTime = Get-Date

try {
    Test-VBORepository -Repository $repository -DetailedOutput -ErrorAction SilentlyContinue 2>&1 | Out-Null
    Write-Log "Detailed verification complete."
} catch {
    Write-Log "Test-VBORepository (detailed) noted: $($_.Exception.Message)" -Level "WARN"
    Write-Log "The detailed log file should still have been written to disk." -Level "WARN"
}

# Locate the integrity log written during this run.
# Logs are written on whichever machine ran the verification -- the backup server
# when no proxy is used, or the proxy otherwise. If found on a proxy the log is
# copied to the local Logs folder so processing always works on a local file.
$veeamLogRelPath = "Veeam\Backup365\Logs\IntegrityVerification\$($repository.Name)"
$logFilter       = "IntegrityVerification_$($repository.Name)_*.log"
$latestLog       = $null

# Helper: find the most recent qualifying log in a given directory
function Find-LatestLog ([string]$Dir) {
    Get-ChildItem -Path $Dir -Filter $logFilter -ErrorAction SilentlyContinue |
        Where-Object { $_.LastWriteTime -ge $verificationStartTime } |
        Sort-Object LastWriteTime -Descending |
        Select-Object -First 1
}

# 1. Try local path
$localLogDir = Join-Path $env:ProgramData $veeamLogRelPath
Write-Log "Checking local integrity log path: $localLogDir"

if (Test-Path $localLogDir -ErrorAction SilentlyContinue) {
    $latestLog = Find-LatestLog $localLogDir
    if ($null -ne $latestLog) {
        Write-Log "Integrity log found locally: $($latestLog.FullName)"
    }
}

# 2. If not found locally, check proxies
if ($null -eq $latestLog) {
    if ($script:EligibleProxies.Count -gt 0) {
        Write-Log "Log not found locally. Checking $($script:EligibleProxies.Count) proxy/proxies..."

        $driveName = "VBOLog"

        foreach ($proxy in $script:EligibleProxies) {
            $proxyHost = $proxy.Hostname
            Write-Log "Checking proxy '$proxyHost'..."

            $cred      = Get-ProxyCredential $proxyHost
            $connected = $false

            try {
                Remove-PSDrive -Name $driveName -Force -ErrorAction SilentlyContinue

                if ($null -ne $cred) {
                    New-PSDrive -Name $driveName -PSProvider FileSystem `
                        -Root "\\$proxyHost\c$" `
                        -Credential $cred `
                        -ErrorAction Stop | Out-Null
                    $connected = $true
                    $searchDir = "${driveName}:\ProgramData\$veeamLogRelPath"
                } else {
                    $searchDir = "\\$proxyHost\c$\ProgramData\$veeamLogRelPath"
                }

                if (Test-Path $searchDir -ErrorAction SilentlyContinue) {
                    $remoteLog = Find-LatestLog $searchDir
                    if ($null -ne $remoteLog) {
                        Write-Log "Found log on proxy '$proxyHost': $($remoteLog.Name)"
                        Write-Log "Copying to local Logs folder for processing..."
                        $localCopyPath = Join-Path $LogDir $remoteLog.Name
                        Copy-Item -Path $remoteLog.FullName -Destination $localCopyPath -Force -ErrorAction Stop
                        $latestLog = Get-Item $localCopyPath
                        Write-Log "Log copied to: $localCopyPath"
                        break
                    }
                }
            } catch {
                Write-Log "Could not access proxy '$proxyHost': $($_.Exception.Message)" -Level "WARN"
            } finally {
                if ($connected) {
                    Remove-PSDrive -Name $driveName -Force -ErrorAction SilentlyContinue
                }
            }
        }
    } else {
        Write-Log "No eligible proxies to check." -Level "WARN"
    }
}

if ($null -eq $latestLog) {
    Write-Log "No integrity verification log for repository '$($repository.Name)' written since $($verificationStartTime.ToString('yyyy-MM-dd HH:mm:ss')) found locally or on any proxy." -Level "WARN"
    Write-Log "Expected local path: $localLogDir" -Level "WARN"
    Exit-Script 0
}

Write-Log "Scanning log file: $($latestLog.FullName)"

# Stream the log with Select-String (avoids loading the whole file into memory).
# Match on the rarer marker first, then confirm the second on the same line.
$unrecoverableLines = @(
    Select-String -Path $latestLog.FullName -Pattern 'RepairHistory: FileWasReset' -ErrorAction Stop |
        Where-Object { $_.Line -match 'Backed-up file is empty' } |
        ForEach-Object { $_.Line }
)

if ($unrecoverableLines.Count -eq 0) {
    Write-Log "No unrecoverable files found. All affected files were successfully redownloaded."
    Exit-Script 0
}

Write-Section "Unrecoverable Files -- $($unrecoverableLines.Count) file(s) could not be recovered"
Write-Log "$($unrecoverableLines.Count) file(s) remain empty in the backup. These files no longer exist in Microsoft 365 and cannot be restored."

$parsedFiles = foreach ($line in $unrecoverableLines) {
    $orgId    = 'N/A'
    $siteId   = 'N/A'
    $webId    = 'N/A'
    $intWebId = 'N/A'
    $fileId   = 'N/A'
    $version  = 'N/A'

    if ($line -match 'OrganizationId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})') { $orgId    = $Matches[1] }
    if ($line -match 'SiteId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})')         { $siteId   = $Matches[1] }
    if ($line -match '(?<!Internal)WebId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})') { $webId = $Matches[1] }
    if ($line -match 'InternalWebId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})')  { $intWebId = $Matches[1] }
    if ($line -match 'file: Id:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})')       { $fileId   = $Matches[1] }
    if ($line -match 'Version:\s*(\d+)')                                                                                    { $version  = $Matches[1] }

    [PSCustomObject]@{
        OrganizationId = $orgId
        SiteId         = $siteId
        WebId          = $webId
        InternalWebId  = $intWebId
        FileId         = $fileId
        Version        = $version
    }
}

Write-TableToScreenAndLog $parsedFiles

Write-Log "For further assistance, raise a Veeam Support case: https://www.veeam.com/kb1771"


# ==============================================================================
# Done
# ==============================================================================
Write-Section "Script Complete -- $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"
Write-Log "All steps completed. Review the full log at: $($script:LogFile)"

Stop-Transcript -ErrorAction SilentlyContinue | Out-Null

Save-VBOProxyCredentials.ps1

Code: Select all

#Requires -Version 5.1
<#
.SYNOPSIS
    Stores credentials for Veeam Backup for Microsoft 365 proxy servers so that
    Repair-VBORepository.ps1 can access proxy integrity logs without prompting.

.DESCRIPTION
    Enumerates all eligible Windows proxies, prompts for credentials (once for
    all proxies or individually per proxy), tests UNC access, and saves each
    credential as an encrypted XML file in the Credentials subfolder.

    Files are encrypted with Windows DPAPI -- they can only be decrypted by the
    same Windows user account on the same machine. Run this script as the account
    that will also run Repair-VBORepository.ps1.

.NOTES
    Applies to: Veeam Backup for Microsoft 365 v8.5
    Must be run from an elevated (Run as Administrator) PowerShell window.
    Credential files are saved to: <ScriptFolder>\Credentials\<Hostname>.xml
#>

[CmdletBinding()]
param()

# ==============================================================================
# Configuration
# ==============================================================================
$ModulePath    = "C:\Program Files\Veeam\Backup365\Veeam.Archiver.PowerShell.dll"
$ScriptDir     = if ($PSScriptRoot) { $PSScriptRoot } else { (Get-Location).Path }
$CredentialDir = Join-Path $ScriptDir "Credentials"


# ==============================================================================
# Prerequisites
# ==============================================================================
if (-not (Test-Path $ModulePath)) {
    Write-Error "Veeam PowerShell module not found at: $ModulePath. Ensure Veeam Backup for Microsoft 365 v8.5 is installed."
    exit 1
}

try {
    Import-Module $ModulePath -ErrorAction Stop
} catch {
    Write-Error "Failed to import Veeam module: $($_.Exception.Message)"
    exit 1
}

if (-not (Test-Path $CredentialDir)) {
    New-Item -Path $CredentialDir -ItemType Directory -Force | Out-Null
    Write-Host "Created credential store: $CredentialDir"
}


# ==============================================================================
# Enumerate eligible proxies
# ==============================================================================
Write-Host ""
Write-Host "Enumerating proxies..."

$proxies = @()
try {
    $proxies = @(Get-VBOProxy -ErrorAction Stop | Where-Object {
        $_.OperatingSystemKind -eq 'Windows' -and
        $_.State               -eq 'Online'  -and
        $_.MaintenanceModeState -eq 'Disabled'
    })
} catch {
    Write-Error "Could not retrieve proxies: $($_.Exception.Message)"
    exit 1
}

if ($proxies.Count -eq 0) {
    Write-Host "No eligible Windows proxies found (must be Online and not in maintenance)."
    exit 0
}

Write-Host "Found $($proxies.Count) eligible proxy/proxies:"
foreach ($p in $proxies) {
    $existing = if (Test-Path (Join-Path $CredentialDir "$($p.Hostname).xml")) { " [credential file exists]" } else { "" }
    Write-Host "  $($p.Hostname)$existing"
}
Write-Host ""


# ==============================================================================
# Ask if same credentials for all proxies
# ==============================================================================
$sameForAll = ""
while ($sameForAll -notin @("Y", "N")) {
    $sameForAll = (Read-Host "Use the same credentials for all proxies? [Y/N]").ToUpper().Trim()
}

$sharedCredential = $null
if ($sameForAll -eq "Y") {
    $sharedCredential = Get-Credential -Message "Enter credentials for all VBO proxy servers"
    if ($null -eq $sharedCredential) {
        Write-Host "No credentials entered. Exiting."
        exit 1
    }
}


# ==============================================================================
# Process each proxy
# ==============================================================================
$results = [System.Collections.Generic.List[PSCustomObject]]::new()

foreach ($proxy in $proxies) {
    $hostname  = $proxy.Hostname
    $driveName = "VBOCredTest"
    Write-Host ""
    Write-Host "--- Proxy: $hostname ---"

    # Start with the shared credential if one was collected; otherwise prompt now
    $cred = if ($null -ne $sharedCredential) {
        $sharedCredential
    } else {
        Get-Credential -Message "Enter credentials for proxy '$hostname'"
    }

    $saved = $false

    while ($true) {
        if ($null -eq $cred) {
            Write-Host "  Skipped -- no credentials entered."
            $results.Add([PSCustomObject]@{ Proxy = $hostname; Test = "Skipped"; Saved = "No" })
            break
        }

        # Test UNC access
        Write-Host "  Testing UNC access to \\$hostname\c$ ..."
        Remove-PSDrive -Name $driveName -Force -ErrorAction SilentlyContinue
        $testPassed  = $false
        $testMessage = ""

        try {
            New-PSDrive -Name $driveName -PSProvider FileSystem `
                -Root "\\$hostname\c$" `
                -Credential $cred `
                -ErrorAction Stop | Out-Null
            $testPassed = $true
            Write-Host "  Access confirmed."
        } catch {
            $testMessage = $_.Exception.Message
            Write-Host "  Access test failed -- $testMessage"
        } finally {
            Remove-PSDrive -Name $driveName -Force -ErrorAction SilentlyContinue
        }

        if ($testPassed) {
            # Save credential file
            $credFile = Join-Path $CredentialDir "$hostname.xml"
            try {
                $cred | Export-Clixml -Path $credFile -Force -ErrorAction Stop
                $saved = $true
                Write-Host "  Saved: $credFile"
                $results.Add([PSCustomObject]@{ Proxy = $hostname; Test = "Passed"; Saved = "Yes -- $credFile" })
            } catch {
                Write-Host "  ERROR saving credential file: $($_.Exception.Message)"
                $results.Add([PSCustomObject]@{ Proxy = $hostname; Test = "Passed"; Saved = "Error -- $($_.Exception.Message)" })
            }
            break
        }

        # Test failed -- prompt for new credentials and retry
        Write-Host "  Enter different credentials to retry, or press Cancel to skip this proxy."
        $cred = Get-Credential -Message "Retry: Enter credentials for proxy '$hostname'"
        if ($null -eq $cred) {
            Write-Host "  Skipped -- no credentials saved for '$hostname'."
            $results.Add([PSCustomObject]@{ Proxy = $hostname; Test = "Failed -- $testMessage"; Saved = "No" })
            break
        }
    }
}


# ==============================================================================
# Summary
# ==============================================================================
Write-Host ""
Write-Host ("=" * 80)
Write-Host "  Summary"
Write-Host ("=" * 80)
$results | Format-Table -AutoSize -Wrap | Out-String -Width 300 | Write-Host
Write-Host "Credential files are stored in: $CredentialDir"
Write-Host "These files can only be decrypted by this user account on this machine."
Write-Host ""
Write-Host "Repair-VBORepository.ps1 will automatically load these files when accessing proxy logs."

[jasonede]

Just a little edit. It seems it needs Disable-ConsoleQuickEdit at the start and resetting at the end otherwise it can pause when the console locks normally happens during the repair stage. Pressing escape on the window should get it going again.

[DaStivi]

I would have another idea/request: check the repositories for linked jobs and, instead of only verifying that they are not currently running, automatically disable those jobs until the move has completed.

as a example i also have tenants with multiple jobs, but pointing to a single repo...
and I've also running jobs every 2hours!

I only learned from your posts (and after carefully reading the KB) that no jobs are allowed to run during the process. That's quite a bummer when dealing with large repositories, where a check/repair can take a considerable amount of time.

[jasonede]

The clean operation puts the repository in maintenance mode anyway so it won't let any job running on it start.

[jasonede]

Untested, but the attached should disable all jobs using the repo and then enable when done. It stores the state so if it crashes it knows what jobs to enable.

Code: Select all

#Requires -Version 5.1
<#
.SYNOPSIS
    Identifies and repairs corrupted files in a Veeam Backup for Microsoft 365
    object-storage repository, following the procedure in KB4874.

.PARAMETER JobName
    Name of the Veeam backup job whose repository will be repaired.
    If omitted, the script prompts for it interactively.

.PARAMETER StartFromStep
    Step number to resume from (1-6). Default is 1 (full run).
    Use this to re-run from a specific step after a failure:
      1 - Full run (integrity check through final verification)
      2 - Re-parse integrity check results (re-uses last Test-VBORepository output)
      3 - Skip integrity check; go straight to repair session
      4 - Skip repair session; just re-enable the job
      5 - Skip repair session and re-enable; just run the backup job
      6 - Skip to final verification only
    Note: When starting from Step 3 or lower, the job is disabled first.
    When starting from Step 4 or higher, the job is assumed to be already enabled.

.PARAMETER ProxyCredential
    Credentials used to access VBO proxy servers via UNC in Step 6.
    Lookup priority:
      1. Per-proxy credential file saved by Save-VBOProxyCredentials.ps1
         (<ScriptFolder>\Credentials\<Hostname>.xml)
      2. This parameter (applies to all proxies)
      3. Interactive Get-Credential prompt (reused for remaining proxies)
    Run Save-VBOProxyCredentials.ps1 once to pre-store credentials so this
    script can run without prompting.

.NOTES
    Applies to: Veeam Backup for Microsoft 365 v8.5
    Must be run from an elevated (Run as Administrator) PowerShell window.
#>

[CmdletBinding()]
param(
    [Parameter(Position = 0)]
    [string]$JobName,

    [Parameter()]
    [ValidateRange(1, 6)]
    [int]$StartFromStep = 1,

    [Parameter()]
    [System.Management.Automation.PSCredential]
    $ProxyCredential
)

# ==============================================================================
# Configuration -- adjust these as needed
# ==============================================================================
$TopSitesCount          = 10    # How many top-affected sites to display after Step 2
$PollIntervalSeconds    = 30    # Polling interval for repair session and backup job
$PostEnableDelaySeconds = 60    # Seconds to wait after re-enabling the job before starting it
$ModulePath             = "C:\Program Files\Veeam\Backup365\Veeam.Archiver.PowerShell.dll"
$ScriptDir              = if ($PSScriptRoot) { $PSScriptRoot } else { (Get-Location).Path }
$CredentialDir          = Join-Path $ScriptDir "Credentials"


# ==============================================================================
# Helper functions
# ==============================================================================

# Disable Windows Console Quick Edit mode so that clicking in the console window
# does not pause the process. Quick Edit causes the entire PowerShell process to
# freeze until the user presses Escape -- indistinguishable from a hung cmdlet.
Add-Type -Name ConsoleHelper -Namespace Win32 -MemberDefinition @'
    [DllImport("kernel32.dll", SetLastError = true)]
    public static extern IntPtr GetStdHandle(int nStdHandle);
    [DllImport("kernel32.dll", SetLastError = true)]
    public static extern bool GetConsoleMode(IntPtr hConsoleHandle, out uint lpMode);
    [DllImport("kernel32.dll", SetLastError = true)]
    public static extern bool SetConsoleMode(IntPtr hConsoleHandle, uint dwMode);
'@ -ErrorAction SilentlyContinue

function Disable-ConsoleQuickEdit {
    try {
        $handle = [Win32.ConsoleHelper]::GetStdHandle(-10)  # STD_INPUT_HANDLE
        $mode   = [uint32]0
        [Win32.ConsoleHelper]::GetConsoleMode($handle, [ref]$mode) | Out-Null
        $script:OriginalConsoleMode = $mode
        [Win32.ConsoleHelper]::SetConsoleMode($handle, ($mode -band (-bnot [uint32]0x0040))) | Out-Null
    } catch { }
}

function Restore-ConsoleQuickEdit {
    try {
        if ($null -ne $script:OriginalConsoleMode) {
            $handle = [Win32.ConsoleHelper]::GetStdHandle(-10)
            [Win32.ConsoleHelper]::SetConsoleMode($handle, $script:OriginalConsoleMode) | Out-Null
            $script:OriginalConsoleMode = $null
        }
    } catch { }
}

function Test-IsAdmin {
    $id = [Security.Principal.WindowsIdentity]::GetCurrent()
    $p  = New-Object Security.Principal.WindowsPrincipal($id)
    return $p.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
}

function Get-SafeFileName ([string]$Name) {
    foreach ($c in [IO.Path]::GetInvalidFileNameChars()) {
        $Name = $Name.Replace([string]$c, '_')
    }
    return $Name
}

function Write-Log {
    param(
        [string]$Message,
        [ValidateSet("INFO","WARN","ERROR")]
        [string]$Level = "INFO"
    )
    $ts   = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
    $line = "[$ts] [$Level] $Message"
    Write-Host $line
    Add-Content -Path $script:LogFile -Value $line
}

function Write-Section ([string]$Title) {
    $bar   = "=" * 80
    $block = @("", $bar, "  $Title", $bar)
    foreach ($line in $block) {
        Write-Host $line
        Add-Content -Path $script:LogFile -Value $line
    }
}

function Write-TableToScreenAndLog ([object[]]$Data, [string[]]$Properties = @()) {
    $table = if ($Properties.Count -gt 0) {
        $Data | Format-Table $Properties -AutoSize -Wrap | Out-String -Width 600
    } else {
        $Data | Format-Table -AutoSize -Wrap | Out-String -Width 600
    }
    Write-Host $table
    Add-Content -Path $script:LogFile -Value $table
}

# Returns a PSCredential for a proxy. Priority:
#   1. Credential file saved by Save-VBOProxyCredentials.ps1 (<CredentialDir>\<hostname>.xml)
#   2. -ProxyCredential parameter (stored in $script:ProxyCredential)
#   3. Interactive Get-Credential prompt (reused for subsequent proxies)
function Get-ProxyCredential ([string]$Hostname) {
    $credFile = Join-Path $CredentialDir "$Hostname.xml"
    if (Test-Path $credFile) {
        try {
            $cred = Import-Clixml -Path $credFile -ErrorAction Stop
            Write-Log "Loaded stored credentials for '$Hostname'."
            return $cred
        } catch {
            Write-Log "Could not load credential file for '$Hostname' ($credFile): $($_.Exception.Message)" -Level "WARN"
        }
    }

    if ($null -ne $script:ProxyCredential) {
        Write-Log "Using -ProxyCredential parameter for '$Hostname'."
        return $script:ProxyCredential
    }

    Write-Log "No stored credentials for '$Hostname'. Prompting..."
    $prompted = Get-Credential -Message "Enter credentials for proxy '$Hostname' (used for UNC log retrieval)"
    if ($null -ne $prompted) {
        $script:ProxyCredential = $prompted
    }
    return $prompted
}

# Called on all exit paths -- re-enables any jobs this script disabled
function Exit-Script ([int]$Code = 0) {
    if ($script:DisabledJobs.Count -gt 0) {
        Write-Log "Re-enabling $($script:DisabledJobs.Count) job(s) before exit..."
        foreach ($jName in @($script:DisabledJobs)) {
            try {
                $j = Get-VBOJob -Name $jName -ErrorAction Stop
                Enable-VBOJob -Job $j -ErrorAction SilentlyContinue | Out-Null
                Write-Log "  Re-enabled: $jName"
            } catch {
                Write-Log "  Could not re-enable '$jName'. Please re-enable manually in the Veeam console." -Level "WARN"
            }
        }
        $script:DisabledJobs.Clear()
    }
    Write-Log "Script exiting with code $Code."
    Restore-ConsoleQuickEdit
    try { Stop-Transcript | Out-Null } catch { }
    exit $Code
}

$script:DisabledJobs         = [System.Collections.Generic.List[string]]::new()
$script:LogFile              = $null   # set below once we have the job name
$script:ProxyCredential      = $ProxyCredential
$script:EligibleProxies      = @()
$script:OriginalConsoleMode  = $null


# ==============================================================================
# Job name -- parameter or interactive prompt
# ==============================================================================
if ([string]::IsNullOrWhiteSpace($JobName)) {
    $JobName = Read-Host "Enter the Veeam backup job name"
}
if ([string]::IsNullOrWhiteSpace($JobName)) {
    Write-Host "[ERROR] No job name provided. Exiting."
    exit 1
}


# ==============================================================================
# Log and transcript setup
# ==============================================================================
$SafeName  = Get-SafeFileName $JobName
$LogDir    = Join-Path (Get-Location).Path "Logs"

if (-not (Test-Path $LogDir)) {
    New-Item -Path $LogDir -ItemType Directory -Force | Out-Null
}

$script:LogFile = Join-Path $LogDir "${SafeName}.log"
$TranscriptFile = Join-Path $LogDir "Transcript_${SafeName}.log"

# Transcript is overwritten on each run
Start-Transcript -Path $TranscriptFile -Force | Out-Null

# Disable Quick Edit so clicking the console window does not freeze the process
Disable-ConsoleQuickEdit

# Append a run-start separator to the log
Add-Content -Path $script:LogFile -Value ""
Add-Content -Path $script:LogFile -Value ("*" * 80)
Add-Content -Path $script:LogFile -Value ("*  NEW RUN -- $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')  StartFromStep=$StartFromStep")
Add-Content -Path $script:LogFile -Value ("*" * 80)

Write-Section "Veeam VBO365 Repository Repair (KB4874) -- $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"
Write-Log "Job name:              $JobName"
Write-Log "Starting from step:    $StartFromStep"
Write-Log "Log file:              $($script:LogFile)"
Write-Log "Transcript:            $TranscriptFile"
Write-Log "Running as admin:      $(Test-IsAdmin)"
Write-Log "Top sites to display:  $TopSitesCount"
Write-Log "Post-enable delay:     ${PostEnableDelaySeconds}s"


# ==============================================================================
# Prerequisites -- module
# ==============================================================================
Write-Section "Checking Prerequisites"

if (-not (Test-Path $ModulePath)) {
    Write-Log "Veeam PowerShell module not found at: $ModulePath" -Level "ERROR"
    Write-Log "Ensure Veeam Backup for Microsoft 365 v8.5 is installed." -Level "ERROR"
    try { Stop-Transcript | Out-Null } catch { }
    exit 1
}

try {
    Import-Module $ModulePath -ErrorAction Stop
    Write-Log "Veeam PowerShell module imported successfully."
} catch {
    Write-Log "Failed to import Veeam module: $($_.Exception.Message)" -Level "ERROR"
    try { Stop-Transcript | Out-Null } catch { }
    exit 1
}


# ==============================================================================
# Locate the job
# ==============================================================================
Write-Section "Locating Backup Job"

$job = $null
try {
    $job = Get-VBOJob -Name $JobName -ErrorAction Stop
} catch {
    Write-Log "Could not find a job named '$JobName': $($_.Exception.Message)" -Level "ERROR"
    try { Stop-Transcript | Out-Null } catch { }
    exit 1
}
Write-Log "Found job: $($job.Name)"


# ==============================================================================
# Check job scope -- must include OneDrive, SharePoint, or Teams
# ==============================================================================
Write-Section "Checking Job Scope"

$items = $null
try {
    $items = @(Get-VBOBackupItem -Job $job -ErrorAction Stop)
} catch {
    Write-Log "Could not retrieve backup items for job '$JobName': $($_.Exception.Message)" -Level "ERROR"
    try { Stop-Transcript | Out-Null } catch { }
    exit 1
}

$hasOneDrive   = ($items | Where-Object { $_.OneDrive } | Measure-Object).Count -gt 0
$hasSharePoint = ($items | Where-Object { $_.Sites }    | Measure-Object).Count -gt 0
$hasTeams      = ($items | Where-Object { $_.Teams }    | Measure-Object).Count -gt 0

Write-Log "Job scope -- OneDrive: $hasOneDrive  |  SharePoint: $hasSharePoint  |  Teams: $hasTeams"

if (-not ($hasOneDrive -or $hasSharePoint -or $hasTeams)) {
    Write-Log "Job '$JobName' does not back up OneDrive, SharePoint, or Teams workloads." -Level "WARN"
    Write-Log "KB4874 remediation applies only to those workloads. No action taken." -Level "WARN"
    try { Stop-Transcript | Out-Null } catch { }
    exit 0
}
Write-Log "Job covers at least one applicable workload. Proceeding."


# ==============================================================================
# Get repository from job
# ==============================================================================
$repository = $job.Repository
if ($null -eq $repository) {
    Write-Log "Could not determine the repository from the job object. Verify the job has an assigned repository." -Level "ERROR"
    try { Stop-Transcript | Out-Null } catch { }
    exit 1
}
Write-Log "Repository: $($repository.Name)"
$DisabledJobsFile = Join-Path $LogDir "DisabledJobs_$(Get-SafeFileName $repository.Name).json"


# ==============================================================================
# Proxy detection and credential check
# ==============================================================================
Write-Section "Proxy Detection"

try {
    $allProxies = @(Get-VBOProxy -ErrorAction Stop)
    $script:EligibleProxies = @($allProxies | Where-Object {
        $_.OperatingSystemKind -eq 'Windows' -and
        $_.State               -eq 'Online'  -and
        $_.MaintenanceModeState -eq 'Disabled'
    })
    Write-Log "$($allProxies.Count) proxy/proxies configured; $($script:EligibleProxies.Count) eligible (Windows, online, not in maintenance)."
} catch {
    Write-Log "Could not enumerate proxies: $($_.Exception.Message)" -Level "WARN"
}

if ($script:EligibleProxies.Count -gt 0) {
    Write-Log "Credential source per proxy (checked at Step 6 log retrieval):"
    foreach ($proxy in $script:EligibleProxies) {
        $credFile = Join-Path $CredentialDir "$($proxy.Hostname).xml"
        $status   = if (Test-Path $credFile) {
            "credential file found"
        } elseif ($null -ne $script:ProxyCredential) {
            "will use -ProxyCredential parameter"
        } else {
            "no stored credentials -- will prompt if needed in Step 6"
        }
        Write-Log "  $($proxy.Hostname): $status"
    }
    Write-Log "To store proxy credentials in advance, run Save-VBOProxyCredentials.ps1."
} else {
    Write-Log "No eligible proxies found. Integrity logs will be read locally."
}


# ==============================================================================
# Pre-Step: Disable the job (only when running Steps 1-3)
# ==============================================================================
if ($StartFromStep -le 3) {
    Write-Section "Pre-Step: Disabling Backup Job"

    # Wait for any currently-running session to finish before disabling
    $activeSession = Get-VBOJobSession -Job $job -Last
    $jobRunningStatuses = @("Running", "Queued", "Updating")

    if ($null -ne $activeSession -and $activeSession.Status -in $jobRunningStatuses) {
        Write-Log "Job '$($job.Name)' is currently running (Status=$($activeSession.Status)). Waiting for it to finish before disabling..."
        Write-Host ""

        $firstPoll     = $true
        $pollLineCount = 0

        while ($true) {
            $activeSession = Get-VBOJobSession -Job $job -Last
            $timestamp     = Get-Date -Format "yyyy-MM-dd HH:mm:ss"

            $displayLines = @(
                "Timestamp  : $timestamp",
                "Job        : $($activeSession.JobName)",
                "Status     : $($activeSession.Status)",
                "Sync Type  : $($activeSession.JobSessionConfigType)"
            )

            if (-not $firstPoll -and $pollLineCount -gt 0) {
                try {
                    [System.Console]::SetCursorPosition(0, [System.Console]::CursorTop - $pollLineCount)
                } catch { }
            }
            foreach ($line in $displayLines) { Write-Host $line }
            $pollLineCount = $displayLines.Count
            $firstPoll     = $false

            Add-Content -Path $script:LogFile -Value "[$timestamp] [POLL] Waiting for running job -- Status=$($activeSession.Status)"

            if ($activeSession.Status -notin $jobRunningStatuses) {
                Write-Host ""
                Write-Log "Job session finished with status '$($activeSession.Status)'. Proceeding to disable."
                break
            }

            Start-Sleep -Seconds $PollIntervalSeconds
        }
    } else {
        Write-Log "Job '$($job.Name)' is not currently running. Proceeding to disable."
    }

    # Find all enabled jobs sharing this repository -- they must all be disabled
    Write-Log "Checking for all enabled jobs using repository '$($repository.Name)'..."
    $allRepoJobs = @()
    try {
        $allRepoJobs = @(Get-VBOJob -ErrorAction Stop | Where-Object {
            $_.Repository.Id -eq $repository.Id -and $_.IsEnabled
        })
    } catch {
        Write-Log "Could not enumerate jobs for repository: $($_.Exception.Message). Falling back to specified job only." -Level "WARN"
        $allRepoJobs = @($job)
    }

    if ($allRepoJobs.Count -gt 1) {
        Write-Log "$($allRepoJobs.Count) enabled jobs share this repository and will all be disabled:"
        foreach ($j in $allRepoJobs) { Write-Log "  $($j.Name)" }
    } else {
        Write-Log "1 enabled job uses this repository: $($job.Name)"
    }

    # Persist the list so a resumed run (StartFromStep >= 4) knows what to re-enable
    @{
        RunTime        = (Get-Date -Format 'yyyy-MM-dd HH:mm:ss')
        RepositoryName = $repository.Name
        DisabledJobs   = @($allRepoJobs | ForEach-Object { $_.Name })
    } | ConvertTo-Json | Set-Content -Path $DisabledJobsFile -Force -ErrorAction SilentlyContinue

    $anyDisableFailed = $false
    foreach ($j in $allRepoJobs) {
        try {
            Disable-VBOJob -Job $j -ErrorAction Stop | Out-Null
            $script:DisabledJobs.Add($j.Name)
            Write-Log "  Disabled: $($j.Name)"
        } catch {
            Write-Log "  Failed to disable '$($j.Name)': $($_.Exception.Message)" -Level "ERROR"
            $anyDisableFailed = $true
        }
    }

    if ($anyDisableFailed -and $script:DisabledJobs.Count -eq 0) {
        Write-Log "Could not disable any jobs. Aborting to avoid running repairs while jobs are active." -Level "ERROR"
        Exit-Script 1
    }

} else {
    Write-Log "Pre-Step: Job disable skipped (StartFromStep=$StartFromStep)."
}


# ==============================================================================
# Steps 1 and 2: Integrity check and review results
# ==============================================================================
if ($StartFromStep -le 2) {

    # ------------------------------------------------------------------
    # Step 1: Integrity check
    # ------------------------------------------------------------------
    Write-Section "Step 1: Running Integrity Check (Test-VBORepository)"
    Write-Log "This may take several hours depending on repository size."

    if (-not (Test-IsAdmin)) {
        Write-Log "WARNING: PowerShell is NOT running as administrator. If Test-VBORepository fails, close this session and re-run the script from an elevated (Run as Administrator) PowerShell window." -Level "WARN"
    }

    Write-Log "Running Test-VBORepository against '$($repository.Name)'..."

    $integrityErrors = @()
    try {
        Test-VBORepository -Repository $repository -ErrorVariable integrityErrors -ErrorAction SilentlyContinue | Out-Null
    } catch {
        $integrityErrors += $_
    }

    # Check whether the failure looks privilege-related
    if ($integrityErrors.Count -gt 0) {
        $combinedMessages = ($integrityErrors | ForEach-Object { "$_" }) -join " "
        if ((-not (Test-IsAdmin)) -and ($combinedMessages -match 'access.denied|denied|privilege|administrator|elevation|unauthorized')) {
            Write-Log "Test-VBORepository appears to have failed because the session is not elevated." -Level "ERROR"
            Write-Log "Please close this window, re-open PowerShell using 'Run as Administrator', and run the script again." -Level "ERROR"
            Exit-Script 1
        }
    }

    Write-Log "Integrity check complete. Reviewing results..."

    # ------------------------------------------------------------------
    # Step 2: Parse and review results
    # ------------------------------------------------------------------
    Write-Section "Step 2: Reviewing Integrity Check Results"

    $corruptedSites = [System.Collections.Generic.List[PSCustomObject]]::new()
    $corruptedLists = [System.Collections.Generic.List[PSCustomObject]]::new()

    foreach ($errObj in $integrityErrors) {
        $msg = "$errObj"

        if ($msg -match "The following site has corrupted file data:\s*(.+?)\s*\(.*?count:\s*(\d+)\)") {
            $siteUrl   = $Matches[1].Trim()
            $siteCount = [int]$Matches[2]
            $corruptedSites.Add([PSCustomObject]@{ Type = "Site"; Path = $siteUrl; Count = $siteCount })
        }

        if ($msg -match "The following list contains items referencing corrupted files:\s*(.+?)\s*\(.*?count:\s*(\d+)\)") {
            $listPath  = $Matches[1].Trim()
            $listCount = [int]$Matches[2]
            $corruptedLists.Add([PSCustomObject]@{ Type = "List"; Path = $listPath; Count = $listCount })
        }
    }

    $corruptionFound = ($corruptedSites.Count -gt 0) -or ($corruptedLists.Count -gt 0)

    if (-not $corruptionFound) {
        Write-Log "No corruption detected in repository '$($repository.Name)'."
        Write-Log "No further action is required."
        Exit-Script 0
    }

    $totalSiteFiles = ($corruptedSites | Measure-Object -Property Count -Sum).Sum
    $totalListItems = ($corruptedLists | Measure-Object -Property Count -Sum).Sum
    $totalAffected  = $totalSiteFiles + $totalListItems

    Write-Log "Corruption detected."
    Write-Log "  Corrupted sites:      $($corruptedSites.Count) site(s) -- $totalSiteFiles affected file(s)"
    Write-Log "  Corrupted lists:      $($corruptedLists.Count) list(s) -- $totalListItems affected item(s)"
    Write-Log "  Total affected items: $totalAffected"

    $allAffected = [System.Collections.Generic.List[PSCustomObject]]::new()
    foreach ($s in $corruptedSites) { $allAffected.Add($s) }
    foreach ($l in $corruptedLists) { $allAffected.Add($l) }

    $topAffected = @($allAffected | Sort-Object Count -Descending | Select-Object -First $TopSitesCount)

    $topHeader = "Top $TopSitesCount most-affected sites/locations (by corrupted item count):"
    Write-Host ""
    Write-Host $topHeader
    Add-Content -Path $script:LogFile -Value ""
    Add-Content -Path $script:LogFile -Value $topHeader

    Write-TableToScreenAndLog $topAffected -Properties @("Type","Count","Path")

} else {
    Write-Log "Steps 1-2: Integrity check skipped (StartFromStep=$StartFromStep)."
}


# ==============================================================================
# Step 3: Start repair session
# ==============================================================================
if ($StartFromStep -le 3) {

    Write-Section "Step 3: Scanning Repository and Flagging Affected Items"
    Write-Log "Starting full repair session on repository '$($repository.Name)'..."
    Write-Log "Note: The repository is locked for all backups and restores during this step."
    Write-Log "      Ensure all backup and restore sessions for this repository are closed."

    $repairSession = $null
    try {
        $repairSession = Start-VBORepositoryRepairSession `
            -Repository $repository `
            -Type CleanMissingFilesData `
            -Mode Full `
            -Scope Full `
            -ErrorAction Stop
        Write-Log "Repair session started. Session ID: $($repairSession.Id)"
    } catch {
        Write-Log "Failed to start repair session: $($_.Exception.Message)" -Level "ERROR"
        Exit-Script 1
    }

    Write-Log "Polling repair session every $PollIntervalSeconds seconds. Waiting for completion..."
    Write-Host ""

    # Non-terminal states -- session is still starting up or running
    $repairNonTerminal = @("Running", "Preparing", "Queued")

    $firstPoll     = $true
    $pollLineCount = 0

    while ($true) {
        $timestamp    = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
        $sessionState = $null

        try {
            $sessionState = Get-VBORepositoryRepairSession -Id $repairSession.Id -ErrorAction Stop
        } catch {
            Add-Content -Path $script:LogFile -Value "[$timestamp] [WARN] State query failed: $($_.Exception.Message) -- retrying in ${PollIntervalSeconds}s."
            Write-Host "[$timestamp] [WARN] State query failed: $($_.Exception.Message). Retrying in ${PollIntervalSeconds}s..."
            Start-Sleep -Seconds $PollIntervalSeconds
            continue
        }

        if ($null -eq $sessionState) {
            Add-Content -Path $script:LogFile -Value "[$timestamp] [WARN] State query returned no result -- retrying in ${PollIntervalSeconds}s."
            Write-Host "[$timestamp] [WARN] State query returned no result. Retrying in ${PollIntervalSeconds}s..."
            Start-Sleep -Seconds $PollIntervalSeconds
            continue
        }

        $displayLines = @(
            "Timestamp  : $timestamp",
            "Id         : $($sessionState.Id)   ",
            "Type       : $($sessionState.Type)   ",
            "Repository : $($sessionState.RepositoryId)   ",
            "State      : $($sessionState.State)   ",
            "Error      : $($sessionState.ErrorMessage)   "
        )

        if (-not $firstPoll -and $pollLineCount -gt 0) {
            try {
                [System.Console]::SetCursorPosition(0, [System.Console]::CursorTop - $pollLineCount)
            } catch { }
        }
        foreach ($line in $displayLines) { Write-Host $line }
        $pollLineCount = $displayLines.Count
        $firstPoll     = $false

        Add-Content -Path $script:LogFile -Value "[$timestamp] [POLL] Repair -- State=$($sessionState.State)  Error=$($sessionState.ErrorMessage)"

        if ($sessionState.State -notin $repairNonTerminal) {
            Write-Host ""
            if ($sessionState.State -eq "Finished" -and [string]::IsNullOrEmpty($sessionState.ErrorMessage)) {
                Write-Log "Repair session completed successfully."
            } else {
                Write-Log "Repair session ended with state '$($sessionState.State)'." -Level "WARN"
                if (-not [string]::IsNullOrEmpty($sessionState.ErrorMessage)) {
                    Write-Log "Session error: $($sessionState.ErrorMessage)" -Level "ERROR"
                    Write-Log "Stopping -- repair session reported an error. Review the error above before retrying." -Level "ERROR"
                    Exit-Script 1
                }
            }
            break
        }

        Start-Sleep -Seconds $PollIntervalSeconds
    }

} else {
    Write-Log "Step 3: Repair session skipped (StartFromStep=$StartFromStep)."
}


# ==============================================================================
# Step 4: Re-enable backup jobs
# ==============================================================================
Write-Section "Step 4: Re-enabling Backup Jobs"

# Determine which jobs to re-enable -- three sources in priority order:
#   1. This run disabled them      -> $script:DisabledJobs (in memory)
#   2. Resuming with a state file  -> Logs\DisabledJobs_<repo>.json
#   3. No record at all            -> all currently-disabled jobs for this repo (with warning)
$jobNamesToEnable = [System.Collections.Generic.List[string]]::new()

if ($script:DisabledJobs.Count -gt 0) {
    Write-Log "This run disabled $($script:DisabledJobs.Count) job(s) -- re-enabling them."
    foreach ($n in $script:DisabledJobs) { $jobNamesToEnable.Add($n) }

} elseif (Test-Path $DisabledJobsFile) {
    try {
        $savedState = Get-Content $DisabledJobsFile -Raw | ConvertFrom-Json
        Write-Log "State file found (written at $($savedState.RunTime)) -- re-enabling $($savedState.DisabledJobs.Count) recorded job(s)."
        foreach ($n in $savedState.DisabledJobs) { $jobNamesToEnable.Add($n) }
    } catch {
        Write-Log "Could not read state file '$DisabledJobsFile': $($_.Exception.Message)" -Level "WARN"
    }
}

if ($jobNamesToEnable.Count -eq 0) {
    Write-Log "No record of which jobs were disabled. Checking all jobs for repository '$($repository.Name)'..." -Level "WARN"
    try {
        $disabledRepoJobs = @(Get-VBOJob -ErrorAction Stop | Where-Object {
            $_.Repository.Id -eq $repository.Id -and -not $_.IsEnabled
        })
        if ($disabledRepoJobs.Count -gt 0) {
            Write-Log "$($disabledRepoJobs.Count) currently-disabled job(s) found for this repository:"
            foreach ($j in $disabledRepoJobs) {
                Write-Log "  $($j.Name)"
                $jobNamesToEnable.Add($j.Name)
            }
            Write-Log "These will be re-enabled. If any were disabled before this repair, re-disable them manually afterwards." -Level "WARN"
        } else {
            Write-Log "No disabled jobs found for this repository -- they may already be enabled."
        }
    } catch {
        Write-Log "Could not query jobs for repository: $($_.Exception.Message)" -Level "WARN"
    }
}

$anyEnabled = $false
foreach ($jName in $jobNamesToEnable) {
    try {
        $j = Get-VBOJob -Name $jName -ErrorAction Stop
        if ($j.IsEnabled) {
            Write-Log "  $jName -- already enabled, skipping."
        } else {
            Enable-VBOJob -Job $j -ErrorAction Stop | Out-Null
            Write-Log "  Re-enabled: $jName"
            $anyEnabled = $true
        }
        $script:DisabledJobs.Remove($jName) | Out-Null
    } catch {
        Write-Log "  Failed to re-enable '$jName': $($_.Exception.Message)" -Level "ERROR"
    }
}

if ($anyEnabled) {
    Write-Log "Waiting $PostEnableDelaySeconds seconds for repository to exit maintenance mode before starting backup..."
    $waitEnd = (Get-Date).AddSeconds($PostEnableDelaySeconds)
    while ((Get-Date) -lt $waitEnd) {
        $remaining = [int]($waitEnd - (Get-Date)).TotalSeconds
        Write-Host "`r  $remaining seconds remaining...    " -NoNewline
        Start-Sleep -Seconds 1
    }
    Write-Host ""
    Write-Log "Wait complete. Proceeding to Step 5."
} else {
    Write-Log "Step 4: All jobs already enabled -- no countdown needed."
}


# ==============================================================================
# Step 5: Run all backup jobs for this repository
# ==============================================================================
if ($StartFromStep -le 5) {

    Write-Section "Step 5: Running Backup Jobs"

    # Run every enabled job that shares this repository, not just the specified one
    $repoJobsToRun = @()
    try {
        $repoJobsToRun = @(Get-VBOJob -ErrorAction Stop | Where-Object {
            $_.Repository.Id -eq $repository.Id -and $_.IsEnabled
        })
    } catch {
        Write-Log "Could not enumerate jobs for repository: $($_.Exception.Message). Running specified job only." -Level "WARN"
        $repoJobsToRun = @($job)
    }

    if ($repoJobsToRun.Count -eq 0) {
        Write-Log "No enabled jobs found for repository '$($repository.Name)'. Skipping backup run." -Level "WARN"
    } else {
        Write-Log "$($repoJobsToRun.Count) job(s) will be run sequentially for repository '$($repository.Name)':"
        foreach ($j in $repoJobsToRun) { Write-Log "  $($j.Name)" }
    }

    $backupTerminal = @("Success", "Warning", "Failed", "Stopped", "NotConfigured")

    foreach ($runJob in $repoJobsToRun) {

        Write-Log ""
        Write-Log "Starting backup job '$($runJob.Name)'..."

        try {
            Start-VBOJob -Job $runJob -ErrorAction Stop | Out-Null
            Write-Log "Backup job started."
        } catch {
            Write-Log "Failed to start backup job '$($runJob.Name)': $($_.Exception.Message)" -Level "ERROR"
            Exit-Script 1
        }

        # Brief pause to allow the session record to be created before first poll
        Start-Sleep -Seconds 5

        Write-Log "Polling '$($runJob.Name)' every $PollIntervalSeconds seconds. Waiting for completion..."
        Write-Host ""

        $firstPoll     = $true
        $pollLineCount = 0

        while ($true) {
            $timestamp      = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
            $currentSession = $null

            try {
                $currentSession = Get-VBOJobSession -Job $runJob -Last -ErrorAction Stop
            } catch {
                Write-Host "[$timestamp] [WARN] Session query failed: $($_.Exception.Message). Retrying in ${PollIntervalSeconds}s..."
                Add-Content -Path $script:LogFile -Value "[$timestamp] [WARN] Session query failed: $($_.Exception.Message) -- retrying."
                Start-Sleep -Seconds $PollIntervalSeconds
                continue
            }

            if ($null -eq $currentSession) {
                Write-Host "[$timestamp] Waiting for backup session to initialise..."
                Add-Content -Path $script:LogFile -Value "[$timestamp] [INFO] Job session not yet started -- waiting."
                Start-Sleep -Seconds $PollIntervalSeconds
                continue
            }

            $displayLines = @(
                "Timestamp  : $timestamp",
                "Job        : $($currentSession.JobName)",
                "Status     : $($currentSession.Status)",
                "Sync Type  : $($currentSession.JobSessionConfigType)"
            )

            if (-not $firstPoll -and $pollLineCount -gt 0) {
                try {
                    [System.Console]::SetCursorPosition(0, [System.Console]::CursorTop - $pollLineCount)
                } catch { }
            }
            foreach ($line in $displayLines) { Write-Host $line }
            $pollLineCount = $displayLines.Count
            $firstPoll     = $false

            Add-Content -Path $script:LogFile -Value "[$timestamp] [POLL] Backup '$($runJob.Name)' -- Status=$($currentSession.Status)  SyncType=$($currentSession.JobSessionConfigType)"

            if ($currentSession.Status -in $backupTerminal) {
                Write-Host ""
                Write-Log "Backup job '$($runJob.Name)' finished. Final status: $($currentSession.Status)"
                if ($currentSession.Status -eq "Failed") {
                    Write-Log "Job reported a failure. Check the Veeam console for details. Proceeding." -Level "WARN"
                }
                break
            }

            Start-Sleep -Seconds $PollIntervalSeconds
        }
    }

} else {
    Write-Log "Step 5: Backup job run skipped (StartFromStep=$StartFromStep)."
}


# ==============================================================================
# Step 6: Final verification and identify unrecoverable files
# ==============================================================================
Write-Section "Step 6: Final Verification (Test-VBORepository -DetailedOutput)"
Write-Log "Running detailed integrity check. This downloads file metadata from M365 and may take time."

# Record time before running so we can identify the log file this run produces
$verificationStartTime = Get-Date

try {
    Test-VBORepository -Repository $repository -DetailedOutput -ErrorAction SilentlyContinue 2>&1 | Out-Null
    Write-Log "Detailed verification complete."
} catch {
    Write-Log "Test-VBORepository (detailed) noted: $($_.Exception.Message)" -Level "WARN"
    Write-Log "The detailed log file should still have been written to disk." -Level "WARN"
}

# Locate the integrity log written during this run.
# Logs are written on whichever machine ran the verification -- the backup server
# when no proxy is used, or the proxy otherwise. If found on a proxy the log is
# copied to the local Logs folder so processing always works on a local file.
$veeamLogRelPath = "Veeam\Backup365\Logs\IntegrityVerification\$($repository.Name)"
$logFilter       = "IntegrityVerification_$($repository.Name)_*.log"
$latestLog       = $null

# Helper: find the most recent qualifying log in a given directory
function Find-LatestLog ([string]$Dir) {
    Get-ChildItem -Path $Dir -Filter $logFilter -ErrorAction SilentlyContinue |
        Where-Object { $_.LastWriteTime -ge $verificationStartTime } |
        Sort-Object LastWriteTime -Descending |
        Select-Object -First 1
}

# 1. Try local path
$localLogDir = Join-Path $env:ProgramData $veeamLogRelPath
Write-Log "Checking local integrity log path: $localLogDir"

if (Test-Path $localLogDir -ErrorAction SilentlyContinue) {
    $latestLog = Find-LatestLog $localLogDir
    if ($null -ne $latestLog) {
        Write-Log "Integrity log found locally: $($latestLog.FullName)"
    }
}

# 2. If not found locally, check proxies
if ($null -eq $latestLog) {
    if ($script:EligibleProxies.Count -gt 0) {
        Write-Log "Log not found locally. Checking $($script:EligibleProxies.Count) proxy/proxies..."

        $driveName = "VBOLog"

        foreach ($proxy in $script:EligibleProxies) {
            $proxyHost = $proxy.Hostname
            Write-Log "Checking proxy '$proxyHost'..."

            $cred      = Get-ProxyCredential $proxyHost
            $connected = $false

            try {
                Remove-PSDrive -Name $driveName -Force -ErrorAction SilentlyContinue

                if ($null -ne $cred) {
                    New-PSDrive -Name $driveName -PSProvider FileSystem `
                        -Root "\\$proxyHost\c$" `
                        -Credential $cred `
                        -ErrorAction Stop | Out-Null
                    $connected = $true
                    $searchDir = "${driveName}:\ProgramData\$veeamLogRelPath"
                } else {
                    $searchDir = "\\$proxyHost\c$\ProgramData\$veeamLogRelPath"
                }

                if (Test-Path $searchDir -ErrorAction SilentlyContinue) {
                    $remoteLog = Find-LatestLog $searchDir
                    if ($null -ne $remoteLog) {
                        Write-Log "Found log on proxy '$proxyHost': $($remoteLog.Name)"
                        Write-Log "Copying to local Logs folder for processing..."
                        $localCopyPath = Join-Path $LogDir $remoteLog.Name
                        Copy-Item -Path $remoteLog.FullName -Destination $localCopyPath -Force -ErrorAction Stop
                        $latestLog = Get-Item $localCopyPath
                        Write-Log "Log copied to: $localCopyPath"
                        break
                    }
                }
            } catch {
                Write-Log "Could not access proxy '$proxyHost': $($_.Exception.Message)" -Level "WARN"
            } finally {
                if ($connected) {
                    Remove-PSDrive -Name $driveName -Force -ErrorAction SilentlyContinue
                }
            }
        }
    } else {
        Write-Log "No eligible proxies to check." -Level "WARN"
    }
}

if ($null -eq $latestLog) {
    Write-Log "No integrity verification log for repository '$($repository.Name)' written since $($verificationStartTime.ToString('yyyy-MM-dd HH:mm:ss')) found locally or on any proxy." -Level "WARN"
    Write-Log "Expected local path: $localLogDir" -Level "WARN"
    Exit-Script 0
}

Write-Log "Scanning log file: $($latestLog.FullName)"

# Stream the log with Select-String (avoids loading the whole file into memory).
# Match on the rarer marker first, then confirm the second on the same line.
$unrecoverableLines = @(
    Select-String -Path $latestLog.FullName -Pattern 'RepairHistory: FileWasReset' -ErrorAction Stop |
        Where-Object { $_.Line -match 'Backed-up file is empty' } |
        ForEach-Object { $_.Line }
)

if ($unrecoverableLines.Count -eq 0) {
    Write-Log "No unrecoverable files found. All affected files were successfully redownloaded."
    Exit-Script 0
}

Write-Section "Unrecoverable Files -- $($unrecoverableLines.Count) file(s) could not be recovered"
Write-Log "$($unrecoverableLines.Count) file(s) remain empty in the backup. These files no longer exist in Microsoft 365 and cannot be restored."

$parsedFiles = foreach ($line in $unrecoverableLines) {
    $orgId    = 'N/A'
    $siteId   = 'N/A'
    $webId    = 'N/A'
    $intWebId = 'N/A'
    $fileId   = 'N/A'
    $version  = 'N/A'

    if ($line -match 'OrganizationId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})') { $orgId    = $Matches[1] }
    if ($line -match 'SiteId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})')         { $siteId   = $Matches[1] }
    if ($line -match '(?<!Internal)WebId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})') { $webId = $Matches[1] }
    if ($line -match 'InternalWebId:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})')  { $intWebId = $Matches[1] }
    if ($line -match 'file: Id:\s*([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})')       { $fileId   = $Matches[1] }
    if ($line -match 'Version:\s*(\d+)')                                                                                    { $version  = $Matches[1] }

    [PSCustomObject]@{
        OrganizationId = $orgId
        SiteId         = $siteId
        WebId          = $webId
        InternalWebId  = $intWebId
        FileId         = $fileId
        Version        = $version
    }
}

Write-TableToScreenAndLog $parsedFiles

Write-Log "For further assistance, raise a Veeam Support case: https://www.veeam.com/kb1771"


# ==============================================================================
# Done
# ==============================================================================
Write-Section "Script Complete -- $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"
Write-Log "All steps completed. Review the full log at: $($script:LogFile)"

Restore-ConsoleQuickEdit
try { Stop-Transcript | Out-Null } catch { }

[aeph]

Any update on this regarding S3-Immutability support?

[Polina]

Hi aeph,

It's expected around mid-August.

[sumeet]

Hi all,

Is it required to apply this fix of running the script to go back into the old restore points to fix the backup ??

I may sound crazy, but VBM365 backup does not have Dr Stranges time stone to back in time to fix the issue.
What it is doing is that using the latest content of the sharepoint sites and applying that in all the previous restore points. What is the point of doing this?

If I do this for my sites (for the ones that have issues) and if I require a restore from 6 months ago, wouldn't the restore be same as using the restore point of the day when I upgraded to v8.5 and ran this script?
This sounds absurd, but is this not correct?

I feel the best approach is to run the commands to identify the site/s and make a note of them. The best case scenario is no site gets listed - but best case.
But if we get sites, make a note in a case or somewhere to know the date and time of v8.5 upgrade and the sites and hope and pray that old restore is never required.

Or have I completely got this wrong - I hope I have got this wrong and VBM365 is able to find the site as it was being changed each day of the last 6-8-12 months since the issue was occuring.

[DaStivi]

For my understanding:

At some point in time, a restore point may have become corrupted. The affected SharePoint/OneDrive/Teams data was backed up successfully from a job perspective, but the stored content consists only of HTML placeholder data instead of the actual file content.

The underlying issue has been fixed, but there is no automatic mechanism to identify all restore points that were affected while the bug existed.

Once the issue is fixed and the source data changes again, the updated content is downloaded correctly and newly created restore points are no longer affected. However, the restore points created between the introduction of the bug and the successful re-download of the affected item remain corrupted and cannot be repaired retroactively.

Perhaps older product versions could have been used to retrieve earlier versions of the content, but since v8.4 the recommendation is essentially to keep only the latest data copy available. Ironically, that also removes a potential workaround in future cases like this.

If the source data has been deleted in the meantime, there is no longer any way to download the original, correct content again because the source object no longer exists. In a restore scenario, this means that only the last known good restore point can be used to recover the data. Any restore points created after the corruption was introduced but before the data was successfully re-downloaded will still contain the corrupted content.

[DaStivi]

I've just run this against my first customers (6TB) repo:

Step 2: Reviewing Integrity Check Results
================================================================================
[2026-08-04 09:34:53] [INFO] Corruption detected.
[2026-08-04 09:34:53] [INFO] Corrupted sites: 183 site(s) -- 27187 affected file(s)
[2026-08-04 09:34:53] [INFO] Corrupted lists: 252 list(s) -- 21061 affected item(s)
[2026-08-04 09:34:53] [INFO] Total affected items: 48248

[aeph]

Thats a lot.

Does anyone have any experience with how long it takes to run the fix command? I have organizations with 100 TB of data. Are there any benchmarks for this?
Haven't run the command because all our customers use immutable s3 object storage.

[DaStivi]

2nd customers I'll test:

[2026-08-04 12:10:34] [INFO] Corruption detected.
[2026-08-04 12:10:34] [INFO] Corrupted sites: 146 site(s) -- 4425 affected file(s)
[2026-08-04 12:10:34] [INFO] Corrupted lists: 158 list(s) -- 2650 affected item(s)
[2026-08-04 12:10:34] [INFO] Total affected items: 7075

Again, quite a lot of affected items...

I had really hoped this wasn't such a common and widespread issue, but it seems to be much larger than I initially imagined.

The repair job for the first tenant ran for about 2 hours on a 4 TB backup repository (1,157 objects). The full backup run for the repair of that tenant took roughly 1 hour.

The next customer in line already has an 8.5 TB repository, and my largest environments are well above 400 TB, so I was a bit concerned when I saw the number of affected objects for the bigger customer this is exponentially higher then too!

One thing I'm actually happy about for now is that I run backups every 2 hours, which creates a very large number of restore points. This decision has its downsides, especially for operations that need to scan all restore points or restores, and I initially assumed the repair process would suffer from the same problem.

Fortunately, it seems the repair job is not significantly impacted by the number of restore points, and the scan/repair times remain within a manageable range so far.

[cw1972]

I found this post yesterday and remembered that I had this issue restoring an excel file, when I opened it in notepad I saw it was an HTML so I started with the instructions in https://www.veeam.com/kb4874

I have a lot of corrupted data...

Test-VBORepository: The following site contains 12 corrupted items
Test-VBORepository: The following site contains 117 corrupted items

etc. etc.

This is just one repository, I've not even checked the other yet.

So I though, ok I'll run the command to fix this repository:

$session = Start-VBORepositoryRepairSession -Repository $repository -Type CleanMissingFilesData -Mode Full -Scope Full

and got error message:

Start-VBORepositoryRepairSession: A parameter cannot be found that matches parameter name 'Mode'.

tried removing mode

Start-VBORepositoryRepairSession: A parameter cannot be found that matches parameter name 'Scope'.

I am using Veeam 365 8.5.0.1014 P20260622

I have launched VBR Powershell directly from the application and also from the created shortcut (and as administrator)

I have tried:

# 1. Unload any existing module data to avoid cross-version contamination
Remove-Module Veeam.Archiver.PowerShell -ErrorAction SilentlyContinue

# 2. Force-load the absolute 8.5 binary file path directly
Import-Module "C:\Program Files\Veeam\Backup365\Veeam.Archiver.PowerShell.dll" -Force

Still no, go - what am I doing wrong - I need to fix these repositories.

[b.vanhaastrecht]

@b.vanhaastrecht I expect closer to the end of this month.

This estimation did not come thru. Do you have a more exact date for this to be released?

[DaStivi]

and got error message:

Start-VBORepositoryRepairSession: A parameter cannot be found that matches parameter name 'Mode'.

what does

Get-Command Start-VBORepositoryRepairSession

give you?

my output of a working environment looks like this:

CommandType Name Version Source
----------- ---- ------- ------
Cmdlet Start-VBORepositoryRepairSession 13.5.0.10… Veeam.Archiver.PowerShell

maybe you'll run some old/broken powershell sessions/modules

[cw1972]

get-command Start-VBORepositoryRepairSession

CommandType Name Version Source
----------- ---- ------- ------
Cmdlet Start-VBORepositoryRepairSession 13.5.0.10… Veeam.Archiver.PowerShell

seems to be the same

full list of the command:

get-command Start-VBORepositoryRepairSession | fl

Name : Start-VBORepositoryRepairSession
CommandType : Cmdlet
Definition :
Start-VBORepositoryRepairSession -Repository <VBORepository> -Type <VBORepositoryRepairType>
[-WaitForSessionTimeout <timespan>] [-ForceStopSession] [-ForceStopSessionTimeout <timespan>]
[<CommonParameters>]

Path :
DLL : C:\Program Files\Veeam\Backup365\Veeam.Archiver.PowerShell.dll
HelpFile : Veeam.Archiver.PowerShell.dll-Help.xml
ParameterSets : {-Repository <VBORepository> -Type <VBORepositoryRepairType> [-WaitForSessionTimeout <timespan>]
[-ForceStopSession] [-ForceStopSessionTimeout <timespan>] [<CommonParameters>]}
ImplementingType : Veeam.Archiver.PowerShell.Cmdlets.RepositoryRepairSession.StartVBORepositoryRepairSession
Verb : Start
Noun : VBORepositoryRepairSession

I started the Repository Repair without the -Mode and -Scope parameters and it accepted that, will see what happens when it finishes:

Get-VBORepositoryRepairSession

cmdlet Get-VBORepositoryRepairSession at command pipeline position 1
Supply values for the following parameters:
Id:
Id: bf45dee4-50a3-44a3-8f28-f41536075105

Id : bf45dee4-50a3-44a3-8f28-f41536075105
Type : CleanMissingFilesData
RepositoryId : 1122a439-b772-4c7a-b9d4-326cbd58578d
State : Running

that has been running for a few hours now, with no other progress indicator that State: Running

[DaStivi]

how big is your repository and what is it backed from? hdd/ssd? s3?

my two bigger ones until now:
8.5TB, only ~7000 affected items, run for 2 1/2h
7.3TB but 303.551 affected items run for 8h !

everything is on nvme-ssd backed netapp ontap S3!

[cw1972]

how big is your repository and what is it backed from? hdd/ssd? s3?

my two bigger ones until now:
8.5TB, only ~7000 affected items, run for 2 1/2h
7.3TB but 303.551 affected items run for 8h !

everything is on nvme-ssd backed netapp ontap S3!

all but two of my repositories are local storage presented as ISCSI from a NAS with SSDs over a 10gbe link, backups usually run around 25-35MB/s, the Test-VBORepository and Start-VBORepositoryRepairSession commands to these repositories only seem to achieve 3-5MB/s though.

the two object storage respositories I have actually require the -Mode parameter and prompt me for it, these are across a 400Mbit WAN link to a NAS with SSDs and the Test-VBORepository and Start-VBORepositoryRepairSession commands seem to achieve around 30-40MB/s processing

so I making my way through repairing my repositories now

[msobm]

This estimation did not come thru. Do you have a more exact date for this to be released?

I also would like this to know!