Catching broken Veeam Instances and other failures

[quinnvanorder]

I work for a MSP that needs to administer hundreds of Veeam instances across hundreds of networks. While event log monitoring is good to a point, as is the various Veeam tools, we have seen instances where Veeam damage or mis-configuration causes service interruptions.

As such, I built a script that is designed to be dropped on each Veeam instance, provided with a few arguments, and then will dump a txt file of any statuses that require attention.

It is designed to work with standard Veeam Jobs, Veeam Backup Copies (offsites), "Good Agents" (Able to be controlled via Veeam, only works properly on U4), and "Bad Agents" (Pointed at Veeam, not able to be controlled via console)

It catches the following cases:

• Veeam Backup service missing

• PowershellSnapin failing to load

• No Jobs Enabled or configured

• No successful runs in past <Specified audit interval>

• Quantity of jobs with a last state of failed (Ignores audit interval)

• Quantity of jobs in <Specified audit interval> that failed

• Quantity of errors in the Veeam event log within <Specified audit interval>

• Count of "bad" agents, meaning cant be controlled via Veeam.

• Count of disabled jobs

It also supports an audit window override file. In our case, 99% of the time, it is a safe assumption that at least one job should have succeeded every 24 hours. However some clients have jobs only running once a week. If the override file is present, the text value within it will replace the provided input of hours to audit. Note that an Override will denote itself by beginning the output string with "[24]" or whatever the provided numerical value is

We utilize a program called Kaseya to manage our endpoints, so it drops the PS1 in place, executes it, and then pulls the value of the output text file into a column. This can be useful for both finding things that event log monitoring misses, but also to provide a real-time glimpse of what is currently failed, which can help prioritize currently failed jobs over jobs that failed earlier but recovered. Inversely, it can be useful to see instances where everything is currently working, but throughout the last audit period it had x failures or errors, as this can indicate problems that arent world ending, but require attention.

Not sure if this would be useful for others, but thought I would pass it along in case!

Syntax is VeeamAudit.ps1 <Hours to audit, we default to 24> <Root directory, namely the root of where the "VeeamAuditOverride.txt" would be if present >

Note the "DEV" function call at the bottom. To play with this, un-comment that line and run it, it will display first the time in seconds for each query, and then when complete, every single data value recorded. There is a lot more data gathered than is alerted upon, as I wanted this to be trivially expandable to report on other things should the need arise. For instance, there is a functioning but unused string that holds a list of all disabled jobs that can be added to the output.

Let me know if I made any stupid mistakes in this, or otherwise overlooked easier ways to achieve the same thing, always open to learn better ways to do things!

Code: Select all

#Quinn Van Order 2019
If(test-path 'C:\Temp\VeeamLogs.txt'){Remove-Item 'C:\Temp\VeeamLogs.txt' -Force} #Clean up line for multiple test re-runs. Verifies the output txt file is destroyed prior to run to prevent improper appending. 

###VARIABLES### 

#ARGUMENTS:
[int]$Global:AuditHours = $args[0]
[String]$Global:Root = $args[1]
#Logging
$Global:LogFile = 'C:\Temp\VeeamLogs.txt'

#Potential Override
$Global:Override = $Global:Root + '\VeeamAuditOverride.txt'
IF(test-path $Global:Override)
{
    [int]$Global:AuditHours = get-content $Global:Override
    $Global:Output = '[' + $Global:AuditHours + '] '
}

#Arrays of job data
$Global:VBREPJob = ''
$Global:VBRJob = ''
$Global:VBRBackupSession = ''


#####Counters#####
#Counters - Veeam  
[int]$Global:CountVeeamEnabled = 0 #Jobs directly through the BDR
[int]$Global:CountVeeamDisabled = 0 #Jobs currently disabled. 
[int]$Global:CountVeeamStateSuccess = 0 #Jobs last complete state is 'Success'
[int]$Global:CountVeeamStateFailed = 0 #Jobs last complete state is 'Failed'
[int]$Global:CountVeeamTotalSuccess = 0 #Count of 'Success' runs in past 24 hours
[int]$Global:CountVeeamTotalFailed = 0 #Count of 'Failed' runs in past 24 hours

#Counters - Good Agent
[int]$Global:CountGoodAgentEnabled = 0 #Good Agent meaning controlled by the BDR
[int]$Global:CountGoodAgentDisabled = 0
[int]$Global:CountGoodAgentStateSuccess = 0
[int]$Global:CountGoodAgentStateFailed = 0
[int]$Global:CountGoodAgentTotalSuccess = 0
[int]$Global:CountGoodAgentTotalFailed = 0

#Counters - Bad Agent
[int]$Global:CountBadAgentEnabled = 0 #Bad Agent meaning controlled via the agent, not the BDR
[int]$Global:CountBadAgentDisabled = 0
[int]$Global:CountBadAgentSuccess = 0
[int]$Global:CountBadAgentFailed = 0

#Counters - Offsite
[int]$Global:CountOffsiteEnabled = 0 
[int]$Global:CountOffsiteDisabled = 0
[int]$Global:CountOffsiteSuccess = 0
[int]$Global:CountOffsiteFailed = 0
[int]$Global:CountOffsiteTotalSuccess = 0
[int]$Global:CountOffsiteTotalFailed = 0

#Counters - Totals
[int]$Global:TotalEnabled = 0 #Count of 0 indicates BDR is not being used
[int]$Global:TotalDisabled = 0
[int]$Global:TotalStateSuccess = 0 
[int]$Global:TotalStateFailed = 0 
[int]$Global:Total24HourSuccess = 0 
[int]$Global:Total24HourFailed = 0 
[int]$Global:CountErrors = 0

#Strings
$Global:DisabledJobs = '' #Lists all jobs that are currently disabled. 

#Checks
$Global:BackupServicePresent = '' #If set to 'False', call GenerateOutput and terminate, device may have some of veeam, but is not functioning as BDR.  "Failed: 'Veeam Backup Service' not present. Is device BDR?"
$Global:SnapinLoaded = '' #If set to 'False' Device has damaged software components, call GenerateOutput and terminate. 



#Functions
########################################################################################################################

####################### Loading #######################  
Function CountErrors #Count of all errors in the Veeam logs within the last 24 hours. Second entry in the output string if present
{
    $Errors = @(Get-EventLog -Log 'Veeam Backup' -EntryType Error -After (Get-Date).addhours(-$Global:AuditHours) -ErrorAction SilentlyContinue ).count
    IF ($Errors -gt 0)
    {
        #$Errors = "{0:D3}" -f [int]$Errors #Formats the Errors string to 3 digits
        $Global:CountErrors += $Errors
    }
}

Function ServiceCheck #Verifies that the primary backup service is present. If not, device may not be a BDR, but instead could be a repository which wouldnt have jobs. If primary service is present, verifies all Veeam services are started prior to moving on
{
    $Services = get-service -displayname "Veeam Backup Service" -ErrorAction SilentlyContinue
    IF (($Services))
    {
        get-service -displayname 'veeam*' | Where {$_.StartType -eq 'Automatic'} | foreach-object { Start-Service -InputObj $_.Name }
        $Global:BackupServicePresent = 'True'
    }
    ELSE
    {
        
        $Global:BackupServicePresent = 'False'
    }
}

Function LoadSnapIn #Loads Powershell snapin, reports on failure
{
    Add-PSSnapin VeeamPSSnapin -ErrorVariable SnapIn -ErrorAction SilentlyContinue
    IF (($SnapIn))
    {
        $Global:SnapinLoaded = 'False'
    }
}

Function LoadData #Populates arrays of the 3 slowest items to query. By filling up an array once, we can then recurse through it without having to re-query it repeatedly. 
{
$Global:VBREPJob = Get-VBREPJob -ErrorVariable SnapIn -ErrorAction SilentlyContinue
    IF (($SnapIn))
    {
        $Global:SnapinLoaded = 'False'
    }
$Global:VBRJob = Get-VBRJob 
$Global:VBRBackupSession = Get-VBRBackupSession
}
####################### /Loading #######################





####################### Veeam ####################### DONE
Function CountVeeamEnabled
{
    $Enabled = $Global:VBRJob | where {$_.JobType -eq 'Backup' -and $_.IsScheduleEnabled -eq $True} 
    $EnabledCount = ($Enabled | Measure-Object).count
    IF ($EnabledCount -gt 0)
    {
        $Global:CountVeeamEnabled += $EnabledCount
    }
}

Function CountVeeamDisabled
{
    $Disabled = $Global:VBRJob | where {$_.JobType -eq 'Backup' -and $_.IsScheduleEnabled -ne $True} 
    $DisabledCount = ($Disabled | Measure-Object).count
    IF ($DisabledCount -gt 0)
    {
        $Global:CountVeeamDisabled += $DisabledCount
        $Output = ''
        FOREACH ($Job in $Disabled)
        {
            $Output += $Job.Name + " | "
        }
        $Global:DisabledJobs += $Output
    }
}

Function CountVeeamStates
{
    $Jobs = $Global:VBRJob | where {($_.JobType -eq 'Backup') -and ($_.IsScheduleEnabled -eq 'True')} 
    [int]$Count = 0
    foreach ($Job in $Jobs)
    {
        $StatusSuccess = ($Global:VBRBackupSession | Where {$_.jobId -eq $Job.Id.Guid} | Sort EndTimeUTC -Descending | Select -First 1 | Where {$_.Result -eq 'Success'}| Select JobName | ft -hidetableheaders)
        if ($StatusSuccess)
        {
            $CountSuccess++
        }
        $StatusFailed = ($Global:VBRBackupSession | Where {$_.jobId -eq $Job.Id.Guid} | Sort EndTimeUTC -Descending | Select -First 1 | Where {$_.Result -eq 'Failed'}| Select JobName | ft -hidetableheaders)
        if ($StatusFailed)
        {
        $CountFailed++
        }
        $SessionsGood = ($Global:VBRBackupSession | Where {$_.jobId -eq $Job.Id.Guid} | Sort EndTimeUTC -Descending | Where {($_.EndTime -ge (Get-Date).addhours(-$Global:AuditHours)) -and ($_.Result -eq 'Success')}| Select JobName)
        FOREACH ($Run in $SessionsGood)
        {
            $CountGood++
        }
        $SessionsBad = ($Global:VBRBackupSession | Where {$_.jobId -eq $Job.Id.Guid} | Sort EndTimeUTC -Descending | Where {($_.EndTime -ge (Get-Date).addhours(-$Global:AuditHours)) -and ($_.Result -eq 'Failed')}| Select JobName)
        FOREACH ($Run in $SessionsBad)
        {
            $CountBad++
        }
        
    }
    $Global:CountVeeamStateSuccess += $CountSuccess
    $Global:CountVeeamStateFailed += $CountFailed
    $Global:CountVeeamTotalSuccess += $CountGood
    $Global:CountVeeamTotalFailed += $CountBad
}

####################### /Veeam ####################### 





####################### Good Agent ####################### DONE
Function CountGoodAgentEnabled
{
    $AgentJobs = 0
    $AgentJobs = $Global:VBRJob | Where {($_.JobType -eq 'EpAgentBackup') -and ($_.IsScheduleEnabled -eq 'True')} 
    $Global:CountGoodAgentEnabled += ($AgentJobs | Measure-Object).count
}

Function CountGoodAgentDisabled
{
    $Disabled = $Global:VBRJob | Where {($_.JobType -eq 'EpAgentBackup') -and ($_.IsScheduleEnabled -ne 'True')} 
    $DisabledCount = ($Disabled | Measure-Object).count
    IF ($DisabledCount -gt 0)
    {
        $Global:CountGoodAgentDisabled += $DisabledCount
        $Output = ''
        FOREACH ($Job in $Disabled)
        {
            $Output += $Job.Name + " | "
        }
        $Global:DisabledJobs += $Output
    }   
}

Function CountGoodAgentStates #Counts all enabled jobs with a last state of Success
{
    $AgentJobs = 0
    $AgentJobs = $Global:VBRJob | Where {($_.JobType -eq 'EpAgentBackup') -and ($_.IsScheduleEnabled -eq 'True')} 
    FOREACH ($Job in $AgentJobs)
    {
        $Sessions = [veeam.backup.core.cbackupsession]::GetByJob($Job.Id)
        $Global:CountGoodAgentStateSuccess += ($Sessions | Sort EndTimeUTC -Descending | Select -First 1 | ? {$_.Result -eq 'Success'} | Select Name| Measure-Object).count
        $Global:CountGoodAgentStateFailed += ($Sessions | Sort EndTimeUTC -Descending | Select -First 1 | ? {$_.Result -eq 'Failed'} | Select Name| Measure-Object).count
        $Global:CountGoodAgentTotalSuccess +=($Sessions | Sort EndTimeUTC -Descending | ? {($_.EndTime -ge (Get-Date).addhours(-$Global:AuditHours)) -and ($_.Result -eq 'Success')} | Select Name| Measure-Object).count
        $Global:CountGoodAgentTotalFailed +=($Sessions | Sort EndTimeUTC -Descending | ? {($_.EndTime -ge (Get-Date).addhours(-$Global:AuditHours)) -and ($_.Result -eq 'Failed')} | Select Name| Measure-Object).count
    }
}
####################### /Good Agent ####################### 





####################### Bad Agent ####################### NOTE: Bad Agents cannot record job history, and some are on wide timeframes. Thus not reporting 24 hour totals. 
Function CountBadAgentEnabled
{
    $Global:CountBadAgentEnabled += ($Global:VBREPJob | Where {$_.IsEnabled -eq 'True'}).count
}

Function CountBadAgentDisabled 
{
    $Global:CountBadAgentDisabled += ($Global:VBREPJob | Where {$_.IsEnabled -ne 'True'}).count
}

Function CountBadAgentSuccess
{
    $Global:CountBadAgentSuccess += ($Global:VBREPJob | Where {($_.LastResult -ne 'Failed') -and($_.IsEnabled -eq 'True')}).count
}

Function CountBadAgentFailed
{
    $Global:CountBadAgentFailed += ($Global:VBREPJob | Where {($_.LastResult -eq 'Failed') -and($_.IsEnabled -eq 'True')}).count
}
####################### /Bad Agent ####################### 





####################### Offsite ####################### 
Function CountOffsiteEnabled
{
    $Enabled = $Global:VBRJob | where {$_.JobType -eq 'BackupSync' -and $_.IsScheduleEnabled -eq $True} 
    $EnabledCount = ($Enabled | Measure-Object).count
    IF ($EnabledCount -gt 0)
    {
        $Global:CountOffsiteEnabled += $EnabledCount
    }
}

Function CountOffsiteDisabled
{
    $Disabled = $Global:VBRJob | where {$_.JobType -eq 'BackupSync' -and $_.IsScheduleEnabled -ne $True} 
    $DisabledCount = ($Disabled | Measure-Object).count
    IF ($DisabledCount -gt 0)
    {
        $Global:CountOffsiteDisabled += $DisabledCount
        $Output = ''
        FOREACH ($Job in $Disabled)
        {
            $Output += $Job.Name + " | "
        }
        $Global:DisabledJobs += $Output
    }
}

Function CountOffsites
{
    $Jobs = $Global:VBRJob | where {($_.JobType -eq 'BackupSync') -and ($_.IsScheduleEnabled -eq 'True')} 
    [int]$Count = 0
    $Status = ''
    foreach ($Job in $Jobs)
    {
        $StatusSuccess = ($Global:VBRBackupSession | Where {$_.jobId -eq $Job.Id.Guid} | Sort EndTimeUTC -Descending | Select -First 1 | Where {$_.Result -ne 'Failed'}| Select JobName | ft -hidetableheaders)
        if ($StatusSucess)
        {
            $CountSuccess++
        }
        $StatusFailed = ($Global:VBRBackupSession | Where {$_.jobId -eq $Job.Id.Guid} | Sort EndTimeUTC -Descending | Select -First 1 | Where {$_.Result -eq 'Failed'}| Select JobName | ft -hidetableheaders)
        if ($StatusFailed)
        {
            $CountFailed++
        }
        $TotalSuccess = ($Global:VBRBackupSession | Where {$_.jobId -eq $Job.Id.Guid} | Sort EndTimeUTC -Descending | Where {($_.EndTime -ge (Get-Date).addhours(-$Global:AuditHours)) -and ($_.Result -ne 'Failed')}| Select JobName)
        FOREACH ($Run in $TotalSuccess)
        {
            $CountTotalSuccess++
        }
        $TotalFailed = ($Global:VBRBackupSession | Where {$_.jobId -eq $Job.Id.Guid} | Sort EndTimeUTC -Descending | Where {($_.EndTime -ge (Get-Date).addhours(-$Global:AuditHours)) -and ($_.Result -eq 'Failed')}| Select JobName)
        FOREACH ($Run in $TotalFailed)
        {
            $CountTotalFailed++
        }
    }
    $Global:CountOffsiteSuccess += $CountSuccess
    $Global:CountOffsiteFailed += $CountFailed
    $Global:CountOffsiteTotalSuccess += $CountTotalSuccess
    $Global:CountOffsiteTotalFailed += $CountTotalFailed
}
   
####################### /Offsite ####################### 





####################### Data ####################### 
Function GatherTotals
{
$Global:TotalEnabled = ($Global:CountVeeamEnabled + $Global:CountGoodAgentEnabled + $Global:CountBadAgentEnabled + $Global:CountOffsiteEnabled)
$Global:TotalDisabled = ($Global:CountVeeamDisabled + $Global:CountGoodAgentDisabled + $Global:CountBadAgentDisabled + $Global:CountOffsiteDisabled)
$Global:TotalStateSuccess = ($Global:CountVeeamStateSuccess + $Global:CountGoodAgentStateSuccess + $Global:CountBadAgentSuccess + $Global:CountOffsiteSuccess) 
$Global:TotalStateFailed = ($Global:CountVeeamStateFailed + $Global:CountGoodAgentStateFailed + $Global:CountBadAgentFailed + $Global:CountOffsiteFailed) 
$Global:Total24HourSuccess = ($Global:CountVeeamTotalSuccess + $Global:CountGoodAgentTotalSuccess + $Global:CountOffsiteTotalSuccess) 
$Global:Total24HourFailed = ($Global:CountVeeamTotalFailed + $Global:CountGoodAgentTotalFailed + $Global:CountOffsiteTotalFailed ) 
}

Function GenerateOutput #Gathers all Variables into single output string. 
{
    IF ($Global:BackupServicePresent -eq 'False')
    {
        $Global:Output = "WARNING: 'Veeam Backup Service' Missing"
    }
    ELSEIF ($Global:SnapinLoaded -eq 'False')
    {
        $Global:Output = "WARNING: PowerShell SnapIn Failed To Load"
    }
    ELSEIF (($Global:TotalStateSuccess + $Global:Total24HourSuccess + $Global:TotalStateFailed + $Global:Total24HourFailed + $Global:CountBadAgentEnabled + $Global:CountBadAgentDisabled + $Global:TotalDisabled) -eq 0)
    {
        $Global:Output = "WARNING: No Jobs Configured"
    }
    ELSEIF ((($Global:TotalStateSuccess + $Global:Total24HourSuccess + $Global:TotalStateFailed + $Global:Total24HourFailed) -eq 0) -and ($Global:TotalDisabled -ne 0))
    {
        $Global:Output = "WARNING: All Jobs Disabled"
    }
    ELSEIF (($Global:Total24HourSuccess -eq 0) -and ($Global:CountBadAgentEnabled -ne 0))
    {
        $Global:Output = "WARNING: No Success Last " + $Global:AuditHours + " Hours. Bad:" + ($Global:CountBadAgentEnabled + $Global:CountBadAgentDisabled)
    }
    ELSEIF (($Global:Total24HourSuccess -eq 0) -and ($Global:CountBadAgentEnabled -eq 0))
    {
        $Global:Output = "WARNING: No Success Last " + $Global:AuditHours + " Hours"
    }
    ELSE
    {
        IF ($Global:TotalStateFailed -ne 0)
        {
            $Global:Output += "FAIL:" + "{0:D2}" -f $Global:TotalStateFailed + " "
        }

        IF ($Global:Total24HourFailed -ne 0)
        {
            $Global:Output += "F" + $Global:AuditHours + "hr:" + "{0:D2}" -f $Global:Total24HourFailed + " "
        }
         
        IF ($Global:CountErrors -ne 0)
        {
            $Global:Output += ("Err:" + "{0:D2}" -f $Global:CountErrors + " ")
        }

        IF (($Global:CountBadAgentEnabled + $Global:CountBadAgentDisabled) -ne 0)
        {
            $Global:Output += ("Bad:" + ($Global:CountBadAgentEnabled + $Global:CountBadAgentDisabled) + " ")
        }
    
        IF ($Global:TotalDisabled -ne 0)
        {
            $Global:Output += ("Dis:" + $Global:TotalDisabled + " ")
        }



        IF (($Global:TotalStateFailed + $Global:Total24HourFailed + $Global:CountBadAgentEnabled + $Global:CountBadAgentDisabled + $Global:TotalDisabled + $Global:CountErrors) -eq 0)
        {
            $Global:Output = ''
        }
        
    }
    
    

    
}
####################### /Data ####################### 

Function DEV #Testing function that outputs all variables, and times each functions completion time for troubleshooting. 
{
    #Function Calls
    Write-Host "============================================================"
    Write-Host "BEGIN QUERY FOR" $Global:AuditHours "HOUR WINDOW. RESULTS REFLECT QUERY TIME IN SECONDS"
    Write-Host " "
    Write-Host "Counting Errors:" (Measure-Command {CountErrors}).Seconds
    Write-Host "Checking Critical Services:" (Measure-Command {ServiceCheck}).Seconds
    Write-Host "Loading PowerShell SnapIn" (Measure-Command {LoadSnapIn}).Seconds
    Write-Host "Loading Job Data" (Measure-Command {LoadData}).Seconds
    Write-Host " "
    Write-Host " ##### COUNTERS - VEEAM ##### "
    Write-Host "Enabled Jobs:" (Measure-Command {CountVeeamEnabled}).Seconds
    Write-Host "Disabled Jobs:" (Measure-Command {CountVeeamDisabled}).Seconds
    IF ($Global:CountVeeamEnabled -ne 0)
    {
        Write-Host "Gathering All States" (Measure-Command {CountVeeamStates}).Seconds
    }

    Write-Host " "
    Write-Host " ##### COUNTERS - GOOD AGENT ##### "
    Write-Host "Enabled Jobs:" (Measure-Command {CountGoodAgentEnabled}).Seconds
    Write-Host "Disabled Jobs:" (Measure-Command {CountGoodAgentDisabled}).Seconds
    IF ($Global:CountGoodAgentEnabled -ne 0)
    {
        Write-Host "Gathering All States:" (Measure-Command {CountGoodAgentStates}).Seconds
    }
    Write-Host " "
    Write-Host " ##### COUNTERS - BAD AGENT ##### "
    Write-Host "Enabled Jobs:" (Measure-Command {CountBadAgentEnabled}).Seconds
    Write-Host "Disabled Jobs:" (Measure-Command {CountBadAgentDisabled}).Seconds
    Write-Host "Last State Success:" (Measure-Command {CountBadAgentSuccess}).Seconds #Not a recursive query, no point in being conditional
    Write-Host "Last State Failed:" (Measure-Command {CountBadAgentFailed}).Seconds
    Write-Host " "
    Write-Host " ##### COUNTERS - OFFSITE ##### "
    Write-Host "Enabled Jobs:" (Measure-Command {CountOffsiteEnabled}).Seconds
    Write-Host "Disabled Jobs:" (Measure-Command {CountOffsiteDisabled}).Seconds
    IF ($Global:CountOffsiteEnabled -ne 0)
    {
        Write-Host "Gathering All States:" (Measure-Command {CountOffsites}).Seconds
    }

    Write-Host " "
    Write-Host "Gathering Totals:" (Measure-Command {GatherTotals}).Seconds
    Write-Host "Generating Output:" (Measure-Command {GenerateOutput}).Seconds
    Write-Host " "
    Write-Host "END QUERY MEASUREMENTS"
    Write-Host "============================================================"
    Write-Host " "
    Write-Host " "
    Write-Host "============================================================"
    Write-Host "BEGIN COUNTER OUTPUT. RESULTS REFLECT DATA GATHERED"
    Write-Host " "
    Write-Host " ##### COUNTERS - VEEAM ##### "
    Write-Host "Enabled Jobs:" $Global:CountVeeamEnabled
    Write-Host "Disabled Jobs:" $Global:CountVeeamDisabled
    Write-Host "Last State Success:" $Global:CountVeeamStateSuccess
    Write-Host "Last State Failed:" $Global:CountVeeamStateFailed 
    Write-Host $Global:AuditHours "Hour Success:" $Global:CountVeeamTotalSuccess
    Write-Host $Global:AuditHours "Hour Failures:" $Global:CountVeeamTotalFailed
    Write-Host " "
    Write-Host " ##### COUNTERS - GOOD AGENT ##### "
    Write-Host "Enabled Jobs:" $Global:CountGoodAgentEnabled
    Write-Host "Disabled Jobs:" $Global:CountGoodAgentDisabled
    Write-Host "Last State Success:" $Global:CountGoodAgentStateSuccess
    Write-Host "Last State Failed:" $Global:CountGoodAgentStateFailed
    Write-Host $Global:AuditHours "Hour Success:" $Global:CountGoodAgentTotalSuccess
    Write-Host $Global:AuditHours "Hour Failures:" $Global:CountGoodAgentTotalFailed
    Write-Host " "
    Write-Host " ##### COUNTERS - BAD AGENT ##### "
    Write-Host "Enabled Jobs:" $Global:CountBadAgentEnabled
    Write-Host "Disabled Jobs:" $Global:CountBadAgentDisabled
    Write-Host "Last State Success:" $Global:CountBadAgentSuccess
    Write-Host "Last State Failed:" $Global:CountBadAgentFailed
    Write-Host " "
    Write-Host " ##### COUNTERS - OFFSITE ##### "
    Write-Host "Enabled Jobs:" $Global:CountOffsiteEnabled 
    Write-Host "Disabled Jobs:" $Global:CountOffsiteDisabled
    Write-Host "Last State Success:" $Global:CountOffsiteSuccess
    Write-Host "Last State Failed:" $Global:CountOffsiteFailed
    Write-Host $Global:AuditHours "Hour Success:" $Global:CountOffsiteTotalSuccess
    Write-Host $Global:AuditHours "Hour Failures:" $Global:CountOffsiteTotalFailed
    Write-Host " "
    Write-Host " ##### COUNTERS - TOTALS ##### "
    Write-Host "Enabled Jobs:" $Global:TotalEnabled
    Write-Host "Disabled Jobs:" $Global:TotalDisabled
    Write-Host "Last State Success:" $Global:TotalStateSuccess
    Write-Host "Last State Failed:" $Global:TotalStateFailed
    Write-Host $Global:AuditHours "Hour Success:" $Global:Total24HourSuccess 
    Write-Host $Global:AuditHours "Hour Failures:" $Global:Total24HourFailed
    Write-Host "Errors" $Global:CountErrors
    Write-Host " " 
    Write-Host "Disabled Jobs:" $Global:DisabledJobs
    Write-Host "============================================================"
    $Global:Output
    Write-Output $Global:Output >> $Global:LogFile
    Exit
}


cls
#DEV  #Un Comment Dev call to manually run script with visible output and measured times per function. 


#Function Calls


ServiceCheck
LoadSnapIn
CountErrors
LoadData
CountVeeamEnabled
CountVeeamDisabled
IF ($Global:CountVeeamEnabled -ne 0)
{
    CountVeeamStates
}
CountGoodAgentEnabled
CountGoodAgentDisabled
IF ($Global:CountGoodAgentEnabled -ne 0)
{
    CountGoodAgentStates
}
CountBadAgentEnabled
CountBadAgentDisabled
CountBadAgentSuccess
CountBadAgentFailed
CountOffsiteEnabled
CountOffsiteDisabled
IF ($Global:CountOffsiteEnabled -ne 0)
{
    CountOffsites
}
GatherTotals
GenerateOutput


$Global:Output
Write-Output $Global:Output >> $Global:LogFile
 

[Mike Resseler]

Hey Quinn,

This looks awesome! I would like to ask you if you want to share this with the broader community. Maybe through GitHub and then linked to Veeam's account?

Great work, and thanks for this!

Mike

[quinnvanorder]

I would be happy to! I am about 75% of the way through v2 of this, which is significantly more powerful and less sloppy! That V1 was a proof of concept more than anything.

I will post it in this thread, and once it's live I'll ping you about how to link it to Veeam's GitHub account.