Active/Running reports

[tkeith]

This may be a feature request but having a large environment i have to ask if there is a report or way to tell whats actively running without having to inspect each job? For example if i want to see a list of all proxies that are currently active and running across our jobs, if i need to do troubleshooting or maintenance.

Its challenging to go through 30 jobs trying to determine which proxies are currently in use...

[Shestakov]

Hello Keith,
You can leverage Veeam One capabilities for that. In Veeam One Monitor you can observe recent proxies activities as well as statistical information and more.
Thanks!

[AMS]

@Shestakov, while the report shows a history and "running tasks", it does not show real time proxy usage. Also, it requires you to click on each proxy individually to get the information. Lastly, there doesn't seem to be a way to see what proxy server is serving what job, what VMDK is being backed up, or which repository is currently being written to.

[Shestakov]

Hello Ari!
You are correct, looks like the report about backup proxies will be helpful.
Do you want any other info about proxies to be shown?
Thank you for the feedback!

[Vitaliy S.]

Hi Keith,

In a perfect world all proxy servers should be used, otherwise you're over-provisioning your Veeam backup infrastructure. Can you please clarify a bit your request? Do you run jobs on different schedule throughout the day?

@Ari, while I agree that this can be useful data, I do also believe that a report would not address this use case, cause by the time you build it, the entire information can change (for example, backup job has finished or backup proxy server became vacant for processing other jobs).

Thanks!

[AMS]

@ Vitaliy I have been discussing with with Keith working on other things so I am familiar with what he is looking for. We were less worried about a report. We were looking to get a view of some kind, to see which Proxy servers are active right now. During the backup, there are a large amount of jobs running on a large amount of Proxy servers. In fact, Veeam is balancing the load across the Proxies rather well. So imagine we have 30 Proxy servers, each capable of running 8 VMDK streams at a time. During the backup window 240 streams are running. So, we need to see in real time which Proxies are active. As the various jobs complete and streams finish, we need to see which proxies are still actively moving data. Also, as the day goes on and perhaps ad-hoc jobs, or development VM jobs are running, we need to see which proxies are active. It would be very helpful to be able to run a PowerShell script even that would show each proxy server, which VMDK it is backing up and which repository it is writing to.

We don't expect this to be available in VeeamOne as a report but rather as a powershell or in the VeeamOne Client perhaps as a monitor.

A VeeamOne report that would be useful is which proxy servers served which VMDK's and which jobs on a particular date to make troubleshooting easier.

[Vitaliy S.]

Ok, I think it should be possible to add some indication of all running proxy servers.

[tsightler]

So, we need to see in real time which Proxies are active. As the various jobs complete and streams finish, we need to see which proxies are still actively moving data. Also, as the day goes on and perhaps ad-hoc jobs, or development VM jobs are running, we need to see which proxies are active. It would be very helpful to be able to run a PowerShell script even that would show each proxy server, which VMDK it is backing up and which repository it is writing to.

In the interim I created a script to do this, posted below for reference. It doesn't include repository, but I could certainly add that if it would be useful:

Code: Select all

# This script attempts to list all current active proxy tasks.
# Unfortunately the Veeam Powershell does not provide complete
# information via the task object model as those objects track only
# a single task for each VM even if the VM has multiple disk
# using different proxies.  Due to this limitation this script
# takes a different approach directly parsing the log entries
# of all actively running jobs and matching currently processing
# disks to proxies.  While this sounds crude, it appears to work
# well.
#
# Caveats:
#  - Doesn't currently take into account any use of proxies by
#    replication jobs (wouldn't be too hard to add)

asnp "VeeamPSSnapIn" -ErrorAction SilentlyContinue

# Get all proxies and create regexes for matching log entries
$proxies = Get-VBRViProxy
$proxies_regex = (‘(?i)(‘ + (($proxies.Name | ForEach {[regex]::escape($_)}) –join “|”) + ‘)’) -replace "\\\*", ".*"
$harddisk_regex = "(?i)(Hard disk\s+\d+)"

# Create empty array to hold results
$proxy_tasks = @()

# Get currently active backup sessions
$active_sessions = Get-VBRBackupSession | ?{$_.JobType -eq "Backup" -and $_.State -eq "Working"} | Sort JobName, Name
if (!$active_sessions) { Write-Host -ForegroundColor Red "No active backup sessions found!";Break }

foreach ($session in $active_sessions) {
    $tasks = $null = $session.GetTaskSessionsbyStatus("InProgress")  # Get all active tasks
    foreach ($task in $tasks) {
        $logs = $task.Logger.GetLog().UpdatedRecords # Get all logs for current task
        $proxy_logs = $null = $logs.Title | Select-String -Pattern $proxies_regex # Select all log lines that mention a proxy server
        $active_task_logs = $null = $logs | ?{$_.Status -eq "ENone"} # Select log entries that are in progress (exist but no completion status)
        foreach ($log_entry in $active_task_logs) {
            # If active log task includes "Hard disk XX" then its using a proxy, find the matching proxy log entry and grab the proxy name
            if ($log_entry.Title -match $harddisk_regex) {
                $harddisk = $matches[1]
                if (($proxy_logs | Select-String -Pattern $harddisk) -match $proxies_regex) {
                    # Insert collected info into an array
                    $proxy_tasks += New-Object -TypeName PSObject -Property (@{Proxy=$matches[1];Job=$session.JobName;VMname=$task.Name;Disk=$log_entry.Title;Progress=$log_entry.Description})
                }
            }
        }
    }
}
# Sort and convert array into a hash table
$proxy_tasks = $proxy_tasks | Sort-Object -Property Proxy,Job,VMname,Disk | Group-Object -Property Proxy -AsHashTable

# Output all of the collected information to the screen
foreach ($proxy in $proxies) {
    write-host -NoNewline -ForegroundColor green $proxy.Name "- Running:" $proxy_tasks.($proxy.Name).Count "of" $proxy.Options.MaxTasksCount "Tasks"
    if ($proxy_tasks.($proxy.Name).Count -eq 0) {write-host;write-host}
    $proxy_tasks.($proxy.Name) | Format-Table Job, VMname, Disk, Progress
}

Produces output that looks something like this:

Code: Select all

repo01.notyourdomain.com - Running: 4 of 4 Tasks
Job            VMname     Disk                               Progress                      
---            ------     ----                               --------                      
Proxy Test     orcl01     Hard disk 1 (40.0 GB)              6.1 GB read at 20 MB/s [CBT]  
Proxy Test     orcl01     Hard disk 2 (50.0 GB)              2.0 GB read at 12 MB/s [CBT]  
Proxy Test     vc02       Hard disk 1 (25.0 GB)              20.5 GB read at 66 MB/s [th...
Proxy Test     vc02       Hard disk 2 (100.0 GB)             27.5 GB read at 162 MB/s [CBT]


proxy01.notyourdomain.com - Running: 3 of 4 Tasks
Job            VMname     Disk                               Progress                      
---            ------     ----                               --------                      
Proxy Test     lnx02      Hard disk 2 (50.0 GB)              7.2 GB read at 28 MB/s [thr...
Proxy Test     lnx02      Hard disk 3 (40.0 GB)              13.6 GB read at 78 MB/s [CBT] 
Test Servers   walrus01   Preparing backup proxy prox...                              

[dellock6]

Lovely!!!!
Tom's the man, as usual

[lussierd16]

Tom this is great! Can you modify it to include the repo and copy jobs? This is very helpful.

Thanks!

[AMS]

Great script. But if this information is so readily available, why is this not a graphical view part of BnR or VeeamOne?

[veremin]

As far as I'm concerned, there have been plans to make this information available in Veeam ONE Monitor. At least similar requests have been already collected. Thanks.

[Vitaliy S.]

And answering your question - because there were not many requests so far, thus the priority of this feature was lower compared to others.

[xefil]

Great script.
It's possible to get the informations from repos as well? If the backup ends directly on a linux repo, seems is not counted.
Thanks, Simon

[tsightler]

This script reports on proxy use during backup jobs. Linux servers can't be proxies, so indeed they would not be counted, but the proxy that is sending data should be shown in the list. If it's not, it could be a bug and I'd be happy to look at it.

[xefil]

Ok, but linux repos have limitations on sessions as well. Would be interesting to see those informations.
For the proxies, well, here I've only a replication in execution and that is not matching any proxy, even if in use and even if I've corrected:

From

Code: Select all

$active_sessions = Get-VBRBackupSession | ?{$_.JobType -eq "Backup" -and $_.State -eq "Working"} | Sort JobName, Name

to

Code: Select all

$active_sessions = Get-VBRBackupSession | ?{$_.State -eq "Working"} | Sort JobName, Name

Thanks!

Simon

[tsightler]

Ok, but linux repos have limitations on sessions as well. Would be interesting to see those informations.

Sure, my simple point is that this script doesn't report on repository session use at all because it wasn't written to do so. That wouldn't be unique to Linux repositories, that would apply to all repositories. If you want the script to report on repository session usage, how would you want that presented? Report proxy usage first and then repository usage? I'm certainly open to adding that and it would actually be much easier because repository session usage is readily available.

For the proxies, well, here I've only a replication in execution and that is not matching any proxy, even if in use and even if I've corrected:

From

Code: Select all

$active_sessions = Get-VBRBackupSession | ?{$_.JobType -eq "Backup" -and $_.State -eq "Working"} | Sort JobName, Name

to

Code: Select all

$active_sessions = Get-VBRBackupSession | ?{$_.State -eq "Working"} | Sort JobName, Name

As the comments in the script specifically notes, replication jobs are not accounted for by the script, so their usage will not show. Simply changing the line above to include all sessions is not enough as replication jobs use different log messages and thus will require different log parsing logic. It wouldn't be especially difficult to add this, but it wasn't in scope when I wrote this script as the customer this script was originally created for did not use replication jobs. If there is enough interest in reporting on replication proxy usage, I'd be willing to consider adding it as well.

[xefil]

Hello,

Thanks for your interesting.
Well, let me explain what I would like to archive.
I would like to trace the usage of the infrastructure mainly in terms of tasks usage, to understand if there are bottlenecks starting from the servers.
I would then graph those values. I know there are products like Veeam ONE, but what I would like to get are less informations I could get with PowerShell scripts without needs to dedicate a VM for VeeamONE.
So I was thinking it would be great to have counted tasks in use for proxies as well for repositories and replications.
I've tried to do a little by myself but seems a little complicated
I know that it's maybe to much to ask what above, btw maybe if that interests someone other, as you've written, that could be a great improovment to the script

Simon

[tsightler]

Thanks for your interesting.
Well, let me explain what I would like to archive.
I would like to trace the usage of the infrastructure mainly in terms of tasks usage, to understand if there are bottlenecks starting from the servers.
I would then graph those values. I know there are products like Veeam ONE, but what I would like to get are less informations I could get with PowerShell scripts without needs to dedicate a VM for VeeamONE.
So I was thinking it would be great to have counted tasks in use for proxies as well for repositories and replications.
I've tried to do a little by myself but seems a little complicated
I know that it's maybe to much to ask what above, btw maybe if that interests someone other, as you've written, that could be a great improovment to the script

If you're just looking for tasks counts for a graph, that's far easier to obtain than what this script is doing. This script is providing more detail, matching individual jobs, and the disk/tasks within a job, to specific proxies. If you just want task counts for graphing proxy/repo I can provide you some examples of various methods to do this. I had actually started an internal project last year that pulled Veeam proxy/repo slot usage into InfluxDB and used Grafana for dahsboard. I had it working for basic proxy/repo usage, but unfortunately other tasks took my time.

[xefil]

If you're just looking for tasks counts for a graph, that's far easier to obtain than what this script is doing. This script is providing more detail, matching individual jobs, and the disk/tasks within a job, to specific proxies. If you just want task counts for graphing proxy/repo I can provide you some examples of various methods to do this.

This would be fantastic!

I had actually started an internal project last year that pulled Veeam proxy/repo slot usage into InfluxDB and used Grafana for dahsboard. I had it working for basic proxy/repo usage, but unfortunately other tasks took my time.

Oh that should be exactly what I need. Well, I'm still using cacti for graphing, but this isn't the problem.
Having an example for counting using slots on proxies/reps is enough to end the task by myself (I hope) to grapf it then.

Thanks, Simon

[tsightler]

Oh that should be exactly what I need. Well, I'm still using cacti for graphing, but this isn't the problem.
Having an example for counting using slots on proxies/reps is enough to end the task by myself (I hope) to grapf it then.

Hey, nothing wrong with Cacti, I used to have a pretty decent deployment of that myself.

Probably the easiest/fastest ways to get at the data is to just query it from WMI. Even in Powershell it's pretty fast because you don't need any external modules.

Here's an example query (you'd think you could select select the the specific field in the WMI query itself, but that doesn't work with Veeam WMI Classes):

Code: Select all

Get-CimInstance -Query "SELECT * FROM Proxy" -Namespace "ROOT\VeeamBS" | Select Name,ConcurrentJobsNow, ConcurrentJobsMax

This will give you a list of all proxies, their current tasks count, and their maximum tasks count.
It's similar for repositories:

Code: Select all

Get-CimInstance -Query "SELECT * FROM Repository" -Namespace "ROOT\VeeamBS" | Select Name,ConcurrentJobsNow, ConcurrentJobsMax

The later code would need some more smarts if you want to parse out things like cloud repos, SOBR extents, etc., but all that is available via WMI as well.

With my little side project I just used WMIC to query the data via WMI straight from the Linux host that was running InfluxDB/Grafana, that way it was 100% agentless. It's been quite a while, but I feel like I remember there being a WMI module for Cacti as well, so that might be an option. Or you can just write some Powershell using the examples above and go from there. Let me know if you need more examples or have issues.

[xefil]

Hello,

Thanks a lot!!!
It's much easier in that way
Does it work on recovery/restore and replication?

Thanks!

Simon

[tsightler]

Yes, any task/slot usage should be accounted for so VM restores and replication will be counted.

[Henrik.Grevelund]

Hi There,

The VMI calls are pretty neat to show the number of streams running.

But what if i just wanted the number of active jobs running.
I am talking about both VM, agent, backup copy and Translog jobs together as a total.
I would like to gather this number over time a show it in a graph.

anyone found an easy way to get this ?