Comprehensive data protection for all workloads
Post Reply
RichardR
Novice
Posts: 8
Liked: 1 time
Joined: May 13, 2010 7:01 am
Full Name: Richard Ramshaw
Contact:

Job Chaining / Maximum concurrent jobs - Fix!

Post by RichardR »

Hi All

There have been many people wanting a job chaning solution for Veeam Backup or a way of running a maximum number of concurrent jobs. Many reasons for wanting to do this (ours was that we have a WAN link and 2 jobs saturates it, 3 make it run badly)

For all of those out there that wanted it here is a powershell script that does exactly that.

It accepts 2 command line parameters, 1 for a batch of jobs to run, this supports wild cards so you can run the script with *daily* to run all backups with Daily in their name. The second parameter is the number of concurrent jobs that you want to have running e.g. 2. The script will find all the jobs that match, list them and then wait untill less than the maximum backups are running, it will then kick off the next backup and wait until another 'slot' becomes avaliable before starting another backup.

As the script checks with Veeam as to which backups are running you can run the script more than once at the same time and it will still keep the total number of backups running accross all scripts to less than the maximum.

To run the script from the command line use something like this:

powershell -command ".\Runjobs.ps1 *daily* 4"
If you need to put spaces in it then you need to get funky with the powershell command line like this:
powershell -command "& 'c:\program files\Veeam\Backup and FastSCP\Runjobs.ps1' '*six hourly' 2"

The contents of the script is below, it will need saving as a powershell script i.e. with a .ps1 extension. If you cannot run scripts in powershell (this is the default) you will need to open powershell and type Set-ExecutionPolicy Unrestricted

This is my first powershell script so any comments/improvements would be welcomed :D .

Cheers

Richard

###############################################
If ($args.Count -ne 2 -or $args[0] -eq "" -or $args[1] -eq 0) {
Write-Host "Incorrect arguments, syntax: runjobs JobContains MaxJobs";
Write-Host "eg: runjobs *Daily* 2";
Exit;
}

add-pssnapin veeampssnapin;

$JobList=@();
$runningjobs=@();
$JobMatch = $args[0];
$Maxjobs = $args[1];

Write-Host "Looking for jobs that match" $JobMatch;

Get-VBRJob | where {$_.Name -like $JobMatch} | foreach {$JobList += $_.name};
Write-Host "Found" $joblist.count "jobs";

If ($joblist -eq $null -or $joblist.count -eq 0) {
Write-Host "No Jobs match that string... exiting";
Exit;
}

$joblist = $joblist | Sort-Object

Write-Host "Jobs to run...";
ForEach ($job in $JobList) {Write-Host " " $job;}

ForEach ($job in $JobList) {
Do {
$runningjobs=@();
Get-VBRJob | where {$_.Control -eq "Start"} | foreach {$RunningJobs += $_.name};
If ($runningjobs.count -ge $Maxjobs) {
Write-Host "Waiting to start '" $job "' as " $runningjobs "are still running";
Start-Sleep -s 60;
}
}
While ($runningjobs.count -ge $Maxjobs)

Write-Host "Starting " $job;
Start-Job -scriptblock {add-pssnapin veeampssnapin; Start-VBRJob $args[0]} -ArgumentList $job;
Write-Host "Waiting for veeam to start the job";
Start-Sleep -s 60;
}
#############################
Alexey D.

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by Alexey D. »

Hi Richard,

Thanks you very much for sharing it with the community!
rto
Lurker
Posts: 1
Liked: never
Joined: Oct 27, 2010 1:51 pm
Full Name: Rasmus Toft
Contact:

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by rto »

Anyone that have got this script running on Veeam version 5?
cmbits
Lurker
Posts: 2
Liked: never
Joined: Mar 01, 2010 2:53 pm
Full Name: Chris Butler
Contact:

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by cmbits »

I used this script under version 4.1 and loved it. After v5 I have been updating the script to work with the new cmdlets. Everything works except for checking on the concurrently running jobs. I was playing with this but I need to figure out all the possible states of a job and haven't had time to do this yet;
$runningjobs = Get-VBRJob | where {$_.FindLastSession().State -eq "Working"}

Does anyone know the best way to check for number of active jobs?

Thanks,
Chris
RichardR
Novice
Posts: 8
Liked: 1 time
Joined: May 13, 2010 7:01 am
Full Name: Richard Ramshaw
Contact:

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by RichardR »

They helpfully changed several of the structures and methods in the PowerShell commandlets in v5.0 without telling anyone and without changing the names of the constructs :shock:

However here is a new version of the script, that works on v5.0 *** IT ONLY WORKS on v5.0 ***

It is very slightly different on the command line as now it also support logging so you will need a command line like this...

Code: Select all

powershell -command "& 'c:\program files\Veeam\Backup and Replication\Replicate.ps1' '*daily*' 2 DailyBackups.Log"
The script is here, copy and paste to notepad, save as replicate.ps1

Code: Select all

$Global:logfile = $args[2];
If ($logfile -ne $null) {Out-File $logfile}


Function WriteLog ([string]$Entry) {
	Write-Host $Entry;
	If ($logfile -ne $null) {Out-File $logfile -append -inputobject $Entry;}
}

If ($args.Count -le 1 -or $args[0] -eq "" -or $args[1] -eq 0) {
	WriteLog "Incorrect arguments, syntax: runjobs JobContains MaxJobs LogFile";
	WriteLog "eg: runjobs *Daily* 2 LogFile.txt";
	Exit;
}

add-pssnapin veeampssnapin;

$JobList=@();
$runningjobs=@();
$JobMatch = $args[0];
$Maxjobs = $args[1];

WriteLog ("Looking for jobs that match " + $JobMatch);

Get-VBRJob | where {$_.Name -like $JobMatch} | foreach {$JobList += $_.name};
WriteLog ("Found " + $joblist.count + " jobs");

If ($joblist -eq $null -or $joblist.count -eq 0) {
	WriteLog "No Jobs match that string... exiting";
	Exit;
}

$joblist = $joblist | Sort-Object

WriteLog "Jobs to run...";
ForEach ($job in $JobList) {WriteLog ("	" + $job);}

ForEach ($job in $JobList) {
	Do {
		$runningjobs=@();
		Get-VBRBackupSession | where {$_.State -eq "Working"} | foreach {$RunningJobs += $_.name};
		If ($runningjobs.count -ge $Maxjobs) {
			WriteLog ("Waiting to start '" + $job + "' as " + $runningjobs + " are still running");
			Start-Sleep -s 60;
		}
	}
	While ($runningjobs.count -ge $Maxjobs)
	
	WriteLog ("Starting " + $job);
	Get-VBRJob | where {$_.Name -eq $job} | foreach {start-vbrjob $_ -RunAsync};
	WriteLog ("Waiting for veeam to start the job");
	Start-Sleep -s 60;
}

Gostev
Chief Product Officer
Posts: 31793
Liked: 7295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by Gostev »

RichardR wrote:They helpfully changed several of the structures and methods in the PowerShell commandlets in v5.0 without telling anyone
Actually, this change is documented in the Release Notes. It would be pretty hard to stick to existing cmdlets structures after adding 4x more functionality to the product than previous version had...

Thank you very much for posting an updated version of your script.
RichardR
Novice
Posts: 8
Liked: 1 time
Joined: May 13, 2010 7:01 am
Full Name: Richard Ramshaw
Contact:

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by RichardR » 1 person likes this post

Hi Gostev, Sorry didn't read the release notes :oops:

But it is normal programming behaviour to never change existing constructs in an externally accessible API, if there is a new way of doing something then it should be given a different name such as Get-VBRJobs_EX for extended. If you change a construct to give completely different information back (as in the case of Get-VBRJobs) then all hell could break loose in other peoples extensions, in this case it just caused more concurrent jobs to run than it should, but what if I had written a cleanup script based on the same function any thing could have hapened!

I understand that you want to add new functions to the API, and thanks for them, but as it is an externally published API then Veeam should follow correct coding standards when changing it.

Richard
Gostev
Chief Product Officer
Posts: 31793
Liked: 7295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by Gostev »

I agree, in ideal world... in reality, because of resource/timelines considerations (just way too many other features to develop and test), our options were either not to provide new capabilities via PowerShell at all, or switch to new cmdlets without keeping compatibility. Testing and maintaining to set of cmdlets was not an option (we would have to re-write and re-test all old cmdlets to work with new API model as well). So, we chose the latter - switch to new cmdlets without keeping compatibility.
Marc
Novice
Posts: 4
Liked: never
Joined: Jul 20, 2009 1:32 am
Full Name: Marc
Contact:

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by Marc »

I have tried implementing the updated script on our v5.0 install, however all the jobs run at once. Testing "Get-VBRBackupSession" gives the following output:

PS C:\> Get-VBRBackupSession
Get-VBRBackupSession : Specified cast is not valid.
At line:1 char:21
+ Get-VBRBackupSession <<<<
+ CategoryInfo : InvalidOperation: (Veeam.Backup.Po...BRBackupSession:GetVBRBackupSession) [Get-VBRBackup
Session], InvalidCastException
+ FullyQualifiedErrorId : Backup,Veeam.Backup.PowerShell.Command.GetVBRBackupSession

What have I done wrong?
cbeilman
Novice
Posts: 4
Liked: 1 time
Joined: Jan 27, 2011 7:14 pm
Full Name: Clint Beilman

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by cbeilman »

I've added this code to retry any jobs that have failed up to $retryJobsCount times per failed job.

Code: Select all

# Retry Failed Jobs
$retryJobsCount = 3

$retryCount = 0
$failedJobs = @{}
[Boolean] $retrydone = $false

Do
{
	Get-VBRJob | Where {$_.Name -like $JobMatch -and $_.Info.LatestStatus -eq "Failed"} | % {
		if ($failedJobs.ContainsKey($_.Name)) {
			if ($failedJobs.get_Item($_.Name) -le $retryJobsCount) 
			{
				Start-VBRJob $_ -RetryBackup -RunAsync
				$failedJobs.set_Item($_.Name,$failedJobs.get_Item($_.Name) + 1)
				Start-Sleep -s 60
			}
		} else {
			$failedJobs.Add($_.Name,1)
			WriteLog ("Retrying " + $_.Name)
			Start-VBRJob $_ -RetryBackup -RunAsync
			Start-Sleep -s 60
		}
		
	}
	$lowestFailedCount = 0
	$failedJobs.Values | % {if ($_ -gt $lowestFailedCount) {$lowestFailedCount = $_}}
	if ($lowestFailedCount -ge $retryJobsCount) {$retrydone = $true}
}
While (Get-VBRBackupSession | Where {$_.State -eq "Working"}) -ne $null -and `
(Get-VBRJob | Where {$_.Name -like $JobMatch -and `
$_.Info.LatestStatus -eq "Failed"}).Count -ne $null -and -not $retryDone
Alexey D.

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by Alexey D. »

Thanks for posting, Clint. Good script for KB.
cbeilman
Novice
Posts: 4
Liked: 1 time
Joined: Jan 27, 2011 7:14 pm
Full Name: Clint Beilman

Re: Job Chaining / Maximum concurrent jobs - Fix!

Post by cbeilman » 1 person likes this post

Whoops, I one mistake in that script. Change the "*Daily" in the While statement at the bottom to $JobMatch.

Code: Select all

    # Retry Failed Jobs
    $retryJobsCount = 3

    $retryCount = 0
    $failedJobs = @{}
    [Boolean] $retrydone = $false

    Do
    {
       Get-VBRJob | Where {$_.Name -like $JobMatch -and $_.Info.LatestStatus -eq "Failed"} | % {
          if ($failedJobs.ContainsKey($_.Name)) {
             if ($failedJobs.get_Item($_.Name) -le $retryJobsCount)
             {
                Start-VBRJob $_ -RetryBackup -RunAsync
                $failedJobs.set_Item($_.Name,$failedJobs.get_Item($_.Name) + 1)
                Start-Sleep -s 60
             }
          } else {
             $failedJobs.Add($_.Name,1)
             WriteLog ("Retrying " + $_.Name)
             Start-VBRJob $_ -RetryBackup -RunAsync
             Start-Sleep -s 60
          }
          
       }
       $lowestFailedCount = 0
       $failedJobs.Values | % {if ($_ -gt $lowestFailedCount) {$lowestFailedCount = $_}}
       if ($lowestFailedCount -ge $retryJobsCount) {$retrydone = $true}
    }
    While (Get-VBRBackupSession | Where {$_.State -eq "Working"}) -ne $null -and `
    (Get-VBRJob | Where {$_.Name -like $JobMatch -and `
    $_.Info.LatestStatus -eq "Failed"}).Count -ne $null -and -not $retryDone
jmc
Service Provider
Posts: 103
Liked: 9 times
Joined: Sep 12, 2011 11:49 am
Full Name: jmc
Location: Duisburg - Germany
Contact:

Backup 50 VM's at the same time! - Max Tasks

Post by jmc »

[merged]

hello all,

i have the problem, that i have up to 50 vm's to backup separatly. i have no idea how long each vm need and sometimes it differ so much. noe i have set all backup slots to the same start time and i hope that veeam has a max task parameter and not all 50 backups will starts together.

i'm right about that or doesn't veeam has this limiter and i got a lot of trouble this night.

ca i set this limit somewhere?

thx and greetings from germany

ps:
it is not possible to group this vm's together in one job ( business reasons :-) )
Post Reply

Who is online

Users browsing this forum: Bing [Bot], robnicholsonmalt and 70 guests