Compacting and Full Backup list of schedule

PowerShell script exchange

Compacting and Full Backup list of schedule

Veeam Logoby ehrnst » Wed Mar 05, 2014 9:06 am

First post, trying to make it useful :)

Let me start out with some background information. The company i work for have quite a large Veeam environment (in short 4 backup servers, 500 jobs and close to 1000vm's). From time to time we have noticed our servers where under heavy load, one day it escalated and resulted in some failed jobs that we had to restart after the load had dropped to normal. Experiencing this we had to investigate further, and one of my colleagues suspected it could be that a large number of our jobs was set to do a full backup and/or compacting backup files on the same time.

Being human, you intend to repeat what you are doing on a daily basis IE, copying existing jobs when you implement a new :wink:

So i wrote a Powershell script to browse through all our backup jobs (on one server only, modify it to connect to other servers as well) an look for when the jobs is set to compact files, and when they are scheduled to do, in our case, their monthly full backup. Probably not the nicest PS script you have seen, but it does the job.

Code: Select all
#load veeam ps snapin
Add-PSSnapin VeeamPSSnapin
#add get to variable an loop through evry job
$Job = Get-VBRJob
foreach ($Object in $Job)
        {
            $options = $Object.getoptions() #job options variable
            $query = $options.GenerationPolicy.CompactFullBackupMonthlyScheduleOptions #Variable for compact schedule
            $query2 = $options.BackupTargetOptions.FullBackupMonthlyScheduleOptions #Variable for full backup
            $dayofweek1 = $query.DayOfWeek #Get day of week from compact
            $dayofweek2 = $query.DayNumberInMonth #get day of month ie, first second etc
            $dayofmonth = $query2.DayNumberInMonth #day of week for full backup
            $dayofmonth2 = $query2.DayOfWeek #day of month for full backup
            [string]$compactday = $dayofweek2, $dayofweek1 #combine in a string to list "first monday" in column
            [string]$fullmonth = $dayofmonth, $dayofmonth2 #combine in a string to list "first monday" in column
           
            New-Object psobject -Property @{
                "Compacting" = $compactday
                "Name" = $Object.Name
                "FullBackup" = $fullmonth
                } | Select-Object Name,Compacting,FullBackup #remove this to, | export-csv -Path \\your-path\file.csv -Append
        }


Code: Select all
Name                                          Compacting                                         FullBackup                                           
----                                                ----------                                             ----------                                           
JOB NAME                                   First Monday                                          Third Wednesday                                       
JOB NAME                                   First Monday                                           First Monday                                         
JOB NAME                                   First Saturday                                         First Monday                                         
JOB NAME                                   First Saturday                                         First Monday                                         
JOB NAME                                   First Monday                                           First Monday                                         
JOB NAME                                   First Monday                                           Third Thursday


Hope you find it usefull, the next (nifty) thing to do would be to modify the script to randomize compacting and full backup, ideas?

Martin
ehrnst
Enthusiast
 
Posts: 35
Liked: 1 time
Joined: Fri Jan 31, 2014 8:24 am
Full Name: Martin Ehrnst

Re: Compacting and Full Backup

Veeam Logoby v.Eremin » Wed Mar 05, 2014 9:35 am

Hi, Martin, so the script lists both active full schedule and compact full schedule, right?

I'm wondering whether you know that compact full operation applies only to backup copy jobs. So, the provided script will lists relevant information regarding compact full, when it queries backup copy job. Meanwhile, for backup jobs the information regarding compact full will be the default one.

Being human, you intend to repeat what you are doing on a daily basis IE, copying existing jobs when you implement a new :wink:

Also, the existing jobs can be copied via Copy-VBRJob cmdlet.

Hope you find it usefull, the next (nifty) thing to do would be to modify the script to randomize compacting and full backup, ideas?

Not sure whether I completely follow you on that. You want to randomize the job active full schedule, so that, not so many full backups are performed on the same day. Or you're after something different?

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Compacting and Full Backup

Veeam Logoby ehrnst » Wed Mar 05, 2014 9:47 am

Hi, Martin, so the script lists both active full schedule and compact full schedule, right?


From what i know, yes. There where a lot of options, classes and properties to look for, but i believe i used the correct ones.

I'm wondering whether you know that compact full operation applies only to backup copy jobs. So, the provided script will lists relevant information regarding compact full, when it queries backup copy job. Meanwhile, for backup jobs the information regarding compact full will be the default one
.

Yes, i figured that after i ran the script the first time. I don't quite understand what operation veeam will perform on the backup jobs on "First Monday" which seems to be the default, will it compact or not?

Being human, you intend to repeat what you are doing on a daily basis IE, copying existing jobs when you implement a new :wink:

Also, the existing jobs can be copied via Copy-VBRJob cmdlet.


I know, but I still have to modify the schedule, right?

Hope you find it usefull, the next (nifty) thing to do would be to modify the script to randomize compacting and full backup, ideas?

Not sure whether I completely follow you on that. You want to randomize the job active full schedule, so that, not so many full backups are performed on the same day. Or you're after something different?


Correct. Having the ability to randomize active full and compact through a script would be great
ehrnst
Enthusiast
 
Posts: 35
Liked: 1 time
Joined: Fri Jan 31, 2014 8:24 am
Full Name: Martin Ehrnst

Re: Compacting and Full Backup list of schedule

Veeam Logoby v.Eremin » Wed Mar 05, 2014 10:56 am 1 person likes this post

Yes, i figured that after i ran the script the first time. I don't quite understand what operation veeam will perform on the backup jobs on "First Monday" which seems to be the default, will it compact or not?

No, it won't. Compact full is the concept of backup copy job that has nothing to with backup job. It's shown there due to the fact that both backup and backup copy job have the same parent class (CJob, I believe), and, thus, have the same combination of options. However, changing these settings (compact full) will have no effect on backup.

May be you want to list synthetic full schedule, instead?

I know, but I still have to modify the schedule, right?

Yes, but other settings will be preserved, and you won't have to input them manually from the scratch.

Correct. Having the ability to randomize active full and compact through a script would be great

How many jobs you want to randomize this way? In other words, how many full backups should be performed per day? How frequently the full backups should be created?

Anyway, I would suggest splitting existing jobs into several chunks in accordance with the number of acceptable daily full backups. Then, modify the settings of given chunks, so that, first group of jobs will create full backups on Monday, second group on Tuesday, etc.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Compacting and Full Backup list of schedule

Veeam Logoby ehrnst » Wed Mar 05, 2014 11:25 am

Thank you.

Let me try to explain a little more in detail what we experienced.
At first we had a lot of backup copy jobs set to do a compact the first saturday/sunday every month, combined with a lot of backup jobs where scheduled to perform an Active Full first monday. Therefor i wrote the script to list all backup and copy jobs with their settings for Compacting and Full backup schedule. Putting some time in to it, I could probably modify the script to split backup an copy jobs and then loop through before checking for full backup for backup jobs, and compact settings for the copy jobs.

As far as randomizing goes, a chunk of jobs is probably a good aproach. for example setting full backup for XX to run first monday YY to second monday and so on.
ehrnst
Enthusiast
 
Posts: 35
Liked: 1 time
Joined: Fri Jan 31, 2014 8:24 am
Full Name: Martin Ehrnst

Re: Compacting and Full Backup list of schedule

Veeam Logoby v.Eremin » Wed Mar 05, 2014 1:27 pm

I could probably modify the script to split backup an copy jobs and then loop through before checking for full backup for backup jobs, and compact settings for the copy jobs.

It can be done with some "auditorial" part that will check the job type.

For instance, you can list all backup jobs with the following script:

Code: Select all
Get-VBRJob | where {$_.Jobtype -eq "Backup"}


Backup Copy Jobs:

Code: Select all
Get-VBRJob | where {$_.Jobtype -eq "BackupSync"}


As far as randomizing goes, a chunk of jobs is probably a good aproach. for example setting full backup for XX to run first monday YY to second monday and so on.

The first step will be to understand how many daily full backup you're willing to have, and, then, group the jobs in accordance with it.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Compacting and Full Backup list of schedule

Veeam Logoby ehrnst » Wed Mar 05, 2014 3:49 pm

Yes, finding the job types isn't a problem. I will have to analyze how many full backup we wan't to run simultaneously before I attempt to write the script.

I guess this is a good start, but it still require us to do some manual labor :)

Code: Select all
Add-PSSnapin VeeamPSSnapin
$BackupJobs = Get-VBRJob | where {($_.jobtype -eq "BackupSync") -and ($_.Name -Match "^E.*") + ($_.Name -Match "^F.*")}
$options = $Object.getoptions()
$options.GenerationPolicy.CompactFullBackupMonthlyScheduleOptions.DayNumberInMonth = "Second"
ForEach ($Object in $BackupJobs)
    {
    $Options.GenerationPolicy.EnableCompactFull = $True
    $BackupJobs.SetOptions($options)
    write-host "OK  "$object.Name
    }


This should (not tested) search through all copy jobs starting with the letter "E" and "F" before setting second day in month (ie the second monday if it was monday earlier) and enable Compact
ehrnst
Enthusiast
 
Posts: 35
Liked: 1 time
Joined: Fri Jan 31, 2014 8:24 am
Full Name: Martin Ehrnst

Re: Compacting and Full Backup list of schedule

Veeam Logoby v.Eremin » Wed Mar 05, 2014 4:30 pm

It seems that following lines should be located within the cycle, not outside of it:

Code: Select all
$options = $Object.getoptions()
$options.GenerationPolicy.CompactFullBackupMonthlyScheduleOptions.DayNumberInMonth = "Second"


Code: Select all
Add-PSSnapin VeeamPSSnapin
$BackupJobs = Get-VBRJob | where {($_.jobtype -eq "BackupSync") -and ($_.Name -Match "^E.*") + ($_.Name -Match "^F.*")}
ForEach ($Object in $BackupJobs)
    {
    $options = $Object.getoptions()
    $options.GenerationPolicy.CompactFullBackupMonthlyScheduleOptions.DayNumberInMonth = "Second"
    $Options.GenerationPolicy.EnableCompactFull = $True
    $BackupJobs.SetOptions($options)
    write-host "OK  "$object.Name
    }
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Compacting and Full Backup list of schedule

Veeam Logoby ehrnst » Fri Mar 07, 2014 11:35 am

Found some time to work on this today, so i created the following script. Worked perfectly. No it is time to modify it so i can reschedule all our backup jobs' full backup as well

Code: Select all
Add-PSSnapin VeeamPSSnapin
$CompanyCode = Read-Host "First letter in CopyJob"
#$CompanyCode2 = Read-Host "First letter in CopyJob (second)"
$CopyJobs = Get-VBRJob | where {($_.jobtype -eq "BackupSync") -and ($_.Name -like "$companycode*")}
$options = $Object.getoptions()
$DayInMonth = ("First","Second","Third","Fourth") | Get-Random
$Day = ("Saturday", "Sunday", "Monday", "Tuesday") | Get-Random
ForEach ($Object in $CopyJobs)
    {
    $options.GenerationPolicy.CompactFullBackupMonthlyScheduleOptions.DayNumberInMonth = "$DayInMonth"
    $options.GenerationPolicy.CompactFullBackupMonthlyScheduleOptions.DayOfWeek = "$Day"
    $Options.GenerationPolicy.EnableCompactFull = $True
    $CopyJobs.SetOptions($options)
    write-host "OK  "$object.Name "Compact full set to $DayInMonth" "$day every month"
    }
ehrnst
Enthusiast
 
Posts: 35
Liked: 1 time
Joined: Fri Jan 31, 2014 8:24 am
Full Name: Martin Ehrnst

Re: Compacting and Full Backup list of schedule

Veeam Logoby v.Eremin » Fri Mar 07, 2014 3:36 pm

Glad to hear that you were able to write the required script.

Having already written that, it wouldn't be that hard to implement similar solution for backup jobs. Probably, the only part that requires changes is the code within the cycle.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Compacting and Full Backup list of schedule

Veeam Logoby ehrnst » Wed Mar 12, 2014 11:02 am 1 person likes this post

Yep, no problem. Code attached for those who's interested.

Code: Select all
#Loop through backup jobs, and set Monthly active full backup#
Add-PSSnapin VeeamPSSnapin #Load veeam snapin
#$CompanyCode = Read-Host "First letter in BackupJob" #User input for first letters in job"
$BackupJobs = Get-VBRJob | where {($_.jobtype -eq "Backup") -and ($_.Name -like "Name*") -and ($_.IsScheduleEnabled -eq $true)} #Search for enabled jobs with imput from user
$FullBackupMonth = "January","March", "May", "July","September","November" #Month's where you want active full backup to run 
   ForEach ($Object in $BackupJobs)
    {
    $options = $null
    $Day = "Sunday","Monday", "Tuesday", "Wednesday", "Thursday", "Friday" | Get-Random #fill inn for weekdays - will randomize
    $DayNumber = "First", "Second", "Third", "Fourth" | get-random #fill inn for week day number - will randomize
    $options = $Object.getoptions() #get options
    $options.BackupStorageOptions.EnableFullBackup = $true
    $options.BackupTargetOptions.FullBackupMonthlyScheduleOptions.DayOfWeek = $Day #insert the random day Monday, Wednesday
    $options.BackupTargetOptions.FullBackupMonthlyScheduleOptions.DayNumberInMonth = $DayNumber #Inserts daynumber
    $options.BackupTargetOptions.FullBackupMonthlyScheduleOptions.Months = $FullBackupMonth #sets month's to perform backup
    $options.BackupTargetOptions.FullBackupScheduleKind = "Monthly" #Monthly Schedule
    $Object.SetOptions($options) #set options to Backupjobs variable
    write-host "OK  "$object.Name "Active Full Backup date set" # list out the jobs and parameters set
    }
ehrnst
Enthusiast
 
Posts: 35
Liked: 1 time
Joined: Fri Jan 31, 2014 8:24 am
Full Name: Martin Ehrnst

Re: Compacting and Full Backup list of schedule

Veeam Logoby ehrnst » Thu Mar 13, 2014 8:40 am

Okay, i have a slight problem. It seems that after I ran my script to randomize all our copy jobs' monthly schedlule, other settings, not in the script was edited. All our copy jobs restore points was set to 7, and settings for jobs with WAN accelerator was set to "direct". Also the time for the jobs to start was reset to "00:00"

Is this by design?
ehrnst
Enthusiast
 
Posts: 35
Liked: 1 time
Joined: Fri Jan 31, 2014 8:24 am
Full Name: Martin Ehrnst

Re: Compacting and Full Backup list of schedule

Veeam Logoby ehrnst » Mon Mar 17, 2014 2:18 pm

Please let me know if this behavior is by design, or if it is wrong. If it is meant to work this way, i would like to know a way to work around it :)
ehrnst
Enthusiast
 
Posts: 35
Liked: 1 time
Joined: Fri Jan 31, 2014 8:24 am
Full Name: Martin Ehrnst

Re: Compacting and Full Backup list of schedule

Veeam Logoby v.Eremin » Thu Mar 20, 2014 7:52 am

As mentioned, have you tried to locate the .getOptions() part within the cycle, not outside it? As far as I can tell from the "backup copy" script, it's still outside the cycle. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Compacting and Full Backup list of schedule

Veeam Logoby ehrnst » Fri Mar 21, 2014 1:53 pm

Hmm... that could be it. I see now, that i have put it within the cycle for the backup job script. I will test in our lab to see what happens. but this means that we all have to acquire the options set for each job, before we can set new or edit existing options.
ehrnst
Enthusiast
 
Posts: 35
Liked: 1 time
Joined: Fri Jan 31, 2014 8:24 am
Full Name: Martin Ehrnst

Next

Return to PowerShell



Who is online

Users browsing this forum: No registered users and 3 guests