Get-VBRRestorePoint extremely slow

[efd121]

I have a script that checks if each VM is meeting its RPO and a few weeks ago it started running significantly longer than normal. I narrowed down the command that is causing the most delay.

Does anyone have a suggestion what would be causing the Get-VBRRestorePoint command to take more than 2 minutes each time it runs? I'm wondering if there is a SQL maintenance job or index I can look at to help speed it up.

Dave

$RestorePoints = Get-VBRRestorePoint -Name $Server.Name | Sort-Object -Property CreationTime -Descending

[tsightler]

Do you happen to have any repositories offline? I've seen this cause performance issues with this cmdlet.

Perhaps more importantly, are you calling this cmdlet interatively in a loop? Get-VBRRestorePoint has never been particularly fast, if your code is calling it mulitple times inside a loop you can typically improve performance significantly by moving to a single run of the command outside of the loop, which get's all objects for all VMs, then only filter that object list per-VM in the loop. Here is a link to a forum thread that shows that example. It won't help with the total execution time of the cmdlet, but will reduce the number of runs of that cmdlet to one, if you were previously calling it multiple times.

[efd121]

Thanks for the reply, you are correct my code is calling it in a loop. I will take a look at that link and try to move my call outside the loop. I was not aware I could adjust my command to get the information for all VM vs filtering it with -Name

Dave

[veremin]

Indeed, you can. We recommend this approach (using hash table), whenever you're dealing with heavy cmdlets, such as Get-VBRRestorePoint, Find-VBRVi(Hv)Entitity, etc. Thanks.

[efd121]

I took a look at the link Tom provided and I'm having a problem getting the sample code to work. I"m receiving "Cannot index into a null array" from the line with the foreach.

I'm not very familiar with hash tables and I have not been able to determine why this occurring

foreach ($row in $DataSet.Tables[0].Rows)

[tsightler]

There is certainly no requirement to use a hash table, although that can provide the most efficient way to access individual VM sessions, but they can be a little tricky. If you want you can just do something like the below.

Outside of the loop:

Code: Select all

$AllRestorePoints = Get-VBRRestorePoint -Name $Server.Name | Sort-Object -Property CreationTime -Descending

So now you have a nice variable that contains every session, already sorted in order so inside the loop just do:

Code: Select all

$RestorePoints = $AllRestorePoint | ?{$_.VmName -eq $Server.Name}

That should still provide a big time savings.

[efd121]

Tom, thanks my script now runs in about 10 minutes instead of more than at day!

Here is the full script and a little background. My first script started with one I learned about while attending VeeamON that collected all backup sessions form the last 14 days. It worked great for me in the beginning but it was limited to VM’s that were already in backup jobs and it assumed a 24hr SLA for every VM.

This script first starts by collecting VM’s (~600) from vCenter from specific clusters. I do this because my Veeam backup environment is currently limited to our Data Center servers. I have a tag applied to each VM that I use determine its status for the report. Currently I have 4 tags, “24 Hours”, “7 Days”, “No Backup Required” and “Retired”. We use retired in the cases when a VM is being removed from our environment. The VM is deleted from the backup job and powered down but it’s not deleted from vCenter for 1-2 weeks depending on the business requirement. The no backup required is for those few VM’s we choose not to backup for various reasons. The 24 hour and 7 day tags are self-explanatory and thanks to the help from Tom I can easily add other tags for systems that require something more frequent than daily backups. I can simply add a new tag, update the script for that tag and as long as I schedule the report to run more frequently than the shortest recovery point I’m covered. I can also see a situation where I may need to run separate reports for each tag but for now a daily report works great for me. When a new VM is discovered that does not have the RPO tag set its reported as “RPO Hours not set!!” making it easy to find new VM’s and set the appropriate RPO tag. You could also add theses VM’s to a new or existing job instead of just flagging them in the report.

I left the code out that uses the $RPOFailure and $RPOSucess counts, I’m sending those counts to a database where we collect these and other statistics for our Data Center environment.

Code: Select all

asnp "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue
asnp "VeeamPSSnapIn" -ErrorAction SilentlyContinue

$vccServer = "<vCENTER SERVER>"
$vbrServer = "<Veeam Console>"
$clusters = @("<Cluster1>", "<Cluser2>", "<Cluster3>")
$reportFile = "c:\Support\Reports\VeeamDailyRPO.csv"
$EmailTo = "<YOUR EMAIL ADDRESS>"
$EmailSubject = "Veeam RPO Report"
$EmailSMTPServer = "<EMAIL SERVER>"
$EmailFrom = "Veeam.DO.NOT.REPLY@rsmus.com"
$EmailAttachment = $reportFile
$Tags = "24 Hours", "7 Days", "Retired", "No Backup Required"
$RPOFailure = 0
$RPOSuccess = 0

# Delete report file if it exists
if (Get-ChildItem -Path $reportFile -ErrorAction SilentlyContinue)
	{
	Remove-Item $reportFile
	}
# Connect to vCenter
try
	{
	Connect-VIServer $vccServer -ErrorAction:Stop | Out-Null
	}
catch
	{
	Write-Host "Could not connect to vCenter, script is exiting!" -ForegroundColor Red
	Exit
	}
# Connect to Veeam Console
try
	{
	if (!(Get-VBRServerSession))
		{
		Connect-VBRServer -Server $vbrServer | Out-Null
		}
	}
catch
	{
	Write-Host "Failed to connect to Veeam Backup Server, script is exiting!" -ForegroundColor Red
	Exit
	}

#Get list of servers and RPO data
$Servers = @()
New-VIProperty -Name Tag -ObjectType VirtualMachine -Value { Get-TagAssignment -Entity $args[0] -Category "RPO" | select -ExpandProperty Tag } | Out-Null
$servers = Get-Cluster -Name $clusters | Get-VM | select Name, Tag

$RPOTime = Get-Date
$results = @()
# Get all Restore Points
$AllRestorePoints = Get-VBRRestorePoint -Name $Servers.Name | Sort-Object -Property CreationTime -Descending
foreach ($server in $servers) 
	{
	$item = "" | Select VM, LastBackup, Status
    switch ($Server.Tag.Name)
		{
		"24 Hours" 
			{
#			Write-Host $Server.VM "is backed every 24hrs" -ForegroundColor Cyan
			$item.LastBackup = "Never" # Set default to Never
			$item.VM = $Server.Name
			$RestorePoints = $AllRestorePoints | ?{$_.VmName -eq $Server.Name}
			if ($RestorePoints -eq $null)
				{
				$item.Status = "Never backed up"
				$results += $item
				Break
				}
			$LastBackup = $RestorePoints[0].CreationTime
			$item.LastBackup = $LastBackup
			if (($LastBackup -lt $RPOTime.addhours(-24))-or ($RestorePoints -eq $null))
				{
				$item.Status = "RPO failure"
				$RPOFailure ++
				}
			else
				{
				$item.Status = "RPO has been met"
				$RPOSuccess ++
				}
			$results += $item
			}
		"7 Days" 
			{
#			Write-Host $Server.Name "is only backed every 7 days" -ForegroundColor Green
			$item.VM = $Server.Name
			$RestorePoints = $AllRestorePoints | ?{$_.VmName -eq $Server.Name}
			if ($RestorePoints -eq $null)
				{
				$item.Status = "Never backed up"
				$results += $item
				Break
				}
			$LastBackup = $RestorePoints[0].CreationTime
			$item.LastBackup = $LastBackup
			if ($LastBackup -gt $RPOTime.adddays(-7))
				{
				$item.Status = "RPO has been met"
				$RPOSuccess ++
				}
			else
				{
				$item.Status = "RPO failure"
				$RPOFailure ++
				}
			$results += $item
			}
		"Retired" 
			{
#			Write-Host $Server.Name "is retired" -ForegroundColor Red	
			$item.VM = $Server.Name
			$item.LastBackup = "NA"
			$item.Status = "Server has been retired"
			$results += $item
			}
		"No Backup Required" 
			{
#			Write-Host $Server.Name "does not require a backup" -ForegroundColor Green
			$item.VM = $Server.Name
			$item.LastBackup = "NA"
			$item.Status = "No backup required"
			$results += $item
			}
		default 
			{
#			Write-Host $Server.Name "Could not find a match!" -ForegroundColor Black
			$item.VM = $Server.Name
			$item.Status = "RPO Hours not set!!"
			$item.LastBackup = "Never"
			$results += $item
			}
		}
	}
$results | Export-Csv $reportFile -NoTypeInformation
Send-MailMessage -To $EmailTo -Subject $EmailSubject -SmtpServer $EmailSMTPServer -From $EmailFrom -Attachments $EmailAttachment 
Disconnect-VIServer $vccServer -Confirm:$false
Disconnect-VBRServer

[tsightler]

Nice, thanks for sharing!

Here is the full script and a little background. My first script started with one I learned about while attending VeeamON that collected all backup sessions form the last 14 days. It worked great for me in the beginning but it was limited to VM’s that were already in backup jobs and it assumed a 24hr SLA for every VM.

I think I may have written that initial script, I certainly talked about one pretty much exactly like that at the last VeeamON in one of my sessions. I'll take a look and see if I can figure out how to convert it to using a hash table. Assuming it's not taking 10 minutes to run Get-VBRBackupSession perhaps we can shave even more time.

[efd121]

Tom, I'm pretty sure you did write that original script. I thought that was the case when I was posting my reply earlier today but I couldn't find my notes and wasn't 100% sure.

[tsightler]

Here's an attempt to convert to using a hash table. There are no changes to the original other than the three lines that deal with the $AllRestorePoints variable. You could probably get a little more efficient by selecting only the specific properties that you use in the script, but I doubt that would make much difference from a performance perspective. Also, this script would not properly deal with cases where mulitple VMs have the same name, but that's true of the original as well.

If you give it a try and have any problems let me know, since you have a lot of sessions I'm hoping that storing the session data in a hash table will offer another significant boost in performance.

Code: Select all

asnp "VMware.VimAutomation.Core" -ErrorAction SilentlyContinue
asnp "VeeamPSSnapIn" -ErrorAction SilentlyContinue

$vccServer = "<vCENTER SERVER>"
$vbrServer = "<Veeam Console>"
$clusters = @("<Cluster1>", "<Cluser2>", "<Cluster3>")
$reportFile = "c:\Support\Reports\VeeamDailyRPO.csv"
$EmailTo = "<YOUR EMAIL ADDRESS>"
$EmailSubject = "Veeam RPO Report"
$EmailSMTPServer = "<EMAIL SERVER>"
$EmailFrom = "Veeam.DO.NOT.REPLY@rsmus.com"
$EmailAttachment = $reportFile
$Tags = "24 Hours", "7 Days", "Retired", "No Backup Required"
$RPOFailure = 0
$RPOSuccess = 0

# Delete report file if it exists
if (Get-ChildItem -Path $reportFile -ErrorAction SilentlyContinue)
   {
   Remove-Item $reportFile
   }
# Connect to vCenter
try
   {
   Connect-VIServer $vccServer -ErrorAction:Stop | Out-Null
   }
catch
   {
   Write-Host "Could not connect to vCenter, script is exiting!" -ForegroundColor Red
   Exit
   }
# Connect to Veeam Console
try
   {
   if (!(Get-VBRServerSession))
      {
      Connect-VBRServer -Server $vbrServer | Out-Null
      }
   }
catch
   {
   Write-Host "Failed to connect to Veeam Backup Server, script is exiting!" -ForegroundColor Red
   Exit
   }

#Get list of servers and RPO data
$Servers = @()
New-VIProperty -Name Tag -ObjectType VirtualMachine -Value { Get-TagAssignment -Entity $args[0] -Category "RPO" | select -ExpandProperty Tag } | Out-Null
$servers = Get-Cluster -Name $clusters | Get-VM | select Name, Tag

$RPOTime = Get-Date
$results = @()
# Get all Restore Points
$AllRestorePoints = Get-VBRRestorePoint | Group-Object -Property VmName -AsHashTable
foreach ($server in $servers) 
   {
   $item = "" | Select VM, LastBackup, Status
    switch ($Server.Tag.Name)
      {
      "24 Hours" 
         {
#         Write-Host $Server.VM "is backed every 24hrs" -ForegroundColor Cyan
         $item.LastBackup = "Never" # Set default to Never
         $item.VM = $Server.Name
         $RestorePoints = $AllRestorePoints.($Server.Name) | Sort-Object -Property CreationTime -Descending
         if ($RestorePoints -eq $null)
            {
            $item.Status = "Never backed up"
            $results += $item
            Break
            }
         $LastBackup = $RestorePoints[0].CreationTime
         $item.LastBackup = $LastBackup
         if (($LastBackup -lt $RPOTime.addhours(-24))-or ($RestorePoints -eq $null))
            {
            $item.Status = "RPO failure"
            $RPOFailure ++
            }
         else
            {
            $item.Status = "RPO has been met"
            $RPOSuccess ++
            }
         $results += $item
         }
      "7 Days" 
         {
#         Write-Host $Server.Name "is only backed every 7 days" -ForegroundColor Green
         $item.VM = $Server.Name
         $RestorePoints = $AllRestorePoints.($Server.Name) | Sort-Object -Property CreationTime -Descending
         if ($RestorePoints -eq $null)
            {
            $item.Status = "Never backed up"
            $results += $item
            Break
            }
         $LastBackup = $RestorePoints[0].CreationTime
         $item.LastBackup = $LastBackup
         if ($LastBackup -gt $RPOTime.adddays(-7))
            {
            $item.Status = "RPO has been met"
            $RPOSuccess ++
            }
         else
            {
            $item.Status = "RPO failure"
            $RPOFailure ++
            }
         $results += $item
         }
      "Retired" 
         {
#         Write-Host $Server.Name "is retired" -ForegroundColor Red   
         $item.VM = $Server.Name
         $item.LastBackup = "NA"
         $item.Status = "Server has been retired"
         $results += $item
         }
      "No Backup Required" 
         {
#         Write-Host $Server.Name "does not require a backup" -ForegroundColor Green
         $item.VM = $Server.Name
         $item.LastBackup = "NA"
         $item.Status = "No backup required"
         $results += $item
         }
      default 
         {
#         Write-Host $Server.Name "Could not find a match!" -ForegroundColor Black
         $item.VM = $Server.Name
         $item.Status = "RPO Hours not set!!"
         $item.LastBackup = "Never"
         $results += $item
         }
      }
   }
$results | Export-Csv $reportFile -NoTypeInformation
Send-MailMessage -To $EmailTo -Subject $EmailSubject -SmtpServer $EmailSMTPServer -From $EmailFrom -Attachments $EmailAttachment 
Disconnect-VIServer $vccServer -Confirm:$false
Disconnect-VBRServer