Get-VBRApplicationRestorePoint very slow

[manuel.aigner]

Hi there,

Is there a possibility to speed up this cmdlet? It seems to me the performance goes worse the more restore points are in Veeam System.
The invocation of this command here takes sometimes more than 5 minutes.

Code: Select all

 $rpStep1 = Get-VBRApplicationRestorePoint -SQL -Name $vmName -ErrorAction Stop

We have around 3000VMs, at least 50 RPs per VM and around 200 SQL-VMs.

[oleg.feoktistov]

Hi Manuel,

Doesn't look expected to me.
Do you have Backup console and Configuration database installed on the same machine where B&R Server is hosted?

Thanks,
Oleg

[manuel.aigner]

Hi Oleg,
Of course. At this case I ran a posh-session on B&R-Server directly. This Server is all in one. SQL-Server (Enterprise Edition) and all Veeam B&R components on same machine. Sized with 12 vCores and 96GiB RAM and a quite fast storage in backend.

[veremin]

Oleg has raised this issue internally, we will reach you, once we have more information to share. Thanks!

[manuel.aigner]

Hi Vladimir,
Many thanks for information.

[veremin]

Just to set proper expectations - as dev and qa teams are currently busy working on releasing product update, we cannot guarantee that we have an answer rather soon.

But as soon as we have it, we will reach you here.

Thanks!

[manuel.aigner]

Ok, that's no problem.

[oleg.feoktistov]

Hi Manuel,

I just wanted to clarify - do you observe this cmdlet's behaviour in VBR v10 or else?
In case it's v10, do you happen to remember if it was the same for the previous version you had?
I'd also advise to reach our support engineers and highlight the case ID here.
Meanwhile, we will follow up in our internal channel.

Thanks,
Oleg

[veremin]

So, we confirmed the reported behaviour internally. We've marked it as bug and are planning to fix it in one of the next product versions. Thanks!

[mdiver]

We observe the same with Get-VBRRestorePoint in a script we have been using for many years now.
It became >50x slower with the upgrade to V10.
On 9.5U4 and before it was still ok.

Any news already on that? Maybe a fix/workaround?

Thanks, Mike

[veremin]

It became >50x slower with the upgrade to V10.
On 9.5U4 and before it was still ok.

Just to confirm - were you using 9.5 U4 or 9.5 U4b? I'm wondering because the bug that we've confirmed seems to have existed in 9.5 U4b too. Thanks!

[mdiver]

We went from 9.5.4.2866 directly to 10.0.0.4461. So no a or b involved with this customer.

[oleg.feoktistov]

Hi @mdiver,

Could you please share any outcome on your performance tests?
How many restore points do you have?
How much time does it take for Get-VBRRestorePoint to return all of them?

Much appreciated,
Oleg

[mdiver]

Background: We are running a script against Veeam that scans full vSphere clusters for available restore points and lists each VM with its most recent backup date.
In addition we list all VMs which never have been backed up. A mail report is generated.

On 9.5U4 the script running against a cluster with e.g. 83 VMs ran slightly less than a single minute.
On V10 the script for the same cluster with 83 VMs takes 36 (!) minutes.

The number of restore points for each VMs is equal.

The total number of restore points for each VM is not easy to estimate as we have backups in different SOBRs with GFS on top.
I would estimate each VM to have 30+another 20 restore points. Including 6 monthlys under GFS control.

[oleg.feoktistov]

Hi,

I have raised this question internally with QA. We will let you know as soon as we have any results and decisions on that.

Thanks,
Oleg

[jhoughes]

Can you share your script? We may be able to provide more efficient methods to get the data.

[oleg.feoktistov]

Hi @mdiver,

We tested Get-VBRRestorePoint with QA and in our labs for v10 execution lasts as long as for v9.5u4.
I'd recommend to raise a ticket with our support and post the script here. Maybe we could also help with its optimization.

Best regards,
Oleg

[mdiver]

Hi Oleg.

The problem happens in the following loop with the Get-VBRRestorePoint method.
We fetch all restore points from the backup chains in the loop.

Code: Select all

$backups = Get-VBRBackup;
for($i = 0; $i -lt $backups.Count; $i++) {
	Write-Host "Backup" ($i + 1) "/" $backups.Count;
	$vms = $backups[$i].GetObjects();
	for ($j = 0; $j -lt $vms.Count; $j++) {
		$restorePoints = Get-VBRRestorePoint $backups[$i] | Where {$_.VMName -eq $vms[$j].Name}		
		if ($restorePoints.Count -gt 1) {
			$lastBackup = $restorePoints[$restorePoints.Count - 1].CreationTime;
		} else {
			$lastBackup = $restorePoints.CreationTime;
		}		
		$object = New-Object System.Object
    	$object | Add-Member -Type NoteProperty -Name Uuid -Value $vms[$j].Uuid;
		$object | Add-Member -Type NoteProperty -Name Name -Value $vms[$j].Name;
		$object | Add-Member -Type NoteProperty -Name LastBackup -Value $lastBackup;;
		$Veeam_VMs += $object;
		Write-Host "    VM" ($j + 1) "/" $vms.Count "[Letzter 'RestorePoint' vom" $lastBackup "]";
	}
}

It used to be ~2s per VM for every iteration. Now we see >40s per VM for a certain job which is a backup copy job with >200 VMs (per-VM-chains).
The smaller jobs run fine. It began with the upgrade to V10. The job was as large in 9.5 U4 already.

Thanks and regards,
Michael

[oleg.feoktistov]

Hi Michael,

I am very sorry for being off the radar - been busy with various tasks lately.
Though we don't posses labs with jobs consisting of such amount of vms, I did test your script in 2 totally different labs to see the full picture. Both labs are with VBR v10:
- one with a total of 280 VMs backed up in almost 4k restore points. It all being a part of small jobs with approximately 2-30 VMs each.
- the other with backup copy job including 150 VMs (per-vm chains) with a total of almost 1.1k restore points.

So, I did see that your script runs differently for smaller vs. bigger jobs: 2 minutes in the first case (for all backups) vs. 11 minutes in the second case ( only for one backup!). And while it took some time in each iteration to get all objects in a backup, I didn't want to believe it takes that amount of time to get the information you are after.
What I finally found to be quite a consumer is a list of all restore points being parsed in each child loop iteration. Instead of just filtering already parsed restore points with needed vm names, powershell takes some time to run through them over and over invoking the cmdlet, extract those with fitting parameters and only then assign to a variable. When I put restore points invocation to the parent loop and just left the filtering in the child one be, it solved the case. Below you can see the whole script modified and stopwatch results for one iteration over backup with 150 VMs:

Code: Select all

$Veeam_VMs = @()
$backups = Get-VBRBackup -Name 'BackupCopy*';
$stopwatch = [System.Diagnostics.Stopwatch]::StartNew()
for($i = 0; $i -lt $backups.Count; $i++) {
	Write-Host "Backup" ($i + 1) "/" $backups.Count;
	$vms = $backups[$i].GetObjects();
        $restorePoints = Get-VBRRestorePoint -Backup $backups[$i]
	for ($j = 0; $j -lt $vms.Count; $j++) {
		  $RPs = $restorePoints | Where {$_.VMName -eq $vms[$j].Name}		
		if ($RPs.Count -gt 1) {
			$lastBackup = $RPs[$RPs.Count - 1].CreationTime;
		} else {
			$lastBackup = $RPs.CreationTime;
		}		
		$object = New-Object System.Object
    	        $object | Add-Member -Type NoteProperty -Name Uuid -Value $vms[$j].Uuid;
		$object | Add-Member -Type NoteProperty -Name Name -Value $vms[$j].Name;
		$object | Add-Member -Type NoteProperty -Name LastBackup -Value $lastBackup;
		$Veeam_VMs += $object;
		Write-Host "    VM" ($j + 1) "/" $vms.Count "[Letzter 'RestorePoint' vom" $lastBackup "]";
	}
}
$stopwatch.Elapsed 


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 38
Milliseconds      : 272
Ticks             : 382725691
TotalDays         : 0.000442969549768519
TotalHours        : 0.0106312691944444
TotalMinutes      : 0.637876151666667
TotalSeconds      : 38.2725691
TotalMilliseconds : 38272.5691

As per the stopwatch, one iteration took 11 minutes before I applied the changes.

Hope that helps!

Oleg