PowerShell script exchange
Post Reply
manuel.aigner
Enthusiast
Posts: 31
Liked: 5 times
Joined: Sep 25, 2017 8:25 am
Full Name: Manuel Aigner
Contact:

Get-VBRApplicationRestorePoint very slow

Post by manuel.aigner »

Hi there,

Is there a possibility to speed up this cmdlet? It seems to me the performance goes worse the more restore points are in Veeam System.
The invocation of this command here takes sometimes more than 5 minutes.

Code: Select all

 $rpStep1 = Get-VBRApplicationRestorePoint -SQL -Name $vmName -ErrorAction Stop
We have around 3000VMs, at least 50 RPs per VM and around 200 SQL-VMs.
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by oleg.feoktistov »

Hi Manuel,

Doesn't look expected to me.
Do you have Backup console and Configuration database installed on the same machine where B&R Server is hosted?

Thanks,
Oleg
manuel.aigner
Enthusiast
Posts: 31
Liked: 5 times
Joined: Sep 25, 2017 8:25 am
Full Name: Manuel Aigner
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by manuel.aigner »

Hi Oleg,
Of course. At this case I ran a posh-session on B&R-Server directly. This Server is all in one. SQL-Server (Enterprise Edition) and all Veeam B&R components on same machine. Sized with 12 vCores and 96GiB RAM and a quite fast storage in backend.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by veremin » 1 person likes this post

Oleg has raised this issue internally, we will reach you, once we have more information to share. Thanks!
manuel.aigner
Enthusiast
Posts: 31
Liked: 5 times
Joined: Sep 25, 2017 8:25 am
Full Name: Manuel Aigner
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by manuel.aigner »

Hi Vladimir,
Many thanks for information.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by veremin »

Just to set proper expectations - as dev and qa teams are currently busy working on releasing product update, we cannot guarantee that we have an answer rather soon.

But as soon as we have it, we will reach you here.

Thanks!
manuel.aigner
Enthusiast
Posts: 31
Liked: 5 times
Joined: Sep 25, 2017 8:25 am
Full Name: Manuel Aigner
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by manuel.aigner » 1 person likes this post

Ok, that's no problem.
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by oleg.feoktistov »

Hi Manuel,

I just wanted to clarify - do you observe this cmdlet's behaviour in VBR v10 or else?
In case it's v10, do you happen to remember if it was the same for the previous version you had?
I'd also advise to reach our support engineers and highlight the case ID here.
Meanwhile, we will follow up in our internal channel.

Thanks,
Oleg
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by veremin »

So, we confirmed the reported behaviour internally. We've marked it as bug and are planning to fix it in one of the next product versions. Thanks!
mdiver
Veeam Legend
Posts: 201
Liked: 33 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by mdiver »

We observe the same with Get-VBRRestorePoint in a script we have been using for many years now.
It became >50x slower with the upgrade to V10.
On 9.5U4 and before it was still ok.

Any news already on that? Maybe a fix/workaround?

Thanks, Mike
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by veremin »

It became >50x slower with the upgrade to V10.
On 9.5U4 and before it was still ok.
Just to confirm - were you using 9.5 U4 or 9.5 U4b? I'm wondering because the bug that we've confirmed seems to have existed in 9.5 U4b too. Thanks!
mdiver
Veeam Legend
Posts: 201
Liked: 33 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by mdiver »

We went from 9.5.4.2866 directly to 10.0.0.4461. So no a or b involved with this customer.
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by oleg.feoktistov »

Hi @mdiver,

Could you please share any outcome on your performance tests?
How many restore points do you have?
How much time does it take for Get-VBRRestorePoint to return all of them?

Much appreciated,
Oleg
mdiver
Veeam Legend
Posts: 201
Liked: 33 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by mdiver »

Background: We are running a script against Veeam that scans full vSphere clusters for available restore points and lists each VM with its most recent backup date.
In addition we list all VMs which never have been backed up. A mail report is generated.

On 9.5U4 the script running against a cluster with e.g. 83 VMs ran slightly less than a single minute.
On V10 the script for the same cluster with 83 VMs takes 36 (!) minutes.

The number of restore points for each VMs is equal.

The total number of restore points for each VM is not easy to estimate as we have backups in different SOBRs with GFS on top.
I would estimate each VM to have 30+another 20 restore points. Including 6 monthlys under GFS control.
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by oleg.feoktistov »

Hi,

I have raised this question internally with QA. We will let you know as soon as we have any results and decisions on that.

Thanks,
Oleg
jhoughes
Veeam Vanguard
Posts: 279
Liked: 112 times
Joined: Apr 20, 2017 4:19 pm
Full Name: Joe Houghes
Location: Castle Rock, CO
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by jhoughes »

Can you share your script? We may be able to provide more efficient methods to get the data.
Husband, Father, Solutions Architect, Geek Extraordinaire | @DenverVMUG, @AustinVMUG & @ATXPowerShell leader | VMware vExpert | Cisco Champion
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by oleg.feoktistov »

Hi @mdiver,

We tested Get-VBRRestorePoint with QA and in our labs for v10 execution lasts as long as for v9.5u4.
I'd recommend to raise a ticket with our support and post the script here. Maybe we could also help with its optimization.

Best regards,
Oleg
mdiver
Veeam Legend
Posts: 201
Liked: 33 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by mdiver »

Hi Oleg.

The problem happens in the following loop with the Get-VBRRestorePoint method.
We fetch all restore points from the backup chains in the loop.

Code: Select all

$backups = Get-VBRBackup;
for($i = 0; $i -lt $backups.Count; $i++) {
	Write-Host "Backup" ($i + 1) "/" $backups.Count;
	$vms = $backups[$i].GetObjects();
	for ($j = 0; $j -lt $vms.Count; $j++) {
		$restorePoints = Get-VBRRestorePoint $backups[$i] | Where {$_.VMName -eq $vms[$j].Name}		
		if ($restorePoints.Count -gt 1) {
			$lastBackup = $restorePoints[$restorePoints.Count - 1].CreationTime;
		} else {
			$lastBackup = $restorePoints.CreationTime;
		}		
		$object = New-Object System.Object
    	$object | Add-Member -Type NoteProperty -Name Uuid -Value $vms[$j].Uuid;
		$object | Add-Member -Type NoteProperty -Name Name -Value $vms[$j].Name;
		$object | Add-Member -Type NoteProperty -Name LastBackup -Value $lastBackup;;
		$Veeam_VMs += $object;
		Write-Host "    VM" ($j + 1) "/" $vms.Count "[Letzter 'RestorePoint' vom" $lastBackup "]";
	}
}
It used to be ~2s per VM for every iteration. Now we see >40s per VM for a certain job which is a backup copy job with >200 VMs (per-VM-chains).
The smaller jobs run fine. It began with the upgrade to V10. The job was as large in 9.5 U4 already.

Thanks and regards,
Michael
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: Get-VBRApplicationRestorePoint very slow

Post by oleg.feoktistov » 1 person likes this post

Hi Michael,

I am very sorry for being off the radar - been busy with various tasks lately.
Though we don't posses labs with jobs consisting of such amount of vms, I did test your script in 2 totally different labs to see the full picture. Both labs are with VBR v10:
- one with a total of 280 VMs backed up in almost 4k restore points. It all being a part of small jobs with approximately 2-30 VMs each.
- the other with backup copy job including 150 VMs (per-vm chains) with a total of almost 1.1k restore points.

So, I did see that your script runs differently for smaller vs. bigger jobs: 2 minutes in the first case (for all backups) vs. 11 minutes in the second case ( only for one backup!). And while it took some time in each iteration to get all objects in a backup, I didn't want to believe it takes that amount of time to get the information you are after.
What I finally found to be quite a consumer is a list of all restore points being parsed in each child loop iteration. Instead of just filtering already parsed restore points with needed vm names, powershell takes some time to run through them over and over invoking the cmdlet, extract those with fitting parameters and only then assign to a variable. When I put restore points invocation to the parent loop and just left the filtering in the child one be, it solved the case. Below you can see the whole script modified and stopwatch results for one iteration over backup with 150 VMs:

Code: Select all

$Veeam_VMs = @()
$backups = Get-VBRBackup -Name 'BackupCopy*';
$stopwatch = [System.Diagnostics.Stopwatch]::StartNew()
for($i = 0; $i -lt $backups.Count; $i++) {
	Write-Host "Backup" ($i + 1) "/" $backups.Count;
	$vms = $backups[$i].GetObjects();
        $restorePoints = Get-VBRRestorePoint -Backup $backups[$i]
	for ($j = 0; $j -lt $vms.Count; $j++) {
		  $RPs = $restorePoints | Where {$_.VMName -eq $vms[$j].Name}		
		if ($RPs.Count -gt 1) {
			$lastBackup = $RPs[$RPs.Count - 1].CreationTime;
		} else {
			$lastBackup = $RPs.CreationTime;
		}		
		$object = New-Object System.Object
    	        $object | Add-Member -Type NoteProperty -Name Uuid -Value $vms[$j].Uuid;
		$object | Add-Member -Type NoteProperty -Name Name -Value $vms[$j].Name;
		$object | Add-Member -Type NoteProperty -Name LastBackup -Value $lastBackup;
		$Veeam_VMs += $object;
		Write-Host "    VM" ($j + 1) "/" $vms.Count "[Letzter 'RestorePoint' vom" $lastBackup "]";
	}
}
$stopwatch.Elapsed 


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 38
Milliseconds      : 272
Ticks             : 382725691
TotalDays         : 0.000442969549768519
TotalHours        : 0.0106312691944444
TotalMinutes      : 0.637876151666667
TotalSeconds      : 38.2725691
TotalMilliseconds : 38272.5691

As per the stopwatch, one iteration took 11 minutes before I applied the changes.

Hope that helps!

Oleg
Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests