Script too slow, looking for optimisation

PowerShell script exchange

Script too slow, looking for optimisation

Veeam Logoby Ctek » Thu Dec 08, 2016 5:15 pm

Hi,

We are suffering from a script slowness due to too much data coming out of the Get-VBRBackupSession command. We have a big Veeam B&R with a LOT of jobs and we need to slim down the processing of the following script below, or accelerate it's processing somehow. The usual suspects have been accounted for (Veeam B&R server has a lot of available resources, All-Flash storage, SQL Standard instead of express etc).

Currently, this script takes over a minute to execute, which is unacceptable for our scenario:

Code: Select all
foreach ($i in (Get-VBRBackupSession | ?{ $_.JobId -eq  "JOBID" } | Sort-Object CreationTime -Descending | select -First 25))
{
               foreach ($j in (Get-VBRTaskSession $i))
                {               
                $sessionProps = @{
                     Name = $j.Name
                     Id = $j.Id
                     SessionId = $i.Id
                     SessionCreationTime = $i.CreationTime
                     SessionResult = $i.Result.ToString()
                     Status = $j.Status.ToString()
                     JobName = $j.JobName
                     VirtualMachine = $j.Info.ObjectName
                     Reason = $j.Info.Reason
                     QueuedTime = $j.Info.QueuedTime
          } 
          New-Object PSObject -Property $sessionProps
       }
}


We are looking to reach our goal of having the information in VBRTaskSession without taking over 60 seconds to load VBRBackupSession.

Any ideas? We will probably open a ticket today for this issue.
VMCE 9 Certified - Systems Administrator
Ctek
Service Provider
 
Posts: 38
Liked: 3 times
Joined: Wed Nov 11, 2015 3:50 pm
Location: Canada

Re: Script too slow, looking for optimisation

Veeam Logoby tsightler » Thu Dec 08, 2016 7:31 pm

The problem is Get-VBRBackupSession alone is probably taking almost all of this time, it's a slow command because it always returns every single session and is non-optimal on the backend. Unfortunately, it looks like it might be even worse in 9.5, at least based on my lab results. Are you on 9.5 already?

I'm trying to think of another way to get at the data you're looking for, but it's not coming to me. Any change you are trying to call Get-VBRBackupSession mulitple times, for example once per job, and perhaps you could move it outside of that code. I know you're not doing that in the code you posted, but that code appears to be running for a specific job, so I was just hoping that it was perhaps just a subset of a bigger loop. That's the only thing I can think of off the top of my head.

Opening a case wouldn't be a bad idea.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Script too slow, looking for optimisation

Veeam Logoby Ctek » Thu Dec 08, 2016 7:41 pm

We're on 9.5 and indeed likewise, it seems to take more time since we upgraded.
VMCE 9 Certified - Systems Administrator
Ctek
Service Provider
 
Posts: 38
Liked: 3 times
Joined: Wed Nov 11, 2015 3:50 pm
Location: Canada

Re: Script too slow, looking for optimisation

Veeam Logoby Ctek » Thu Dec 08, 2016 8:13 pm

tsightler wrote:The problem is Get-VBRBackupSession alone is probably taking almost all of this time, it's a slow command because it always returns every single session and is non-optimal on the backend. Unfortunately, it looks like it might be even worse in 9.5, at least based on my lab results. Are you on 9.5 already?

I'm trying to think of another way to get at the data you're looking for, but it's not coming to me. Any change you are trying to call Get-VBRBackupSession mulitple times, for example once per job, and perhaps you could move it outside of that code. I know you're not doing that in the code you posted, but that code appears to be running for a specific job, so I was just hoping that it was perhaps just a subset of a bigger loop. That's the only thing I can think of off the top of my head.

Opening a case wouldn't be a bad idea.


Thanks for the input Tom, case opened #01999855.

PS: Nice meeting you at VeeamPAC, very informative and great sessions.
VMCE 9 Certified - Systems Administrator
Ctek
Service Provider
 
Posts: 38
Liked: 3 times
Joined: Wed Nov 11, 2015 3:50 pm
Location: Canada

Re: Script too slow, looking for optimisation

Veeam Logoby tsightler » Thu Dec 08, 2016 9:53 pm

OK, I'm also following up with this on some internal channels as well. I'll follow the case.

PAC was great, good participation from everyone! Thanks for coming out and joining us!
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Script too slow, looking for optimisation

Veeam Logoby tsightler » Fri Dec 09, 2016 3:01 am 3 people like this post

Did some digging into this and there's definitely something going on in the code underneath this cmdlet that is non-ideal. I've passed my findings on to PM/R&D and we'll see what they have to say soon enough. I've also let the support person working on this case know that this issue is not unique to your environment.

In the interim, I wanted to share the following workaround which I hacked together based on what I found. It's ugly, as it's calling directly into the backend .NET code, but it's doing things in a little different way from the cmdlet and this method returns only the sessions for the job very quickly (in my lab the differences is ~150 seconds vs ~5 seconds) and allows them to be filtered down to the most recent 25 very quickly before entering the inner loop. Unfortunately this call returns a slightly different object type than the Get-VBRTaskSession needs, so I have to make a call to another .NET function to return the correct object. It's something about this function that is actually the core problem with Get-VBRBackupSessions, in my lab calls to this function take ~25ms each time, and Get-VBRBackupSessions calls this function for every single session in the database while, with my hack, we only have to call it 25 times for the specific sessions we need.

This is just a hackish workaround and I'll be trying to get a fix out of dev, but it made the script run 20x faster in my lab, so I thought you might want to give it a spin and let me know how it works for you and if there's any bugs that I didn't catch.

Code: Select all
foreach ($i in ([Veeam.Backup.DBManager.CDBManager]::Instance.BackupJobsSessions.GetSessionsForJob($job.id) | Sort-Object CreationTime -Descending | select -First 25))
{
               foreach ($j in (Get-VBRTaskSession ([Veeam.Backup.Core.CBackupSession]::Create($i, $null))))
                {               
                $sessionProps = @{
                     Name = $j.Name
                     Id = $j.Id
                     SessionId = $i.Id
                     SessionCreationTime = $i.CreationTime
                     SessionResult = $i.Result.ToString()
                     Status = $j.Status.ToString()
                     JobName = $j.JobName
                     VirtualMachine = $j.Info.ObjectName
                     Reason = $j.Info.Reason
                     QueuedTime = $j.Info.QueuedTime
          } 
          New-Object PSObject -Property $sessionProps
       }
}
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Script too slow, looking for optimisation

Veeam Logoby v.Eremin » Fri Dec 09, 2016 3:19 pm

Just so you know - we're going to investigate the issue closer during next week. I'll update the topic, should more information be available. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Script too slow, looking for optimisation

Veeam Logoby Ctek » Fri Dec 09, 2016 3:37 pm

Thanks a lot Tom and Vladimir, we are eagerly waiting for further feedback on this issue. :)

The lab results that you have Tom are exactly what we are suffering from. We will look into the dirty workaround that you gave us and try to test it in our lab environment.
VMCE 9 Certified - Systems Administrator
Ctek
Service Provider
 
Posts: 38
Liked: 3 times
Joined: Wed Nov 11, 2015 3:50 pm
Location: Canada

Re: Script too slow, looking for optimisation

Veeam Logoby v.Eremin » Mon Dec 12, 2016 2:34 pm 1 person likes this post

The root cause of the experienced behaviour has been found. Currently, we're planning to provide a fix for it in Update 2. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Script too slow, looking for optimisation

Veeam Logoby Ctek » Mon Dec 12, 2016 6:53 pm

Great then, thanks for the prompt answers.

Another question, we have a similar delay behavior with Find-VBRHvEntity cmdlet, which takes now a serious amount of time when executed. Could it be a similar/related issue or it's in another ballpark? I think it's a separate matter since this cmdlet is looking into the backup infrastructure instead of Session in the DB.
VMCE 9 Certified - Systems Administrator
Ctek
Service Provider
 
Posts: 38
Liked: 3 times
Joined: Wed Nov 11, 2015 3:50 pm
Location: Canada

Re: Script too slow, looking for optimisation

Veeam Logoby Ctek » Mon Dec 12, 2016 8:52 pm

v.Eremin wrote:Just so you know - we're going to investigate the issue closer during next week. I'll update the topic, should more information be available. Thanks.


We applied Tom's fix in production, it's incredibly faster, we shaved off around 98% of the time it took before, thanks again.
VMCE 9 Certified - Systems Administrator
Ctek
Service Provider
 
Posts: 38
Liked: 3 times
Joined: Wed Nov 11, 2015 3:50 pm
Location: Canada

Re: Script too slow, looking for optimisation

Veeam Logoby v.Eremin » Wed Dec 14, 2016 10:52 am

Another question, we have a similar delay behavior with Find-VBRHvEntity cmdlet, which takes now a serious amount of time when executed. Could it be a similar/related issue or it's in another ballpark?

Nope. The problem related to performance drop of Get-VBRBackupSession cmdlet has been caused by additional piece of code added behind the curtains to the underlying procedure. The said piece does not seem to be present in Find-VBRHvEntity. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Script too slow, looking for optimisation

Veeam Logoby v.Eremin » Tue Dec 20, 2016 2:16 pm

Currently, we're planning to provide a fix for it in Update 2.

It seems that we will be able to make it into Update 1. It's turned out the fix is not that hard to implement. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Script too slow, looking for optimisation

Veeam Logoby Ctek » Tue Dec 20, 2016 3:06 pm

Excellent news Vladimir, thanks for the update.
VMCE 9 Certified - Systems Administrator
Ctek
Service Provider
 
Posts: 38
Liked: 3 times
Joined: Wed Nov 11, 2015 3:50 pm
Location: Canada

Re: Script too slow, looking for optimisation

Veeam Logoby dragos.rosu » Wed Dec 21, 2016 8:42 am

Hello,

i am also impacted by this slowness in running the scripts, is there any official release date for the update ?
dragos.rosu
Novice
 
Posts: 9
Liked: 9 times
Joined: Wed Oct 23, 2013 1:10 pm
Full Name: Dragos Rosu

Next

Return to PowerShell



Who is online

Users browsing this forum: No registered users and 4 guests