PowerShell script exchange
Ctek
Service Provider
Posts: 83
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Script too slow, looking for optimisation

Post by Ctek »

Hi,

We are suffering from a script slowness due to too much data coming out of the Get-VBRBackupSession command. We have a big Veeam B&R with a LOT of jobs and we need to slim down the processing of the following script below, or accelerate it's processing somehow. The usual suspects have been accounted for (Veeam B&R server has a lot of available resources, All-Flash storage, SQL Standard instead of express etc).

Currently, this script takes over a minute to execute, which is unacceptable for our scenario:

Code: Select all

foreach ($i in (Get-VBRBackupSession | ?{ $_.JobId -eq  "JOBID" } | Sort-Object CreationTime -Descending | select -First 25))
{ 
               foreach ($j in (Get-VBRTaskSession $i)) 
                {               
                $sessionProps = @{ 
                     Name = $j.Name
                     Id = $j.Id
                     SessionId = $i.Id
                     SessionCreationTime = $i.CreationTime
                     SessionResult = $i.Result.ToString()
                     Status = $j.Status.ToString()
                     JobName = $j.JobName
                     VirtualMachine = $j.Info.ObjectName
                     Reason = $j.Info.Reason
                     QueuedTime = $j.Info.QueuedTime
          }  
          New-Object PSObject -Property $sessionProps 
       }
}
We are looking to reach our goal of having the information in VBRTaskSession without taking over 60 seconds to load VBRBackupSession.

Any ideas? We will probably open a ticket today for this issue.
VMCE
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Script too slow, looking for optimisation

Post by tsightler »

The problem is Get-VBRBackupSession alone is probably taking almost all of this time, it's a slow command because it always returns every single session and is non-optimal on the backend. Unfortunately, it looks like it might be even worse in 9.5, at least based on my lab results. Are you on 9.5 already?

I'm trying to think of another way to get at the data you're looking for, but it's not coming to me. Any change you are trying to call Get-VBRBackupSession mulitple times, for example once per job, and perhaps you could move it outside of that code. I know you're not doing that in the code you posted, but that code appears to be running for a specific job, so I was just hoping that it was perhaps just a subset of a bigger loop. That's the only thing I can think of off the top of my head.

Opening a case wouldn't be a bad idea.
Ctek
Service Provider
Posts: 83
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: Script too slow, looking for optimisation

Post by Ctek »

We're on 9.5 and indeed likewise, it seems to take more time since we upgraded.
VMCE
Ctek
Service Provider
Posts: 83
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: Script too slow, looking for optimisation

Post by Ctek »

tsightler wrote:The problem is Get-VBRBackupSession alone is probably taking almost all of this time, it's a slow command because it always returns every single session and is non-optimal on the backend. Unfortunately, it looks like it might be even worse in 9.5, at least based on my lab results. Are you on 9.5 already?

I'm trying to think of another way to get at the data you're looking for, but it's not coming to me. Any change you are trying to call Get-VBRBackupSession mulitple times, for example once per job, and perhaps you could move it outside of that code. I know you're not doing that in the code you posted, but that code appears to be running for a specific job, so I was just hoping that it was perhaps just a subset of a bigger loop. That's the only thing I can think of off the top of my head.

Opening a case wouldn't be a bad idea.
Thanks for the input Tom, case opened #01999855.

PS: Nice meeting you at VeeamPAC, very informative and great sessions.
VMCE
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Script too slow, looking for optimisation

Post by tsightler »

OK, I'm also following up with this on some internal channels as well. I'll follow the case.

PAC was great, good participation from everyone! Thanks for coming out and joining us!
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Script too slow, looking for optimisation

Post by tsightler » 3 people like this post

Did some digging into this and there's definitely something going on in the code underneath this cmdlet that is non-ideal. I've passed my findings on to PM/R&D and we'll see what they have to say soon enough. I've also let the support person working on this case know that this issue is not unique to your environment.

In the interim, I wanted to share the following workaround which I hacked together based on what I found. It's ugly, as it's calling directly into the backend .NET code, but it's doing things in a little different way from the cmdlet and this method returns only the sessions for the job very quickly (in my lab the differences is ~150 seconds vs ~5 seconds) and allows them to be filtered down to the most recent 25 very quickly before entering the inner loop. Unfortunately this call returns a slightly different object type than the Get-VBRTaskSession needs, so I have to make a call to another .NET function to return the correct object. It's something about this function that is actually the core problem with Get-VBRBackupSessions, in my lab calls to this function take ~25ms each time, and Get-VBRBackupSessions calls this function for every single session in the database while, with my hack, we only have to call it 25 times for the specific sessions we need.

This is just a hackish workaround and I'll be trying to get a fix out of dev, but it made the script run 20x faster in my lab, so I thought you might want to give it a spin and let me know how it works for you and if there's any bugs that I didn't catch.

Code: Select all

foreach ($i in ([Veeam.Backup.DBManager.CDBManager]::Instance.BackupJobsSessions.GetSessionsForJob($job.id) | Sort-Object CreationTime -Descending | select -First 25))
{ 
               foreach ($j in (Get-VBRTaskSession ([Veeam.Backup.Core.CBackupSession]::Create($i, $null)))) 
                {               
                $sessionProps = @{ 
                     Name = $j.Name
                     Id = $j.Id
                     SessionId = $i.Id
                     SessionCreationTime = $i.CreationTime
                     SessionResult = $i.Result.ToString()
                     Status = $j.Status.ToString()
                     JobName = $j.JobName
                     VirtualMachine = $j.Info.ObjectName
                     Reason = $j.Info.Reason
                     QueuedTime = $j.Info.QueuedTime
          }  
          New-Object PSObject -Property $sessionProps 
       }
}
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Script too slow, looking for optimisation

Post by veremin »

Just so you know - we're going to investigate the issue closer during next week. I'll update the topic, should more information be available. Thanks.
Ctek
Service Provider
Posts: 83
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: Script too slow, looking for optimisation

Post by Ctek »

Thanks a lot Tom and Vladimir, we are eagerly waiting for further feedback on this issue. :)

The lab results that you have Tom are exactly what we are suffering from. We will look into the dirty workaround that you gave us and try to test it in our lab environment.
VMCE
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Script too slow, looking for optimisation

Post by veremin » 1 person likes this post

The root cause of the experienced behaviour has been found. Currently, we're planning to provide a fix for it in Update 2. Thanks.
Ctek
Service Provider
Posts: 83
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: Script too slow, looking for optimisation

Post by Ctek »

Great then, thanks for the prompt answers.

Another question, we have a similar delay behavior with Find-VBRHvEntity cmdlet, which takes now a serious amount of time when executed. Could it be a similar/related issue or it's in another ballpark? I think it's a separate matter since this cmdlet is looking into the backup infrastructure instead of Session in the DB.
VMCE
Ctek
Service Provider
Posts: 83
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: Script too slow, looking for optimisation

Post by Ctek »

v.Eremin wrote:Just so you know - we're going to investigate the issue closer during next week. I'll update the topic, should more information be available. Thanks.
We applied Tom's fix in production, it's incredibly faster, we shaved off around 98% of the time it took before, thanks again.
VMCE
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Script too slow, looking for optimisation

Post by veremin »

Another question, we have a similar delay behavior with Find-VBRHvEntity cmdlet, which takes now a serious amount of time when executed. Could it be a similar/related issue or it's in another ballpark?
Nope. The problem related to performance drop of Get-VBRBackupSession cmdlet has been caused by additional piece of code added behind the curtains to the underlying procedure. The said piece does not seem to be present in Find-VBRHvEntity. Thanks.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Script too slow, looking for optimisation

Post by veremin »

Currently, we're planning to provide a fix for it in Update 2.
It seems that we will be able to make it into Update 1. It's turned out the fix is not that hard to implement. Thanks.
Ctek
Service Provider
Posts: 83
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: Script too slow, looking for optimisation

Post by Ctek »

Excellent news Vladimir, thanks for the update.
VMCE
dragos.rosu
Novice
Posts: 9
Liked: 9 times
Joined: Oct 23, 2013 1:10 pm
Full Name: Dragos Rosu
Contact:

Re: Script too slow, looking for optimisation

Post by dragos.rosu »

Hello,

i am also impacted by this slowness in running the scripts, is there any official release date for the update ?
tdewin
Veeam Software
Posts: 1775
Liked: 646 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: Script too slow, looking for optimisation

Post by tdewin »

There is no official "fixed release date". We release when we consider it is ready to run in production. But do consider this post:
vmware-vsphere-f24/esxi-and-vcenter-6-5 ... ml#p216329
Ctek
Service Provider
Posts: 83
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: Script too slow, looking for optimisation

Post by Ctek »

v.Eremin wrote:Currently, we're planning to provide a fix for it in Update 2.
It seems that we will be able to make it into Update 1. It's turned out the fix is not that hard to implement. Thanks.
Hi Vladimir, it appears that this issue is not fixed in 9.5 Update 1. We tested it in several DEV environment and there is no change in the behavior.

We are still running on Tom's proposed fix in this thread in the meantime, which is running very well and performance is great.

Support case : 01999855
Do you know if you have a planned private hotfix or if the fix has been pushed to Update 2?

Thanks
VMCE
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Script too slow, looking for optimisation

Post by veremin »

We've removed additional db call that supposedly was making cmdlet crawl. However, it's been reported by several customers including yourself, the fix does not make any difference. So, we keep investigating it further. Thanks.
cromero
Influencer
Posts: 12
Liked: never
Joined: May 04, 2016 6:39 pm
Full Name: Carlos Romero
Contact:

Re: Script too slow, looking for optimisation

Post by cromero »

Hi guys,

I'm having same issue with this line of my script (It's taking 2h after 9.5 upgrade)

Code: Select all

$vbrtasksessions = (Get-VBRBackupSession |
    Where-Object {$_.JobType -eq "Backup" -and $_.EndTime -ge (Get-Date).addhours(-$HourstoCheck)}) |
    Get-VBRTaskSession | Where-Object {$_.Status -ne "Failed"}
Do you know how can I improve it?

Thanks!
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Script too slow, looking for optimisation

Post by veremin »

Kindly, take a look at the workaround proposed by Tom. Based on what I know, it works significantly faster than the cmdlet in question does.

In the meantime, we're about to conduct an internal cmdlet performance review to find and address week spots.

Thanks.
cromero
Influencer
Posts: 12
Liked: never
Joined: May 04, 2016 6:39 pm
Full Name: Carlos Romero
Contact:

Re: Script too slow, looking for optimisation

Post by cromero »

Thanks!

Don't know how to exactly query the DB here. Maybe Tom can help me?

Thanks!
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Script too slow, looking for optimisation

Post by tsightler »

The code below looks quite ugly, but I believe it will give you the same result in significantly less time, however, you'll definitely want to test it for accuracy in case I missed something, but I tried to keep the overall logic exactly the same it's just using some internal functions to get around the performance issues:

Code: Select all

$vbrtasksessions =
    foreach ($bsession in ([Veeam.Backup.DBManager.CDBManager]::Instance.BackupJobsSessions.GetAll() | Where-Object {$_.JobType -eq "Backup" -and $_.EndTime -ge (Get-Date).addhours(-$HourstoCheck)}))
    { 
        Get-VBRTaskSession ([Veeam.Backup.Core.CBackupSession]::Create($bsession, $null)) | Where-Object {$_.Status -ne "Failed"}
    }
cromero
Influencer
Posts: 12
Liked: never
Joined: May 04, 2016 6:39 pm
Full Name: Carlos Romero
Contact:

Re: Script too slow, looking for optimisation

Post by cromero »

Thanks! but it's failing :(

Code: Select all

Exception calling "Create" with "2" argument(s): "Object reference not set to an instance of an object."
At line:4 char:9
+         Get-VBRTaskSession ([Veeam.Backup.Core.CBackupSession]::Creat ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : NullReferenceExceptio
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Script too slow, looking for optimisation

Post by tsightler »

My guess is this is due to the fact that the initial script I posted had a small typo. I'm not really sure how that happened as I copied and pasted the script, but regardless, it was there, so I have corrected it in the post above.

If you look in the code you posted, near the very start of the second line, the $bsession variable is misspelled as $bsessison. That's most likely causing the error. Sorry about that.
cromero
Influencer
Posts: 12
Liked: never
Joined: May 04, 2016 6:39 pm
Full Name: Carlos Romero
Contact:

Re: Script too slow, looking for optimisation

Post by cromero »

Thanks!!!! Didn't notice!!! :( Monday... hehe

Now it works fine!
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Script too slow, looking for optimisation

Post by tsightler »

OK, let me know what kind of speedup you see. If it's still too slow there may be other things we can do.
cromero
Influencer
Posts: 12
Liked: never
Joined: May 04, 2016 6:39 pm
Full Name: Carlos Romero
Contact:

Re: Script too slow, looking for optimisation

Post by cromero »

tsightler wrote:OK, let me know what kind of speedup you see. If it's still too slow there may be other things we can do.
Now it's perfect! It was taking 2.30h before. Now 10 min :)
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Script too slow, looking for optimisation

Post by veremin »

Update to everyone experiencing this issue. We've implemented a hot fix and started distributing among our clients.

If you'd like to try it out, create a ticket and refer to case number 02046581.

Thanks.
masonit
Service Provider
Posts: 325
Liked: 23 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Maso
Contact:

Re: Script too slow, looking for optimisation

Post by masonit »

Hi

Will permanent fix be included in 9.5 update 2?

\Masonit
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Script too slow, looking for optimisation

Post by veremin »

Correct.
Post Reply

Who is online

Users browsing this forum: No registered users and 30 guests