Script too slow, looking for optimisation

[Ctek]

Hi,

We are suffering from a script slowness due to too much data coming out of the Get-VBRBackupSession command. We have a big Veeam B&R with a LOT of jobs and we need to slim down the processing of the following script below, or accelerate it's processing somehow. The usual suspects have been accounted for (Veeam B&R server has a lot of available resources, All-Flash storage, SQL Standard instead of express etc).

Currently, this script takes over a minute to execute, which is unacceptable for our scenario:

Code: Select all

foreach ($i in (Get-VBRBackupSession | ?{ $_.JobId -eq  "JOBID" } | Sort-Object CreationTime -Descending | select -First 25))
{ 
               foreach ($j in (Get-VBRTaskSession $i)) 
                {               
                $sessionProps = @{ 
                     Name = $j.Name
                     Id = $j.Id
                     SessionId = $i.Id
                     SessionCreationTime = $i.CreationTime
                     SessionResult = $i.Result.ToString()
                     Status = $j.Status.ToString()
                     JobName = $j.JobName
                     VirtualMachine = $j.Info.ObjectName
                     Reason = $j.Info.Reason
                     QueuedTime = $j.Info.QueuedTime
          }  
          New-Object PSObject -Property $sessionProps 
       }
}

We are looking to reach our goal of having the information in VBRTaskSession without taking over 60 seconds to load VBRBackupSession.

Any ideas? We will probably open a ticket today for this issue.

[tsightler]

The problem is Get-VBRBackupSession alone is probably taking almost all of this time, it's a slow command because it always returns every single session and is non-optimal on the backend. Unfortunately, it looks like it might be even worse in 9.5, at least based on my lab results. Are you on 9.5 already?

I'm trying to think of another way to get at the data you're looking for, but it's not coming to me. Any change you are trying to call Get-VBRBackupSession mulitple times, for example once per job, and perhaps you could move it outside of that code. I know you're not doing that in the code you posted, but that code appears to be running for a specific job, so I was just hoping that it was perhaps just a subset of a bigger loop. That's the only thing I can think of off the top of my head.

Opening a case wouldn't be a bad idea.

[Ctek]

We're on 9.5 and indeed likewise, it seems to take more time since we upgraded.

[Ctek]

The problem is Get-VBRBackupSession alone is probably taking almost all of this time, it's a slow command because it always returns every single session and is non-optimal on the backend. Unfortunately, it looks like it might be even worse in 9.5, at least based on my lab results. Are you on 9.5 already?

I'm trying to think of another way to get at the data you're looking for, but it's not coming to me. Any change you are trying to call Get-VBRBackupSession mulitple times, for example once per job, and perhaps you could move it outside of that code. I know you're not doing that in the code you posted, but that code appears to be running for a specific job, so I was just hoping that it was perhaps just a subset of a bigger loop. That's the only thing I can think of off the top of my head.

Opening a case wouldn't be a bad idea.

Thanks for the input Tom, case opened #01999855.

PS: Nice meeting you at VeeamPAC, very informative and great sessions.

[tsightler]

OK, I'm also following up with this on some internal channels as well. I'll follow the case.

PAC was great, good participation from everyone! Thanks for coming out and joining us!

[tsightler]

Did some digging into this and there's definitely something going on in the code underneath this cmdlet that is non-ideal. I've passed my findings on to PM/R&D and we'll see what they have to say soon enough. I've also let the support person working on this case know that this issue is not unique to your environment.

In the interim, I wanted to share the following workaround which I hacked together based on what I found. It's ugly, as it's calling directly into the backend .NET code, but it's doing things in a little different way from the cmdlet and this method returns only the sessions for the job very quickly (in my lab the differences is ~150 seconds vs ~5 seconds) and allows them to be filtered down to the most recent 25 very quickly before entering the inner loop. Unfortunately this call returns a slightly different object type than the Get-VBRTaskSession needs, so I have to make a call to another .NET function to return the correct object. It's something about this function that is actually the core problem with Get-VBRBackupSessions, in my lab calls to this function take ~25ms each time, and Get-VBRBackupSessions calls this function for every single session in the database while, with my hack, we only have to call it 25 times for the specific sessions we need.

This is just a hackish workaround and I'll be trying to get a fix out of dev, but it made the script run 20x faster in my lab, so I thought you might want to give it a spin and let me know how it works for you and if there's any bugs that I didn't catch.

Code: Select all

foreach ($i in ([Veeam.Backup.DBManager.CDBManager]::Instance.BackupJobsSessions.GetSessionsForJob($job.id) | Sort-Object CreationTime -Descending | select -First 25))
{ 
               foreach ($j in (Get-VBRTaskSession ([Veeam.Backup.Core.CBackupSession]::Create($i, $null)))) 
                {               
                $sessionProps = @{ 
                     Name = $j.Name
                     Id = $j.Id
                     SessionId = $i.Id
                     SessionCreationTime = $i.CreationTime
                     SessionResult = $i.Result.ToString()
                     Status = $j.Status.ToString()
                     JobName = $j.JobName
                     VirtualMachine = $j.Info.ObjectName
                     Reason = $j.Info.Reason
                     QueuedTime = $j.Info.QueuedTime
          }  
          New-Object PSObject -Property $sessionProps 
       }
}

[veremin]

Just so you know - we're going to investigate the issue closer during next week. I'll update the topic, should more information be available. Thanks.

[Ctek]

Thanks a lot Tom and Vladimir, we are eagerly waiting for further feedback on this issue.

The lab results that you have Tom are exactly what we are suffering from. We will look into the dirty workaround that you gave us and try to test it in our lab environment.

[veremin]

The root cause of the experienced behaviour has been found. Currently, we're planning to provide a fix for it in Update 2. Thanks.

[Ctek]

Great then, thanks for the prompt answers.

Another question, we have a similar delay behavior with Find-VBRHvEntity cmdlet, which takes now a serious amount of time when executed. Could it be a similar/related issue or it's in another ballpark? I think it's a separate matter since this cmdlet is looking into the backup infrastructure instead of Session in the DB.

[Ctek]

Just so you know - we're going to investigate the issue closer during next week. I'll update the topic, should more information be available. Thanks.

We applied Tom's fix in production, it's incredibly faster, we shaved off around 98% of the time it took before, thanks again.

[veremin]

Another question, we have a similar delay behavior with Find-VBRHvEntity cmdlet, which takes now a serious amount of time when executed. Could it be a similar/related issue or it's in another ballpark?

Nope. The problem related to performance drop of Get-VBRBackupSession cmdlet has been caused by additional piece of code added behind the curtains to the underlying procedure. The said piece does not seem to be present in Find-VBRHvEntity. Thanks.

[veremin]

Currently, we're planning to provide a fix for it in Update 2.

It seems that we will be able to make it into Update 1. It's turned out the fix is not that hard to implement. Thanks.

[Ctek]

Excellent news Vladimir, thanks for the update.

[dragos.rosu]

Hello,

i am also impacted by this slowness in running the scripts, is there any official release date for the update ?

[tdewin]

There is no official "fixed release date". We release when we consider it is ready to run in production. But do consider this post:
vmware-vsphere-f24/esxi-and-vcenter-6-5 ... ml#p216329

[Ctek]

Currently, we're planning to provide a fix for it in Update 2.
It seems that we will be able to make it into Update 1. It's turned out the fix is not that hard to implement. Thanks.

Hi Vladimir, it appears that this issue is not fixed in 9.5 Update 1. We tested it in several DEV environment and there is no change in the behavior.

We are still running on Tom's proposed fix in this thread in the meantime, which is running very well and performance is great.

Support case : 01999855
Do you know if you have a planned private hotfix or if the fix has been pushed to Update 2?

Thanks

[veremin]

We've removed additional db call that supposedly was making cmdlet crawl. However, it's been reported by several customers including yourself, the fix does not make any difference. So, we keep investigating it further. Thanks.

[cromero]

Hi guys,

I'm having same issue with this line of my script (It's taking 2h after 9.5 upgrade)

Code: Select all

$vbrtasksessions = (Get-VBRBackupSession |
    Where-Object {$_.JobType -eq "Backup" -and $_.EndTime -ge (Get-Date).addhours(-$HourstoCheck)}) |
    Get-VBRTaskSession | Where-Object {$_.Status -ne "Failed"}

Do you know how can I improve it?

Thanks!

[veremin]

Kindly, take a look at the workaround proposed by Tom. Based on what I know, it works significantly faster than the cmdlet in question does.

In the meantime, we're about to conduct an internal cmdlet performance review to find and address week spots.

Thanks.

[cromero]

Thanks!

Don't know how to exactly query the DB here. Maybe Tom can help me?

Thanks!

[tsightler]

The code below looks quite ugly, but I believe it will give you the same result in significantly less time, however, you'll definitely want to test it for accuracy in case I missed something, but I tried to keep the overall logic exactly the same it's just using some internal functions to get around the performance issues:

Code: Select all

$vbrtasksessions =
    foreach ($bsession in ([Veeam.Backup.DBManager.CDBManager]::Instance.BackupJobsSessions.GetAll() | Where-Object {$_.JobType -eq "Backup" -and $_.EndTime -ge (Get-Date).addhours(-$HourstoCheck)}))
    { 
        Get-VBRTaskSession ([Veeam.Backup.Core.CBackupSession]::Create($bsession, $null)) | Where-Object {$_.Status -ne "Failed"}
    }

[cromero]

Thanks! but it's failing

Code: Select all

Exception calling "Create" with "2" argument(s): "Object reference not set to an instance of an object."
At line:4 char:9
+         Get-VBRTaskSession ([Veeam.Backup.Core.CBackupSession]::Creat ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : NullReferenceExceptio

[tsightler]

My guess is this is due to the fact that the initial script I posted had a small typo. I'm not really sure how that happened as I copied and pasted the script, but regardless, it was there, so I have corrected it in the post above.

If you look in the code you posted, near the very start of the second line, the $bsession variable is misspelled as $bsessison. That's most likely causing the error. Sorry about that.

[cromero]

Thanks!!!! Didn't notice!!! Monday... hehe

Now it works fine!

[tsightler]

OK, let me know what kind of speedup you see. If it's still too slow there may be other things we can do.

[cromero]

OK, let me know what kind of speedup you see. If it's still too slow there may be other things we can do.

Now it's perfect! It was taking 2.30h before. Now 10 min

[veremin]

Update to everyone experiencing this issue. We've implemented a hot fix and started distributing among our clients.

If you'd like to try it out, create a ticket and refer to case number 02046581.

Thanks.

[masonit]

Hi

Will permanent fix be included in 9.5 update 2?

\Masonit

[veremin]

Correct.