V10 - Slower B&R console after startup

[FrancWest]

Hi,

Anyone else noticed that after you started the V10 B&R console it takes some time to load the joblists, running jobs, etc? You see the green dots in the top of the window. Next time you click on the nodes during the same session it's fast again. However, if you close the console and start it again, it's slow again until everything is loaded. I assume this has to do with the new caching of the database views that was introduced in V10?

Franc.

[Dima P.]

Hello Franc,

Indeed new caching logic may affect the UI performance during the first start (since cache is empty and must be populated with the date first). If you notice the same behavior after reopening UI, please raise a support case and share the case ID with us. Thank you!

[Gostev]

Dima, cached content is only "good" for a limited period of time. It is expected that re-opening UI after for example an hour cannot leverage cached data collected some time ago, because it no longer represents the actual state of what's in the configuration database (new VMs appeared, new restore points have been created, etc.)

[FrancWest]

Well, in my case it's quite random. After closing the UI and opening it, some nodes take some time to load again. For example, go to history -> system, I see the green 'loading' dots mostly of the time after restarting the console, even when re-opening after 10 minutes.

Clicking on 'running jobs' also takes longer than before to show up.

Is it possible to disable caching? It seems in our small environment it makes the gui slower.

[Gostev]

This is expected, as the cache is automatically invalidated when database tables behind cached views change. So, displaying the affected view requires issuing the actual SQL query against the database.

Unfortunately, caching cannot be disabled, as it's the part of the architecture now. One of the reason it was put in place is that previously, we had support cases when database load from UI activities was affecting the reliability of backup and restore operations.

We do want to investigate use cases of slower experience in small environments, because one of the design goals was not impact user experience in such environments. We do have some pre-defined sets of timeouts used based on the environment size, which can be tweaked. In fact, this was a plan from the start to adjust those based on the feedback from real-world environments

[FrancWest]

Ok, thanks. I opened a case already, #04023048

[mweissen13]

We are also seeing this behavior. Isn't it odd that caching would make things SLOWER instead of faster. We never had those problems before v10.
Also, I am seeing a constant high load on the SQL Server. Even when no UI is open.

According to SQL Activity monitor it is constantly doing queries like:
SELECT TOP (500) * FROM ReportFilteredSessionsView WHERE (([job_type] = 0 OR [job_type] = 40 OR [job_type] = 52 OR [job_type] = 54 OR [job_type] =
13000 OR [job_type] = 14000 OR ([job_type] = 4000 AND [platform] <> )) AND ( ([usn] > 0)) ORDER BY creation_time desc

Should I open a support case to investigate?

[Gostev]

As I noted above, the primary goal was not to make the UI faster, but to reduce load on the configuration database (especially in large environments with multiple console users) for improved backup and restore reliability. Caching does not always means faster: it only means faster when the required data is in cache, but it can mean slower when the required data is not in cache AND you're trying to optimize/reduce load on the actual data source, like - don't just slam it with huge queries the moment you need some fresh data, but take other activities running in parallel in consideration too.

If you see some particular queries hogging your SQL Server then yes, it would really help us if you open a support case, so that our SQL developer can take a look at it.

[mweissen13]

Understood. However, since I am also seeing increased load on the VeaamBackup configuration database, I will open a support case. I should add that this happens primarily on the Cloud Connect Repository Server.

[Gostev]

Ah, this might be an issue specific to Cloud Connect infrastructure then... it is safe to assume that most SQL profiling and testing was done with regular test labs.

[PeterRolf]

We’ve been noticing an increasing latency issue on our VBR consoles over time. After a few days of investigation, we found that the problem was with the SQL view ‘ReportFilteredSessionsView’. From watching the SQL activity monitor, that view is called many times concurrently when the console is started and each instance would take anywhere between 20 seconds and 5 minutes to execute. Memory wait time drastically increased on the SQL Server during this period.

My understanding based on Gostev’s comments in this thread is that this is part of the UI caching process.

We are a service provider and the latency issue is found on our Cloud Connect servers where the Backup.Model.JobSessions table can be well over a million rows. The "ROW_NUMBER() OVER ( PARTITION..." syntax has to go through all of those million + rows twice per call to the "ReportFilteredSessionsView".

In our case, over 99% of the rows in Backup.Model.JobSessions have job_type = 300 and orig_session_id = null. I assume that 'job_type = 300' is related to the Cloud Connect in some way and this was confirmed as all of the Backup.Model.JobSessions (job_type=300) records map to rows in Backup.Model.CloudSessions.

Since orig_session_id = null, the The "ROW_NUMBER() OVER ( PARTITION..." syntax serves no purpose in my opinion.

A temporary fix I used to improve the performance of this view was to add the job_type != 300 to the existing filter in that view. In order to ensure that the view returned the same number of records as the original, I also added a UNION query where job_type = 300 and set a value of 0 to [retry_count] and [recheck_retry_count] thereby removing the ROW_NUMBER calculation for Cloud Connect job sessions.
I realize that using the 'Union' operator may not be the most elegant solution... but it was the fastest way for us to deploy a fix. I will leave it to the Veeam development team to determine a more elegant solution at a future date.

I have given to the full details of my investigation and updated view definition to the help desk. Case # 04084153.

[mweissen13]

Thank you very much @PeterRolf for that in-depth analysis. This sounds 100% like the problem we are also facing. Currently we are working around it by not opening the Veeam Console on the Cloud Connect Host. Otherwise it also affects running backups too much due to heavy SQL Server load. In the past few weeks we had other priorities and that's why I didn't open a support case for the problem yet.

[alexander sh]

Hello PeterRolf!
Thanks a lot for the analysis. We're looking into the problem you described, there's certainly some room for improvement, especially when it comes to large service provider deployments.
I'll post to this thread when there's some progress.
Cheers, Alexander, Veeam R&D.

[FrancWest]

It's not only service providers that have the issue, we have a small environment and also experience the issue, that's why I opened this thread.

[alexander sh]

Hey guys, we did a few improvements to the problematic report (one of the fixes is based on UNION idea from Peter Rolf, thanks again!). The improvements are pretty drastic in our test lab. I asked responsible QA team to test the fixes, the idea being to create a private fix out of it, so that you don't have to wait until 10a is out.

@mweissen13 The query you posted also is many times faster now. I'll post here again when the private fix is ready to be distributed via our support.

[FrancWest]

Sounds good! Thanks!

[FrancWest]

I just installed P1, but unfortunately I don’t see any difference. The GUI is still slower than v9.5 when showing the information on the various nodes (backup jobs, copy job, running, etc). Still the green dots for several seconds.

[Gostev]

P1 most certainly does not have any fixes around UI whatsover. The list of resolved issues in KB3127 is complete.

[FrancWest]

So P1 is not the same as 10a? Support advised me to install this update for this issue.

[Gostev]

Yes, nothing in common. We don't even have a timeline for 10a at the moment, as it depends on a few unknowns such as our vSphere 7.0 GA testing results, and a few other. It's not impossible that we will have P2 before 10a, to fix another portion of common bugs which were late for P1.

P1 bug candidates were picked about 3 weeks ago, the patch itself was created 2 weeks ago, after that it was regression testing and bug fix. So, the above stuff did not have even a slightest chance to be included... which is exactly why Alexander mentioned a private fix right away.

[FrancWest]

I understand, but apparently support isn’t aware of this. See case notes #04023048.

[mweissen13]

So, should we install P1 or not? Will the private fix, when available, also work for P1?

[Gostev]

Hard to say before the fix is available, but I think it won't matter.

[alexander sh]

Hey guys,

I just received confirmation from the QA team that the hotfix has been approved for distribution. You can request it from your support engineer, refer to bug id 221060. The hotfix can be installed whether or not you have Cumulative Patch 1 deployed.

Thanks all for your feedback and help with this issue!

[FrancWest]

Just installed the patch and it looks like the responsiveness of the gui is better now. Thanks!

[wm-it-team]

How do I get access to this patch? The console is unbearable upon first opening it.

[FrancWest]

See second post above yours..

[cfizz34]

Thanks for posting this as I thought it was just me. Can't wait until this is resolved in a CU.

[FrancWest]

Is there any further improvement to be expected in this area? Although it’s better after the hotfix, it’s still not on par with v9.5. I still see the green dots every now and then. Especially on the running tasks screen. On small occasions it even shows the wrong (cached) information and I have to close and restart the console in order to get the correct information. Since this is not reproducible it’s hard to narrow it down though.

[Gostev]

I don't believe we have any outstanding bugs other than those CP2 addresses. So we want to review any occurrences when wrong information shown etc.

Green dots by itself is not a problem, they show there's some data querying activity, when in previous versions UI would just "hang".

The UI reaction in v10 can be slightly slower than in previous versions, because our goal was to relief configuration database from high load from UI queries, which would sometimes cause backup and restore reliability issues with previous versions due to database being too busy to serve critical requests of running jobs. Support can tweak some registry parameters if you want more responsive UI at the cost of higher configuration database load. I know that out of the box, we have three presets of timeouts based on the environment size, which is determined based on the number of records in a certain database tables.