Veeam 12 - History GUI very slow initially to show results

[Lewpy]

Since I upgraded a client of mine to V12 the other week, I have noticed that the History section of the Veeam B&R GUI has become extremely slow to produce results, and I am left watching the little green dots at the top fill up, then empty, then fill up, then empty
Previously, it wasn't anything that I noticed as being particularly slow, so I can't put a time to how long I would have waited for the results to come back. But I just timed how long it took for the System section to return results, and it took 1 minute and 45 seconds. Once results are returned, filtering is near-instant. And I can flick between different sections that have returned results and instantly see the results: so if I have waited for Jobs and System to both return results, I can select either of them and have instant results shown. It is only the first time I click on the individual sections that things are incredibly slow.
The Veeam server is a not-too-ancient Dell R740, with 8 core CPU & 64GB RAM, running Windows Server 2019.
Veeam is running the latest April 2023 patch.
The Veeam database is running in SQL Express 2017 on the same box.
The do.Backup.Model.BackupTaskSessions table has 179,376 records, while the dbo.Backup.Model.BackupTaskSessionProgress table has 2,734,962 entries.
SQL Activity monitor shows a very nested select query sitting there for the whole time I am waiting ("with bk_sess_by_policy as ( select ....").

Is there any tuning I can do, or is this a known change/issue?

Thanks.

[Mildur]

Hello Lewis

Please open a case with our customer support and upload the latest log package.
The debug logs will help us to understand what's happening on your backup server and MSSQL instance.
Analyzing such issues over the forum is not possible.

May I ask is this a All-In-One server which also runs the repository and proxy roles?

Best,
Fabian

[BackupBytesTim]

I'm curious if this also happens anytime you refresh the displayed data, I would expect it is as to my knowledge the filtering and switching between the two lists entirely don't load any data again, just changes how the existing data is displayed on the screen. Also curious how many items/jobs you have your console set to load when you show the history, I have mine set to show 500 and it regularly takes about 45 seconds to load the list, and again anytime I refresh it.

Assuming it also loads slowly like that then it's the same for me, I assumed it was normal. If Fabian there is accurate in suggestion it's abnormal then perhaps I should look into it more.

[Lewpy]

Hello Lewis

Please open a case with our customer support and upload the latest log package.
The debug logs will help us to understand what's happening on your backup server and MSSQL instance.
Analyzing such issues over the forum is not possible.

May I ask is this a All-In-One server which also runs the repository and proxy roles?

Best,
Fabian

Hi Fabian,

It is an "All-In-One" server, although it also uploads to Cloud storage (SOBR) and has Copy Jobs to a remote Veeam Repository server.
If the performance I have seen is not expected, I will get the client to raise a Support Ticket.

Thanks,
Lewis.

[Lewpy]

I'm curious if this also happens anytime you refresh the displayed data, I would expect it is as to my knowledge the filtering and switching between the two lists entirely don't load any data again, just changes how the existing data is displayed on the screen. Also curious how many items/jobs you have your console set to load when you show the history, I have mine set to show 500 and it regularly takes about 45 seconds to load the list, and again anytime I refresh it.

Assuming it also loads slowly like that then it's the same for me, I assumed it was normal. If Fabian there is accurate in suggestion it's abnormal then perhaps I should look into it more.

Hi Tim,

I had changed the History settings down to "Show only last 100", and "Keep only last 53 weeks".
It was set to 1000, so tried to improve matters by dropping it to 100, but didn't really help much.
Pressing F5 causes it to appear to reload the view, but it does so near-instantly, so I don't think it is re-querying the database. I can't find any other way to refresh the History view.

Thanks,
Lewis.

[Lewpy]

I've just gone on to another Client's Veeam server (very similar configuration) that is still running v11 Veeam, and the History view loads within about 1 second. And that is set to a list view of 500 (and 53 week retention).
Looking in SQL Management Studio, the SQL queries for populating the History in v11 and v12 are completely different: v11 uses the ReportFilteredSessonsView (with a SELECT TOP(xxx), where xxx is the size of your "show only last xxx" sessions setting [I changed it to be sure]), while v12 has some massive nested selection (and the analysis in SQL Management Studio says 98% of the query cost is in an XML data reader).

[alexander sh]

Hi Lewis.
This is something we in R&D would be happy to look into. If you could open a support case and share its number here, we'll be able to quickly start working with you on this.

Thanks in advance, Alexander.

[stfconsulting]

Way to be on top of things on the Veeam side. Happy to see this.

[rlspeeks]

I'm interested in outcome, I'm seeing a little slowness as well.

[Mildur]

Hello @stfconsulting, Hello @rlspeeks

If you see the same slowness, please open a support case, provide the logs and share with us the case number.
We can only help if we get your log files.

Best,
Fabian

[oliver@sita]

Hello Lewis,

we had same issue and also open a case at Veeam.

Solution is to upgrade to the actual lastest Version of VBR "Veeam v12 P20230718"

This helps to fix the issue and many many more things

Regards
Oliver

[pirx]

Case 06283879
I see general slowness in VBR console after v12 update, opening 24h view takes minutes but it's everywhere. We are on latest version.

[cptkommin]

Any feedback on this, apart from upgrading to the latest patch?

Currently on P20230412 with hotfix for the block clone issue.

Have a case open: 06282105

[Mildur]

Hello guys

I talked with our teams.
We are planning to release a couple of fixes for the history performance in our next release v12a.

If you see the history tab taking a long time to list the historic job sessions, we may have a private fix ready in a 2-3 weeks. It would be helpful for us, if you can upload a copy of your backup servers configuration database to the case. This will allow us to verify the private hotfix.

Please let me know if you see such issue and if you are ok to provide the database via your support case. I can then inform our QA team about the uploaded database.

Thank you
Fabian

[roland]

I had the same performance problem in the Gui after v12 P20230412 migration.
The trouble is in many views and the console is very unresponsive and with lag on screen refresh.
I patched it to the lastest level. 12.0.0.1420 p20230718 and it's now a little better in some views but not everywhere.
The only way i found to speed it up is to disable the Full or Compact view to Off.

[olibru]

I had the same Issue with the History GUI, very slow load.
I installed a new Windows Server 2022 VM with the same Hostname, imported the Configuration Backup (without Session History) and Repositories.
Now its very responsive.

I have to say that the old Backup Server had a runtime over 8 years. Two product upgrades (V10-V12) and DB migration from MSSQL to Postgresql.
It was time for a fresh installation.