Issues after update to v8 (indexing, PostgreSQL, performance)

[tvveen]

Hi all.

We're still having issues after the update to v8. We updated to v8 on 31 December 2024, this because of the initial indexing time of PostgreSQL in the not-so-busy time of the year.

Timeframe (short version):
I raised the first case (#07556545) on 2 Januari 2025 based on jobs not running/failing/endless processing, terrible performance of the controller GUI, still repositories which aren't being indexed and many PostgreSQL errors. Because of the support policy of handling one issue per case I raised on 8 Januari 2025 another case (#07562418) based on jobs not running on schedule. The first case is now mainly about the repositories not being indexed.
On 28 Januari 2025 I've used the 'talk-to-a-manager' button for the first time in my Veeam career. I referred to the two cases, but got only response on the first case which was prioritized and escalated to the RnD team. I asked for a response on my other questions and case, but I got no reply.
On 18 Februari 2025 I contacted our accountmanager from Veeam and explained our situation. He and the technical contact promised to check internally about our cases.
After the latest update (13.1.0.331) which we installed on 20 Februari 2025 the remote console we're using on a jumpstation is unworkable, it crashes even more and reconnecting is not working. The only 'workable' console is now directly on the controllerserver itself.

Our setup:
- 1 controller with the configuration database
- 3 proxies working based on these 3 workloads: Mail/Archive, OneDrive, SharePoint/Teams
- About 150 registrated organisations
- About 450 S3 compatible local repositories for each workload (QNAP QuObjects)
- About 450 Wasabi repositories for each workload (immutable enabled)
- About 900 backup jobs (regular + copy)

It has now been more than 6 weeks and we still have some repositories which aren't indexed (RnD team is working on a solution, I think?). Every day we have around 30-40 jobs which aren't started on schedule and the performance of the GUI controller is still horrible.

I think most of the issues are PostgreSQL-related. This is the main thing that has been changed since v7 and the error messages we're getting are things like 'PostgreSQL database operation failed multiple times with transient error'. We know about the IOPS requirement for PostgreSQL and yes we do use SSD's which are more than cabable to handle these IOPS.
We'll look into proxy pools (new in v8), but I'm not really sure the proxies are our problem.

Do more people have similar issues like us or do you have some ideas we can try to trace the issues or maybe solve them?

[MaartenA]

Move PostgreSQL to a dedicated server is what i would do, or create a new environment with a dedicated PostgreSQL and move your buckets / jobs one by one
https://www.veeam.com/kb4676 is for moving PostgreSQL to a dedicated server but creating a new environment is maybe a safer option.

Do not forget to run https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=80 on your dedicated PostgreSQL server.

[Gostev]

OPs issue is due to a very large number of repositories per controller. The current recommendation is 200 repositories per controller, so OP is 4.5x over the recommended maximum.

The dev team has aspirations to get to 1000 repositories per controller, which would cover OP's deployment, however it's not just about fixing a few bugs. The path there is clear, but it will take multiple optimizations, some of which are significant changes requiring thorough QA, thus it may take them some time to get there.

[MaartenA]

@Gostev

When will Veeam make official numbers available so that people don't run into this problem? You want to know this kind of information before you migrate to V8. Surely there would now be enough data available to be able to make some numbers when you have to put PostgreSQL and possibly NATS on separate servers?

We have been in a similar situation and ultimately resolved it by migrating to a separate PostgreSQL and in our case a completely new environment but this is something we wanted to know before because the impact is big when you need to move later on.

[Gostev]

For releases with major architectural changes (like V8) this usually only happens a decent amount of time after GA, once all the teething issues are addressed, R&D teams stops actively optimizing the code, and this "final" state has been proven in the field.

Whenever you're upgrading before a new major release stabilizes for use in large-scale deployments, I recommend that you reach out to your assigned Veeam Solution Architect to get the most current best practices and limitation for the specific build you're installing. As at this stage the situation is quite fluid with significant improvements from build to build, while R&D team is actively working through specific scalability issues reported from the field.

By the way, I've just been notified by our QA leadership that apparently the dev team is moving much faster than expected to the above-mentioned aspirational target. Our performance QA team has just completed a series of tests of the current build on an infrastructure with 900 repositories and 120 proxies, and the results are very optimistic. Perhaps it could be the best path forward for you to get this new code in a form of a hotfix, without waiting for the next public build.

[tvveen]

@MaartenA
What OS did you choose for your PostgreSQL server?

@Gostev
That sounds very promising. We would like to test this new code if this comes available in any form whatsoever.

I agree with Maarten. If we knew these numbers, we would have waited to upgrade to v8. I guess we'll contact a VSA in the future for major updates.
The odd thing is that our setup worked perfectly fine in v7.

[MaartenA]

@tvveen
Windows Server 2022, it is running fine now. (350 org / 370 buckets)

Our VSA told us that we need to put PostgreSQL on a dedicated server at +10,000 seats but nothing about the number of repositories, so maybe that is some new information and that is also the point i am trying to make that Veeam needs to share this information in a better way. We have also switched to one job per customer instead of dividing this into Sharepoint/OneDrive/Exchange based on Gostev's advice in another forum post, which also works fine. However, we have now administratively frozen this POD for 20,000 users and are now continuing with a new one, technically it is possible with more i think, but also a bit of risk spreading.

[tvveen]

By the way, I've just been notified by our QA leadership that apparently the dev team is moving much faster than expected to the above-mentioned aspirational target. Our performance QA team has just completed a series of tests of the current build on an infrastructure with 900 repositories and 120 proxies, and the results are very optimistic. Perhaps it could be the best path forward for you to get this new code in a form of a hotfix, without waiting for the next public build.

Any updates on this @Gostev?

[Gostev]

I believe 8.1.1 shipping this week already includes these scalability improvements.

[pat_ren]

I'm also very interested in this, have been working with support and R&D constantly since upgrading to v8 many months ago, it hasn't been as reliable as version 7 for a single day. We had no idea when we upgraded about any of these limitations, this was all information made available much later. We have 3 servers, one has ~800 repos and 8 proxy servers which has been the biggest pain for me over the last few months. I would like to believe 8.1.1 will improve things, but I'm also running about 10 private fixes so I'm not sure if or when i'll be able to upgrade.