API is very slow

[munklarsen]

Hi,

I'm runnning 3.0 and we're facing issues with the API being very slow (to the point of being unusable) when backups are starting. We're running backups for all jobs every 4 hours (which I talked about here veeam-backup-for-microsoft-office-365-f ... 58652.html) and this means that every 4 hours my API calls go from having a response time of seconds to +60-90 seconds.

Current status is roughly 210 organizations with 213 jobs.

This is actively hindering us from deploying a frontend to the products for self-service since it will fail every 4 hours.

Is there any way to make the API behave better?

[nielsengelen]

Could you give some more information on the configuration of the controller in terms of resources (CPU/memory)? Are you using the controller purely as a management box or are you also using it as a proxy/repo?

[munklarsen]

Console:
- OS: 2019 Standard
- 4 cores
- 16GB

Proxies
- OS: 2019 Standard
- 8 cores
- 32 GB

Each proxy has 6 repositories running in "image backup"-mode with a retention of 1 year. Currently we have 2 proxies.

[nielsengelen]

The REST service is mostly CPU intensive, can u monitor the resource usage on the console? It might be good to increase it to 6 or 8 cores.

[munklarsen]

Hi Niels,

A month ago we discovered how retention was "working" with the original implementation of repository retention. Because of this, we changed all repositories to running in "keep forever" mode. This however triggered other issues (we ended up with +1000 jetdb databases which caused the system to be unable to boot since you allocated 0,01% of system memory pr. jetdb-database). The "keep forever" policy vastly improved performance for the few hours the system was online. The issue with jetdb databases forced me to redesign the entire setup (we moved away from the 1:1 relationship between backups jobs and repositories).

In the new setup we use "image" style backups instead and then move back to having a 1 year retention. This performed okay for the first few weeks until now but the issue has reappeared. Earlier today, I changed all repositories to "keep forever" policy and this has once again improved performance.

Example. Before when I tried to list the users for a given organisation while backups was running it could take 1-2 hours for that API call to complete for an organization with 3 users. Now, with the retention set to keep forever it takes "only" 30-45 seconds (still pretty slow, but it is however usable).

Throughput on the proxies also seem to have improved vastly. GUI responsiveness is also much better (was unusable before).

My theory back then was (and still is now) that when I choose to have a retetion that is anything else than "keep forever", Veeam creates an index and to track files and creation/deletion dates so that you know when to delete. When "keep forever" is set, this index is then no longer needed and as such, it all works much faster.

Tomorrow I will post some pictures with how the workload changed. Just want a bit more information that what I can currently provide to prove my point. Could you check with developement if my theory has some truth to it?

[Polina]

Hi Michael,

First of all, your assumption on index files is correct.

Next, if I'm not mistaken, I followed your recent support case along with the dev team. Since some issues are still present, it'd be much more efficient to provide all the screenshots and logs via a support case (instead of or in addition to forums). The information will be delivered to devs for further investigation, and we'll figure out why the performance decreased again after the first few weeks. And please share your case ID in this thread so that we could follow the details.

Thanks!

[munklarsen]

Hi Polina,

The issues flagged in the support case was "fixed" (in the sense that we had to delete a bunch of databases to bring it online). The performance issues with the API wasn't reported since we'd just started seeing them at the time and since they were fixed by the redesign of the platform. Since then, the amount of organizations has grown by 30% and thus the issues have reappeared (which makes be think that the index of files was smaller when doing image mode backups with 1 year retention - which should make sense).

I could probably venture the guess that the index was stored within the jetdb-databases and the reason for the sluggishness is.

1) slow storage not suited for such workloads (but well suited for storing large amounts of data)
2) many many organizations with few users as compared to few organizations with many users

Lastly, I'm posting this in here to help my fellow service providers avoid the same issues I have seen. Just because Veeam is aware of issues like these doesn't mean that you necessarily share the information.

Also, it's not like this is so hard to recreate. I've done it twice. Load up 180 organizations with avg 2-5 users into the console, store data on 7K drives and choose image type backups with 1 year retentions and do backups every 4 hours and watch it crawl to a halt. My issue is that currently, I've fixed the issue and the only way to recreate it would be to cause issues for my customers.

[Mike Resseler]

Hey Michael,

Thanks for all the additional information. A question to be sure, you are seeing these issues at this moment right? If so, I really would want to work with you for the logs. While our QC probably can recreate the issue, we never have the same type of environment as you have so those additional logs can speed up the check for the root cause.

So let us know (PS: And yes, I agree... Helping each other out is the main reason of this forum!)

[munklarsen]

Hi Mike,

As I wrote, the issues disappeared when I changed retention to "Keep Forever". So I cannot recreate in the same scale as before. We still see it in a much smaller scale every 4 hours when all the jobs start at the same time (please fix veeam-backup-for-microsoft-office-365-f ... 58652.html).