Maintain control of your Microsoft Office 365 email data
Post Reply
munklarsen
Service Provider
Posts: 20
Liked: 8 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

API is very slow

Post by munklarsen » May 29, 2019 6:44 am

Hi,

I'm runnning 3.0 and we're facing issues with the API being very slow (to the point of being unusable) when backups are starting. We're running backups for all jobs every 4 hours (which I talked about here veeam-backup-for-microsoft-office-365-f ... 58652.html) and this means that every 4 hours my API calls go from having a response time of seconds to +60-90 seconds.

Current status is roughly 210 organizations with 213 jobs.

This is actively hindering us from deploying a frontend to the products for self-service since it will fail every 4 hours.

Is there any way to make the API behave better?

nielsengelen
Veeam Software
Posts: 2702
Liked: 563 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: API is very slow

Post by nielsengelen » May 29, 2019 7:02 am

Could you give some more information on the configuration of the controller in terms of resources (CPU/memory)? Are you using the controller purely as a management box or are you also using it as a proxy/repo?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

munklarsen
Service Provider
Posts: 20
Liked: 8 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

Re: API is very slow

Post by munklarsen » May 29, 2019 8:08 am

Console:
- OS: 2019 Standard
- 4 cores
- 16GB

Proxies
- OS: 2019 Standard
- 8 cores
- 32 GB

Each proxy has 6 repositories running in "image backup"-mode with a retention of 1 year. Currently we have 2 proxies.

nielsengelen
Veeam Software
Posts: 2702
Liked: 563 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: API is very slow

Post by nielsengelen » May 29, 2019 12:48 pm

The REST service is mostly CPU intensive, can u monitor the resource usage on the console? It might be good to increase it to 6 or 8 cores.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

munklarsen
Service Provider
Posts: 20
Liked: 8 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

Re: API is very slow

Post by munklarsen » May 29, 2019 7:08 pm

Hi Niels,

A month ago we discovered how retention was "working" with the original implementation of repository retention. Because of this, we changed all repositories to running in "keep forever" mode. This however triggered other issues (we ended up with +1000 jetdb databases which caused the system to be unable to boot since you allocated 0,01% of system memory pr. jetdb-database). The "keep forever" policy vastly improved performance for the few hours the system was online. The issue with jetdb databases forced me to redesign the entire setup (we moved away from the 1:1 relationship between backups jobs and repositories).

In the new setup we use "image" style backups instead and then move back to having a 1 year retention. This performed okay for the first few weeks until now but the issue has reappeared. Earlier today, I changed all repositories to "keep forever" policy and this has once again improved performance.

Example. Before when I tried to list the users for a given organisation while backups was running it could take 1-2 hours for that API call to complete for an organization with 3 users. Now, with the retention set to keep forever it takes "only" 30-45 seconds (still pretty slow, but it is however usable).

Throughput on the proxies also seem to have improved vastly. GUI responsiveness is also much better (was unusable before).

My theory back then was (and still is now) that when I choose to have a retetion that is anything else than "keep forever", Veeam creates an index and to track files and creation/deletion dates so that you know when to delete. When "keep forever" is set, this index is then no longer needed and as such, it all works much faster.

Tomorrow I will post some pictures with how the workload changed. Just want a bit more information that what I can currently provide to prove my point. Could you check with developement if my theory has some truth to it?

Polina
Veeam Software
Posts: 940
Liked: 148 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: API is very slow

Post by Polina » May 30, 2019 10:55 am

Hi Michael,

First of all, your assumption on index files is correct.

Next, if I'm not mistaken, I followed your recent support case along with the dev team. Since some issues are still present, it'd be much more efficient to provide all the screenshots and logs via a support case (instead of or in addition to forums). The information will be delivered to devs for further investigation, and we'll figure out why the performance decreased again after the first few weeks. And please share your case ID in this thread so that we could follow the details.

Thanks!

munklarsen
Service Provider
Posts: 20
Liked: 8 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

Re: API is very slow

Post by munklarsen » May 30, 2019 7:32 pm

Hi Polina,

The issues flagged in the support case was "fixed" (in the sense that we had to delete a bunch of databases to bring it online). The performance issues with the API wasn't reported since we'd just started seeing them at the time and since they were fixed by the redesign of the platform. Since then, the amount of organizations has grown by 30% and thus the issues have reappeared (which makes be think that the index of files was smaller when doing image mode backups with 1 year retention - which should make sense).

I could probably venture the guess that the index was stored within the jetdb-databases and the reason for the sluggishness is.

1) slow storage not suited for such workloads (but well suited for storing large amounts of data)
2) many many organizations with few users as compared to few organizations with many users

Lastly, I'm posting this in here to help my fellow service providers avoid the same issues I have seen. Just because Veeam is aware of issues like these doesn't mean that you necessarily share the information.

Also, it's not like this is so hard to recreate. I've done it twice. Load up 180 organizations with avg 2-5 users into the console, store data on 7K drives and choose image type backups with 1 year retentions and do backups every 4 hours and watch it crawl to a halt. My issue is that currently, I've fixed the issue and the only way to recreate it would be to cause issues for my customers.

Mike Resseler
Product Manager
Posts: 5765
Liked: 618 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: API is very slow

Post by Mike Resseler » Jun 03, 2019 5:48 am

Hey Michael,

Thanks for all the additional information. A question to be sure, you are seeing these issues at this moment right? If so, I really would want to work with you for the logs. While our QC probably can recreate the issue, we never have the same type of environment as you have so those additional logs can speed up the check for the root cause.

So let us know (PS: And yes, I agree... Helping each other out is the main reason of this forum!)

munklarsen
Service Provider
Posts: 20
Liked: 8 times
Joined: Nov 15, 2012 11:02 pm
Full Name: Michael Munk Larsen
Contact:

Re: API is very slow

Post by munklarsen » Jun 06, 2019 8:04 am

Hi Mike,

As I wrote, the issues disappeared when I changed retention to "Keep Forever". So I cannot recreate in the same scale as before. We still see it in a much smaller scale every 4 hours when all the jobs start at the same time (please fix veeam-backup-for-microsoft-office-365-f ... 58652.html).

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests