Maintain control of your Microsoft 365 data
Post Reply
emcs
Novice
Posts: 8
Liked: 2 times
Joined: Jul 01, 2022 12:43 pm
Full Name: ed h
Contact:

Teams Export API - Excessive Costs

Post by emcs »

I've got a support case open for this, but no resolution as yet - hoping the community can help out.

We have had to disable Teams Chat backup, due to excessive Graph API costs of £170+ per day.

On an average day, according to Teams User Activity Reports we have around 40-50 channel posts, replies etc - these (not 1-2-1 chats) are what is being backed up via the Teams Export API.

As per (https://community.veeam.com/blogs-and-p ... iewed-2815) the calculation for API cost is simply: #NumberOfMessages * $0.00075

So on an average day, we would expect 50 * 0.00075 = $0.0375

Now, even though the £ is very weak at the moment, there is no way $0.0375 = £170+! :D

For our billing to be correct, we would need to be consuming in excess of 200k messages per day!

Checking the Veeam job stats, the number of items processed is roughly in-line with the number of messages per day from the Teams User Activity Reports - give or take, depending on other items in the scope of the backup such as files etc. - it is certainly not in the 200k region.

When requesting access to the Teams Export API, only the Veeam Azure App was given access, and we do not use any other metered APIs.

We have also raised our concerns with MS via teamsapibilling@microsoft.com, but as yet no response.

Is anyone else seeing something similar? Might be worth checking your Azure bills! :lol:
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Teams Export API - Excessive Costs

Post by Polina »

Hi!

Please provide the ID of your support case with Veeam
emcs
Novice
Posts: 8
Liked: 2 times
Joined: Jul 01, 2022 12:43 pm
Full Name: ed h
Contact:

Re: Teams Export API - Excessive Costs

Post by emcs »

Hi Polina,

Case # 05679075
emcs
Novice
Posts: 8
Liked: 2 times
Joined: Jul 01, 2022 12:43 pm
Full Name: ed h
Contact:

Re: Teams Export API - Excessive Costs

Post by emcs »

Update: After disabling Teams chat backup, the API costs have stopped increasing - so we have confirmed that it is definitely Veeam causing the excessive charges on the protected API.
emcs
Novice
Posts: 8
Liked: 2 times
Joined: Jul 01, 2022 12:43 pm
Full Name: ed h
Contact:

Re: Teams Export API - Excessive Costs

Post by emcs »

Polina wrote: Oct 21, 2022 3:32 pm Hi!

Please provide the ID of your support case with Veeam
Are you able to share any insight into what might be causing this, or whether any customers have reported the same issue?
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Teams Export API - Excessive Costs

Post by Mike Resseler »

Unfortunately there is not much we can do about this. Those API calls come with an additional cost and is charged by MSFT. I am aware that (at least) one of our customers is talking with us about this and that MSFT is internally discussing this. But I am not aware of the status of that conversation I'm afraid
VNR-LF
Influencer
Posts: 24
Liked: 4 times
Joined: Oct 28, 2020 1:14 pm
Full Name: Lars Freiberger
Location: Germany
Contact:

Re: Teams Export API - Excessive Costs

Post by VNR-LF » 2 people like this post

hi all,

just wanted to chime in on this one. we have been using the teams graph api for chat backup since september 20th. i was checking the cost in azure portal on a weekly basis and was wondering there was no cost at all being displayed. this is until on november 10th i received an email from GraphProtectedAPIs@microsoft.com stating that billing has finally been set up, "Billing begins for consumed messages starting November 10th, 2022". i somehow felt lucky that the initial backup of all the old channel messages that accumulated over the past years obviously was not going to get billed and estimated a monthly cost of about $3 (# of channel messages x $0.00075 per message) for the channel messages there are on average every month.

i had another look at the cost just now and can't believe what i saw... accumulated cost from the 10th until today, so for 6 days in total, is $454. this would mean some 600000 messages, if my calculation is correct. but according to the teams usage reports, we've had 214 channel messages during the past 7 days. even if i add all posts, replies, mentions and reactions, this just sums up to 634 items in total. if you look at this the other way round, $454 / 634 = $0,72 per item.

maybe this is due to some error messages i receive during teams backup, specifically for team posts ("Processing team posts <name of the team> failed with error: UnknownError. The remote server returned an error: (410) Gone.“ on 4 teams, "Processing team posts <name of the team> failed with error: An item with the same key has already been added." on another one - see case # 05677067), which might cause some excessive and unexpected api calls. maybe this is something else. i don't know. but something is really off here. my boss just asked me to disable chat backup due to the cost.

kind regards,
lars
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Teams Export API - Excessive Costs

Post by Polina »

Hi Lars,

First, thanks for sharing your support case ID. There're a couple of similar escalations investigated by our RnD team as we speak, and the more data we have, the better.
Next, were there any modifications to the repositories that store your Team chat data after you switched to the new APIs?

Thanks!
emcs
Novice
Posts: 8
Liked: 2 times
Joined: Jul 01, 2022 12:43 pm
Full Name: ed h
Contact:

Re: Teams Export API - Excessive Costs

Post by emcs » 1 person likes this post

Hi lars.

I am somewhat happy that at least it is not just our organisation having the issue! :)

I have a joint issue open with Microsoft and Veeam and both parties are working together to identify the cause of this problem. I will certainly update here once we have identified a solution.
VNR-LF
Influencer
Posts: 24
Liked: 4 times
Joined: Oct 28, 2020 1:14 pm
Full Name: Lars Freiberger
Location: Germany
Contact:

Re: Teams Export API - Excessive Costs

Post by VNR-LF »

Polina wrote: Nov 21, 2022 3:01 pm Hi Lars,

First, thanks for sharing your support case ID. There're a couple of similar escalations investigated by our RnD team as we speak, and the more data we have, the better.
Next, were there any modifications to the repositories that store your Team chat data after you switched to the new APIs?

Thanks!
sorry, for the late reply.

the last modifications to the repositories i made was upgrading them after upgrading vbo from v5 to v6.

i ran another test yesterday involving a new job and a new repository, since i have an issue with channel message backup that is being investigated. steps taken:
- created a new S3 bucket
- edited iam policy to allow the iam user used by VBO to access the newly created S3 bucket
- created new object storage repository using the newly created s3 bucket
- created new backup repository. retention policy: item-level, 90 days. configured to offload data to the new object storage repository
- edited my organization in vbo -> re-enabled backup of teams chats
- created new backup job for one team, including channel message backup, using the new repository
- ran the job and captured the traffic using fiddler
- disabled channel message backup on the job, disabled the job, disabled chat message backup on organization level

thus, i am currently sure the only job accessing the api was the one i ran for testing yesterday. i had another look at the azure portal today to check for costs incurred. it's $7.41 for yesterday, which translates to 9880 messages exported via the api. i asked the veeam support engineer how many message have been backed up, and he replied "I have checked the logs for all Teams jobs, and I got a hit on around 2500 findings regarding messages.".

so something indeed does not add up.

kind regards,
lars
emcs
Novice
Posts: 8
Liked: 2 times
Joined: Jul 01, 2022 12:43 pm
Full Name: ed h
Contact:

Re: Teams Export API - Excessive Costs

Post by emcs » 1 person likes this post

Good morning Lars,

Veeam provided a private fix to us, which upon testing has resolved the issue (API costs are now significantly reduced).

It would appear this fix has been included in the latest build 6.1.0.438. Note the following 'enhancement' in the release notes: https://www.veeam.com/kb4347?ad=in-text-link

Enhancements
Optimized requests for backup operations and management of organizations using the new protected Teams Export APIs.


I find it odd that this has been included as an enhancement, rather than a resolved issue. I would have expected Veeam, as a reputable vendor, to urgently inform all customers about this issue, as it may be causing significant costs that they could potentially be unaware of. Unfortunately this doesn't appear to be the case, and it look like it will be slipped in under the radar as an enhancement, which is very disappointing.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Teams Export API - Excessive Costs

Post by Mike Resseler »

Hi @emcs

First, allow me to apologize that we haven't sent the communication yet.
We haven't sent it out (it is being prepared though) because we wanted to make sure that CP3 (6.1.0.438) didn't cause any other issues. As you might know, just before the New Year the issue with the Soap header arrived (MSFT is rolling that change back but in phases) so we wanted to be double sure that there are no other issues before sending out the communication and getting lots of people into trouble and opening support tickets.

I hope you understand, and always open to discuss in PM and listen to your thoughts on how we can improve such items in the future

Cheers
Mike
sumeet
Service Provider
Posts: 105
Liked: 14 times
Joined: Apr 23, 2021 6:40 am
Full Name: Sumeet P
Contact:

Re: Teams Export API - Excessive Costs

Post by sumeet »

We have run into same issue with our client reporting that for the month of April they have got a bill of $24K (The new Teams export API was enabled on 13th April).
When we checked for the month of May, Teams admin center shows 1024 Channel messages between 30th April to 6th of May but the cost for Teams Export API from 1st of May to 7th of May is $1,940.

Build is 6.1.0.1015 P20230322

Case# 06050418

We have disabled the backup job since 8th of May.

Case is with Veeam R&D and it is still being investigated.
Not happy that we are missing the SLA, but also client unhappy due to the high cost.

-Sumeet.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Teams Export API - Excessive Costs

Post by Polina »

Hi Sumeet,

The RnD team identified the root cause and is preparing a fix for you. Please keep working with our support engineers.

Thanks!
sumeet
Service Provider
Posts: 105
Liked: 14 times
Joined: Apr 23, 2021 6:40 am
Full Name: Sumeet P
Contact:

Re: Teams Export API - Excessive Costs

Post by sumeet » 3 people like this post

Hi Polina,

Thanks for the update, and good to know that the root cause has been identified.
Have not heard back from support yet, will wait.

Not sure what the issue was, but when such APIs which incur additional cost are involved, I feel the coding around this has to be designed such that it stops calling the API when an unexpected scenario occurs (assuming in this case an expected scenario has occured, causing the API to run more than expected). Instead of continuing with excessive calls and us/clients getting bill shocks.

In this case, we are not sure how to handle this cost of $26,000 due to this API

Best approach in my suggestion is for Veeam to provide these options:
  • Provide a option similar to Throttle network traffic for this Teams export API too. For example we can set that max API run in each backup run should not exceed 10000 ($7.5). If the backup job requires reaches this limit, it has to stop, indicating this limit has reached
  • The daily backup report should list the number of times the API has been called in the last backup run
  • This parameter listing the number of times this Teams export API is called should be accessible using Veeam REST API, so that we can setup daily monitoring
This will also give us a handle to match on how many times the API is called and match it with the cost Microsoft is listing.

Regards,
-Sumeet.
kaffeine
Enthusiast
Posts: 32
Liked: 11 times
Joined: Jun 04, 2018 8:03 am
Full Name: Espresso Doppio
Location: Austria
Contact:

Re: Teams Export API - Excessive Costs

Post by kaffeine » 1 person likes this post

I think you've raised some very valid points. In the cloud world where fee-based APIs reign things can get out of hand quite fast, and in extreme cases one's left with pile of burning cash.

Software bugs are to be expected, and subsequent patches shall fix them sooner or later, but in such cases there's no going back, the resultant bill it's the only thing left.

I would be great if Veeam introduced some kind of fail-safe to avoid API calls exploding to infinity unnoticed. A possible idea could be introducing a small DWORD registry flag as a first step (something like API_DailyRateLimit), and maybe later more detailed cost/threshold settings on future major versions.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Teams Export API - Excessive Costs

Post by Gostev » 1 person likes this post

@Mike Resseler suggest adding a warning in backup jobs after hitting the certain API costs threshold (as defined/customized by user) in the trailing 1 week.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Teams Export API - Excessive Costs

Post by Mike Resseler » 1 person likes this post

All,

What @Gostev suggested is something which is being worked on. There are multiple paths being looked at, including one together with MSFT where we would be able to (more or less) predict the cost before the backup happens. But that is still under discussion and requires a change on their backend.
sumeet
Service Provider
Posts: 105
Liked: 14 times
Joined: Apr 23, 2021 6:40 am
Full Name: Sumeet P
Contact:

Re: Teams Export API - Excessive Costs

Post by sumeet »

Hello Mike,

Just to add more updates, two weeks ago, we were provided two private fixes. Part 1 fix was to be applied first and run the backup and then collect logs for Engg to review. Then they asked me to apply part 2 fix. We managed to run both of these, but every run had to be first checked with our client and post the backup run, had to wait couple of days to verify the Azure billing to make sure by how much has the value increased. Only after verifying that the value has changed by few dollars, we were allowed to run the private fix.
Even after verifying both the fixes, the job is still in disabled state as our client is not so keen on having this run.

Anyone would be in this state, including us, after having got a shock of $26,000+ of bill shock.

Upon this, we have many other clients for whom the teams backup is enabled, but all of these clients are fine, as the steps to switch them to use the new Teams export API was done in Jan 2023, after which MSFT has changed the process of enabling the billing. We have time until June 30th to perform step 2 of this KB https://www.veeam.com/kb4322
But we are concerned of doing this as want to avoid the excessive bill shock as was observed for this one client, for whom we enabled the Teams export API in April, by when these new MSFT steps was provided.

I understand your longer term goals of predicting the cost before backup happens and other paths, but we need something in short term to have a handle over this nightmare of excessive cost. As I mentioned in my earlier post, there are few other options that I feel can be done in short term.

Need your help with the following
1. Please check with RnD to provide a patch on v6.1. Support has informed that no patch will be provided and v6 will only have private fix.
2. Provide a short term option to have a control over this API use during backup

Thanks,
-Sumeet.
sumeet
Service Provider
Posts: 105
Liked: 14 times
Joined: Apr 23, 2021 6:40 am
Full Name: Sumeet P
Contact:

Re: Teams Export API - Excessive Costs

Post by sumeet »

Until we get a handle on how to manage Veeam calling this Teams API, I would recommend everyone using the API to setup cost alerting on the Azure resource used for the API.

https://learn.microsoft.com/en-us/azure ... e-spending

We have done this for our own backup.
Though this will not help if the backup run calls the API millions of time on a night, but it will save us from the backup calling this API millions of times everyday, until we get a excessive bill shock at month end.

-Sumeet.
RO_Ubi
Novice
Posts: 4
Liked: never
Joined: Nov 15, 2022 4:41 pm
Contact:

Re: Teams Export API - Excessive Costs

Post by RO_Ubi »

Mike Resseler wrote: May 19, 2023 6:26 am All,

What @Gostev suggested is something which is being worked on. There are multiple paths being looked at, including one together with MSFT where we would be able to (more or less) predict the cost before the backup happens. But that is still under discussion and requires a change on their backend.
Hi @Mike Resseler, is this still being worked on between Veeam and Microsoft?

We would like to enable the Teams Export API however the several reports of excessive costs being incurred understandably make us quite wary about doing so.

Being able to predict the cost prior to enabling the API would allow us to move forwards with this.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Teams Export API - Excessive Costs

Post by Polina »

Hi @RO_Ubi,

All the known issues related to the cost of Teams backup have been resolved in patches. So far, I haven't heard any new concerns on this matter. You can upgrade to the latest build 7.0.0.4388.

Thanks
sumeet
Service Provider
Posts: 105
Liked: 14 times
Joined: Apr 23, 2021 6:40 am
Full Name: Sumeet P
Contact:

Re: Teams Export API - Excessive Costs

Post by sumeet »

Hi Polina,

I will please request to have something in place to be able to control how many times this API gets call.
Like I used Veeam backup for Salesforce the first time last week and it has a way to limit the Salesforce API being called. Same has to be implemented in this case too. Not sure why we would wait for an unexpected scenario to happen which would again cause a very high API usage and cost.

Currently Azure cost alerts have successfully helped to report high API usage if there is high change in the teams usage.

-Sumeet.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: Teams Export API - Excessive Costs

Post by Polina »

Hi Sumeet,

You can check backup job logs to get this information. There should be something like this:

Code: Select all

[%timestamp%]   xx (xx) Graph API request (client-request-id: %ID% timestamp: %timestamp%): https://graph.microsoft.com/v1.0/teams/%id%/channels/getAllMessages?$top=500&model=B&$filter=lastModifiedDateTime gt 0001-01-01T00:00:00.0000000Z and lastModifiedDateTime lt 9999-12-31T23:59:59.9999999Z
[%timestamp%]   xx (xx) Graph API response (client-request-id: %ID%, request-id: %ID%, timestamp: %timestamp%): https://graph.microsoft.com/v1.0/teams/%id%/channels/getAllMessages?$top=500&model=B&$filter=lastModifiedDateTime gt 0001-01-01T00:00:00.0000000Z and lastModifiedDateTime lt 9999-12-31T23:59:59.9999999Z
[%timestamp%]   xx (xx) Issued getAllMessages request (Team ID: %ID%). Received x messages (parents: x, replies: x)
sumeet
Service Provider
Posts: 105
Liked: 14 times
Joined: Apr 23, 2021 6:40 am
Full Name: Sumeet P
Contact:

Re: Teams Export API - Excessive Costs

Post by sumeet »

Hi Polina,

From my discussions with support, they too suggested this same few months back. And from what I remember, this is not a single summarized entry in the log at the end of the job that gives the total API used. I will have to parse all the log files of the whole backup job of the last night, finding these entries and adding them up all.
This is not possible.

Thanks,
-Sumeet.
sumeet
Service Provider
Posts: 105
Liked: 14 times
Joined: Apr 23, 2021 6:40 am
Full Name: Sumeet P
Contact:

Re: Teams Export API - Excessive Costs

Post by sumeet »

Hi,

Any update on this? It is important to have a handle on how many times the API is invoked. Best would be to be able to extrat this info using powershell command so that monitoring can be setup. Monitoring will raise alerts if the usage of the API is above a specific value, and this value will vary from client to client.

Thanks,
-Sumeet.
Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests