Teams Export API - Excessive Costs

[emcs]

I've got a support case open for this, but no resolution as yet - hoping the community can help out.

We have had to disable Teams Chat backup, due to excessive Graph API costs of £170+ per day.

On an average day, according to Teams User Activity Reports we have around 40-50 channel posts, replies etc - these (not 1-2-1 chats) are what is being backed up via the Teams Export API.

As per (https://community.veeam.com/blogs-and-p ... iewed-2815) the calculation for API cost is simply: #NumberOfMessages * $0.00075

So on an average day, we would expect 50 * 0.00075 = $0.0375

Now, even though the £ is very weak at the moment, there is no way $0.0375 = £170+!

For our billing to be correct, we would need to be consuming in excess of 200k messages per day!

Checking the Veeam job stats, the number of items processed is roughly in-line with the number of messages per day from the Teams User Activity Reports - give or take, depending on other items in the scope of the backup such as files etc. - it is certainly not in the 200k region.

When requesting access to the Teams Export API, only the Veeam Azure App was given access, and we do not use any other metered APIs.

We have also raised our concerns with MS via teamsapibilling@microsoft.com, but as yet no response.

Is anyone else seeing something similar? Might be worth checking your Azure bills!

[Polina]

Hi!

Please provide the ID of your support case with Veeam

[emcs]

Hi Polina,

Case # 05679075

[emcs]

Update: After disabling Teams chat backup, the API costs have stopped increasing - so we have confirmed that it is definitely Veeam causing the excessive charges on the protected API.

[emcs]

Hi!

Please provide the ID of your support case with Veeam

Are you able to share any insight into what might be causing this, or whether any customers have reported the same issue?

[Mike Resseler]

Unfortunately there is not much we can do about this. Those API calls come with an additional cost and is charged by MSFT. I am aware that (at least) one of our customers is talking with us about this and that MSFT is internally discussing this. But I am not aware of the status of that conversation I'm afraid

[VNR-LF]

hi all,

just wanted to chime in on this one. we have been using the teams graph api for chat backup since september 20th. i was checking the cost in azure portal on a weekly basis and was wondering there was no cost at all being displayed. this is until on november 10th i received an email from GraphProtectedAPIs@microsoft.com stating that billing has finally been set up, "Billing begins for consumed messages starting November 10th, 2022". i somehow felt lucky that the initial backup of all the old channel messages that accumulated over the past years obviously was not going to get billed and estimated a monthly cost of about $3 (# of channel messages x $0.00075 per message) for the channel messages there are on average every month.

i had another look at the cost just now and can't believe what i saw... accumulated cost from the 10th until today, so for 6 days in total, is $454. this would mean some 600000 messages, if my calculation is correct. but according to the teams usage reports, we've had 214 channel messages during the past 7 days. even if i add all posts, replies, mentions and reactions, this just sums up to 634 items in total. if you look at this the other way round, $454 / 634 = $0,72 per item.

maybe this is due to some error messages i receive during teams backup, specifically for team posts ("Processing team posts <name of the team> failed with error: UnknownError. The remote server returned an error: (410) Gone.“ on 4 teams, "Processing team posts <name of the team> failed with error: An item with the same key has already been added." on another one - see case # 05677067), which might cause some excessive and unexpected api calls. maybe this is something else. i don't know. but something is really off here. my boss just asked me to disable chat backup due to the cost.

kind regards,
lars

[Polina]

Hi Lars,

First, thanks for sharing your support case ID. There're a couple of similar escalations investigated by our RnD team as we speak, and the more data we have, the better.
Next, were there any modifications to the repositories that store your Team chat data after you switched to the new APIs?

Thanks!

[emcs]

Hi lars.

I am somewhat happy that at least it is not just our organisation having the issue!

I have a joint issue open with Microsoft and Veeam and both parties are working together to identify the cause of this problem. I will certainly update here once we have identified a solution.

[VNR-LF]

Hi Lars,

First, thanks for sharing your support case ID. There're a couple of similar escalations investigated by our RnD team as we speak, and the more data we have, the better.
Next, were there any modifications to the repositories that store your Team chat data after you switched to the new APIs?

Thanks!

sorry, for the late reply.

the last modifications to the repositories i made was upgrading them after upgrading vbo from v5 to v6.

i ran another test yesterday involving a new job and a new repository, since i have an issue with channel message backup that is being investigated. steps taken:
- created a new S3 bucket
- edited iam policy to allow the iam user used by VBO to access the newly created S3 bucket
- created new object storage repository using the newly created s3 bucket
- created new backup repository. retention policy: item-level, 90 days. configured to offload data to the new object storage repository
- edited my organization in vbo -> re-enabled backup of teams chats
- created new backup job for one team, including channel message backup, using the new repository
- ran the job and captured the traffic using fiddler
- disabled channel message backup on the job, disabled the job, disabled chat message backup on organization level

thus, i am currently sure the only job accessing the api was the one i ran for testing yesterday. i had another look at the azure portal today to check for costs incurred. it's $7.41 for yesterday, which translates to 9880 messages exported via the api. i asked the veeam support engineer how many message have been backed up, and he replied "I have checked the logs for all Teams jobs, and I got a hit on around 2500 findings regarding messages.".

so something indeed does not add up.

kind regards,
lars

[emcs]

Good morning Lars,

Veeam provided a private fix to us, which upon testing has resolved the issue (API costs are now significantly reduced).

It would appear this fix has been included in the latest build 6.1.0.438. Note the following 'enhancement' in the release notes: https://www.veeam.com/kb4347?ad=in-text-link

Enhancements
Optimized requests for backup operations and management of organizations using the new protected Teams Export APIs.

I find it odd that this has been included as an enhancement, rather than a resolved issue. I would have expected Veeam, as a reputable vendor, to urgently inform all customers about this issue, as it may be causing significant costs that they could potentially be unaware of. Unfortunately this doesn't appear to be the case, and it look like it will be slipped in under the radar as an enhancement, which is very disappointing.

[Mike Resseler]

Hi @emcs

First, allow me to apologize that we haven't sent the communication yet.
We haven't sent it out (it is being prepared though) because we wanted to make sure that CP3 (6.1.0.438) didn't cause any other issues. As you might know, just before the New Year the issue with the Soap header arrived (MSFT is rolling that change back but in phases) so we wanted to be double sure that there are no other issues before sending out the communication and getting lots of people into trouble and opening support tickets.

I hope you understand, and always open to discuss in PM and listen to your thoughts on how we can improve such items in the future

Cheers
Mike

[sumeet]

We have run into same issue with our client reporting that for the month of April they have got a bill of $24K (The new Teams export API was enabled on 13th April).
When we checked for the month of May, Teams admin center shows 1024 Channel messages between 30th April to 6th of May but the cost for Teams Export API from 1st of May to 7th of May is $1,940.

Build is 6.1.0.1015 P20230322

Case# 06050418

We have disabled the backup job since 8th of May.

Case is with Veeam R&D and it is still being investigated.
Not happy that we are missing the SLA, but also client unhappy due to the high cost.

-Sumeet.

[Polina]

Hi Sumeet,

The RnD team identified the root cause and is preparing a fix for you. Please keep working with our support engineers.

Thanks!

[sumeet]

Hi Polina,

Thanks for the update, and good to know that the root cause has been identified.
Have not heard back from support yet, will wait.

Not sure what the issue was, but when such APIs which incur additional cost are involved, I feel the coding around this has to be designed such that it stops calling the API when an unexpected scenario occurs (assuming in this case an expected scenario has occured, causing the API to run more than expected). Instead of continuing with excessive calls and us/clients getting bill shocks.

In this case, we are not sure how to handle this cost of $26,000 due to this API

Best approach in my suggestion is for Veeam to provide these options:

• Provide a option similar to Throttle network traffic for this Teams export API too. For example we can set that max API run in each backup run should not exceed 10000 ($7.5). If the backup job requires reaches this limit, it has to stop, indicating this limit has reached

• The daily backup report should list the number of times the API has been called in the last backup run

• This parameter listing the number of times this Teams export API is called should be accessible using Veeam REST API, so that we can setup daily monitoring

This will also give us a handle to match on how many times the API is called and match it with the cost Microsoft is listing.

Regards,
-Sumeet.

[kaffeine]

I think you've raised some very valid points. In the cloud world where fee-based APIs reign things can get out of hand quite fast, and in extreme cases one's left with pile of burning cash.

Software bugs are to be expected, and subsequent patches shall fix them sooner or later, but in such cases there's no going back, the resultant bill it's the only thing left.

I would be great if Veeam introduced some kind of fail-safe to avoid API calls exploding to infinity unnoticed. A possible idea could be introducing a small DWORD registry flag as a first step (something like API_DailyRateLimit), and maybe later more detailed cost/threshold settings on future major versions.

[Gostev]

@Mike Resseler suggest adding a warning in backup jobs after hitting the certain API costs threshold (as defined/customized by user) in the trailing 1 week.

[Mike Resseler]

All,

What @Gostev suggested is something which is being worked on. There are multiple paths being looked at, including one together with MSFT where we would be able to (more or less) predict the cost before the backup happens. But that is still under discussion and requires a change on their backend.

[sumeet]

Hello Mike,

Just to add more updates, two weeks ago, we were provided two private fixes. Part 1 fix was to be applied first and run the backup and then collect logs for Engg to review. Then they asked me to apply part 2 fix. We managed to run both of these, but every run had to be first checked with our client and post the backup run, had to wait couple of days to verify the Azure billing to make sure by how much has the value increased. Only after verifying that the value has changed by few dollars, we were allowed to run the private fix.
Even after verifying both the fixes, the job is still in disabled state as our client is not so keen on having this run.

Anyone would be in this state, including us, after having got a shock of $26,000+ of bill shock.

Upon this, we have many other clients for whom the teams backup is enabled, but all of these clients are fine, as the steps to switch them to use the new Teams export API was done in Jan 2023, after which MSFT has changed the process of enabling the billing. We have time until June 30th to perform step 2 of this KB https://www.veeam.com/kb4322
But we are concerned of doing this as want to avoid the excessive bill shock as was observed for this one client, for whom we enabled the Teams export API in April, by when these new MSFT steps was provided.

I understand your longer term goals of predicting the cost before backup happens and other paths, but we need something in short term to have a handle over this nightmare of excessive cost. As I mentioned in my earlier post, there are few other options that I feel can be done in short term.

Need your help with the following
1. Please check with RnD to provide a patch on v6.1. Support has informed that no patch will be provided and v6 will only have private fix.
2. Provide a short term option to have a control over this API use during backup

Thanks,
-Sumeet.

[sumeet]

Until we get a handle on how to manage Veeam calling this Teams API, I would recommend everyone using the API to setup cost alerting on the Azure resource used for the API.

https://learn.microsoft.com/en-us/azure ... e-spending

We have done this for our own backup.
Though this will not help if the backup run calls the API millions of time on a night, but it will save us from the backup calling this API millions of times everyday, until we get a excessive bill shock at month end.

-Sumeet.

[RO_Ubi]

All,

What @Gostev suggested is something which is being worked on. There are multiple paths being looked at, including one together with MSFT where we would be able to (more or less) predict the cost before the backup happens. But that is still under discussion and requires a change on their backend.

Hi @Mike Resseler, is this still being worked on between Veeam and Microsoft?

We would like to enable the Teams Export API however the several reports of excessive costs being incurred understandably make us quite wary about doing so.

Being able to predict the cost prior to enabling the API would allow us to move forwards with this.

[Polina]

Hi @RO_Ubi,

All the known issues related to the cost of Teams backup have been resolved in patches. So far, I haven't heard any new concerns on this matter. You can upgrade to the latest build 7.0.0.4388.

Thanks

[sumeet]

Hi Polina,

I will please request to have something in place to be able to control how many times this API gets call.
Like I used Veeam backup for Salesforce the first time last week and it has a way to limit the Salesforce API being called. Same has to be implemented in this case too. Not sure why we would wait for an unexpected scenario to happen which would again cause a very high API usage and cost.

Currently Azure cost alerts have successfully helped to report high API usage if there is high change in the teams usage.

-Sumeet.

[Polina]

Hi Sumeet,

You can check backup job logs to get this information. There should be something like this:

Code: Select all

[%timestamp%]   xx (xx) Graph API request (client-request-id: %ID% timestamp: %timestamp%): https://graph.microsoft.com/v1.0/teams/%id%/channels/getAllMessages?$top=500&model=B&$filter=lastModifiedDateTime gt 0001-01-01T00:00:00.0000000Z and lastModifiedDateTime lt 9999-12-31T23:59:59.9999999Z
[%timestamp%]   xx (xx) Graph API response (client-request-id: %ID%, request-id: %ID%, timestamp: %timestamp%): https://graph.microsoft.com/v1.0/teams/%id%/channels/getAllMessages?$top=500&model=B&$filter=lastModifiedDateTime gt 0001-01-01T00:00:00.0000000Z and lastModifiedDateTime lt 9999-12-31T23:59:59.9999999Z
[%timestamp%]   xx (xx) Issued getAllMessages request (Team ID: %ID%). Received x messages (parents: x, replies: x)

[sumeet]

Hi Polina,

From my discussions with support, they too suggested this same few months back. And from what I remember, this is not a single summarized entry in the log at the end of the job that gives the total API used. I will have to parse all the log files of the whole backup job of the last night, finding these entries and adding them up all.
This is not possible.

Thanks,
-Sumeet.

[sumeet]

Hi,

Any update on this? It is important to have a handle on how many times the API is invoked. Best would be to be able to extrat this info using powershell command so that monitoring can be setup. Monitoring will raise alerts if the usage of the API is above a specific value, and this value will vary from client to client.

Thanks,
-Sumeet.

[sumeet]

Hi,

As mentioned earlier, any API usage that has cost associated with it, should have to throttle or monitor it.
Will appreciate if I can get an update.

Thanks,
-Sumeet.

[ansie]

Hi,
i also waiting for an solution.
Without cost limits or estimates, this backup solution is not complete and unusable.

[crnix]

Hi @RO_Ubi,

All the known issues related to the cost of Teams backup have been resolved in patches. So far, I haven't heard any new concerns on this matter. You can upgrade to the latest build 7.0.0.4388.

Thanks

Hi all,

Just looking for some more information on the above, how is this being calculated and is there any news on rate limits, or what is the best way to estimate usage? This is a feature I'd like to use but based on what I've read on others receiving massive bills, I need to know a bit more before deciding.

[StoopidMonkey]

Is it possible to view these costs within the Subscription blade in Azure? It's odd that a Graph Services account object is created with a Resource Group specified, but when I browse the RG no objects appear. Due to the fact that we have our Subscription set up via a CSP agreement most of the Cost + Billing blade is inaccessible to us, which is the only other area that was mentioned to find these costs outside of looking at the monthly invoice.