Teams Export API - Excessive Costs

[m.novelli]

As a Microsoft Partner I let our Customers to see the Cost + Billing blade, is your right

You can migrate the subscription with no downtime to my Company , probably you will even save money, we just invoice what we see in the Cost + Billing blade with no additional fee

Marco

[sumeet]

Hi All,

As informed earlier, we ask all our clients to configure Azure Budget alerts on the resource group.
One of our clients, where in we setup teams backup in Jan, had a similar alert when the first full backup for teams ran. Which was expected as first full backup will use the Teams export API to download all teams channel posts.

But then no alerts in Feb. And now in March first week, the alert is raised for usage of $85 (which when converted to USD and then divided by 0.00075 is a very high value). We checked the teams admin report and the teams channel posts usage for the last one month is 817. Do not see why the daily incremental backup would use so many API calls. Unless any of the latest 8.1 patches causes a full rescan or re-sync.

Have this case opened - 07630174 and getting some not so useful replies from support. Have requested to get this escalated.

As requested many times above, we need a way to know how many times VBM365 calls the API - this is important to have a control on this.

-Sumeet.

[Polina]

Hi Sumeet,

There were no changes in 8.1 patches, but a full sync of messages from private and shared channels should be expected after upgrading to 8.1:

To ensure backup consistency, a full backup of messages from private and shared Teams channels will be performed after you upgrade Veeam Backup for Microsoft 365 from v7.0 or v7a to v8.1. This may result in a one-time increase in costs for using the Teams Exports APIs. The full backup is necessary because your backup repositories may already contain some of the messages, but the date when Microsoft APIs began returning messages from private and shared channels is unknown and may vary across different channels, teams, and organizations.

Atm, I can't provide you with any ETA for the ability to control the paid requests count, but I can assure you that it is on our roadmap.

Thanks!

[sumeet]

Hi Polina,

Good to know that this is on your roadmap.

This is not a upgrade from v7x to v8.1. This is freshly v8.0 setup, with an upgrade from v8.0 to v8.1.
Based on your above message, upgrade from v8.0 to v8.1 should not require a full backup of messages.

So must be something else. We have had to disable the backup job, until more details are available.
The latest update is that the case is now escalated. Waiting for more details.

-Sumeet.

[Polina]

Let's see what support finds. I'll keep an eye on your case.

[sumeet]

Thanks Polina.

The case is slow - I have provided lot of information, but minimal response from support - not sure if all the info I have provided is being absorbed or considered.

Will be helpful if this can be escalated.

-Sumeet.

[sumeet]

Hi,

I have got an update in the case that this case is now referred to R&D team. Will wait for more updates.

-Sumeet.

[sumeet]

Hi Team,

Now more than a month and the update on the case is that it is with R&D team.
We and our clients are paying extra cost for the high usage of the API. Will appreciate if this can be looked into case 07630174

Thanks,
-Sumeet.

[Polina]

Hi Sumeet,

AFAIK, after inspecting your logs our engineers have not detected any abnormal or excessive requests that potentially could have led to bigger bills from Microsoft. They suggested to perform one more isolation test and are expecting an updates from your side (or maybe already received them - I might have missed the latest updates).

[sumeet]

Hi Polina,

I was asked to run a Fiddler tests in the last week of March which was done and logs provided. Since then the only update I have is that the issue is with R&D team.
No further update.

What happened today is that we have got Azure budget alert for our own Teams backup. When I check the cost in Azure, the cost has increased slightly since v8 upgrade, but drastically increased since we started getting an error in teams backup with a teams starting 23rd March and has kept increasing since that day onwards. The behavior is different than what we are observing at our client, where-in the cost for each day is fixed, our is increasing each day and has got to a stage of 15x since what it was with v7.

Regards,
-Sumeet.

[Polina]

Hi Sumeet,

1) Have you already shared this feeback with the support team?
2) What error specifically do you mean?

[sumeet]

Hi Polina,

Yes this feedback has been shared with support team. I have given them daily cost graph for the resource group of Teams export API and it shows gradual increase since v8 upgrade.

Day before yesterday, support team provided few config.xml and proxy.xml changes to apply, informing this should fix the problem.
We applied the changes to our backup and also to our clients. We do not see any change in the cost for our own backup, haven't checked with our client yet as it is holiday here, will check on Monday. But looking at our own cost usage, I do not expect this to make any difference. In two more weeks, it will be 60days since we have opened the case and still waiting to be resolved. With extra cost being paid each day this is taking to solve.

Error is -- Failed to process team: <team-name>. One or more dependencies failed
The response status code does not indicate success: 400 (Bad Request). {"error":{"code":"BadRequest","message":"Provided hosted content identifier is invalid.","innerError":{"date":"2025-04-24T11:38:44","request-id":"xxxx-xxx-xxx-xxx-dbfcf050ec30","client-request-id":"xxxx-xx-xxx-xxxxx-bfcf050ec30"}}}


-Sumeet.

[tdthomas]

I am very keen to know how this topic evolves. We too have the problem of excessive API costs that are neither remotely explanable with the amount of objects that Veeam BfM365 claims in the GUI reports and in the log files nor with the messages in the team chats.
Fortunately the multiple costs are all in all still small in our case, but this problem at best very annoying and ultimately inaceptable.

interestingly we also have such an error message "Provided hosted content identifier is invalid." when processing one specific team chat, but we could trace that problem to a certain message in a certain channel in that team.
With this message deleted the backup runs.
And then it breaks again in the same way.
And then it breaks some more.
And then it breaks completely and Veeam BfM365 claims to be unable to resolve anything.

Or

Code: Select all

Proxy server was offline at the time the job was scheduled to run.

which is hilarious because of course the "proxy server" is the same VM that Veeam BfM365 was running on at that moment ...

[mjr.epicfail]

Keep in mind that the proxy is a service that runs on the OS, maybe it crashed and is not running or restarted automatically but during the job was not started. Windows event Application log should
Mention this.

For the other part , create a veeam case and let support look at it. Post the case number here also