Maintain control of your Microsoft 365 data
Post Reply
mlumsden76
Service Provider
Posts: 28
Liked: 9 times
Joined: Sep 13, 2018 3:00 pm
Full Name: Michael Lumsden
Contact:

VBO365 Upgrade v7 - Large Azure Egress Cost

Post by mlumsden76 »

Veeam Support - Case # 06037449
We have 25 VBO365 v7 object storage repositories in Azure, with a total of about 65tb stored.
The repositories were configured to have their retention policy run nightly at 12am. This has been the setting going back to VBO365 v5.

We upgraded from VBO365 v6 to VBO365 v7 in mid-April. Since the upgrade, we have been getting hit with a 2.4tb daily egress charge. Azure bandwidth monitor shows the event begins at 12am, runs for about 12 hours and transfers data at a rate of about 500mbps - 800mbps to our on prem repository. I can observe the Veeam.Archiver.Proxy.exe process downloading data from Azure and writing to the C:\ProgramData\Veeam\Backup365\Logs folder on the repository.

Prior to v7, there was a nightly egress of about 45gb. What could the retention policy task be doing post upgrade that wasn't happening on v6?

This left us with a surprise $6k Azure bill for Egress traffic over 15 days before we noticed.

We moved the retention policy task to run weekly to keep this from happening each night. We don't mind running it weekly or even monthly, but we do want to understand why it is now downloading 2.4tb of data each time instead of 45gb prior to the upgrade.
pybfr
Veeam Software
Posts: 131
Liked: 13 times
Joined: Sep 26, 2022 9:54 am
Full Name: Pierre-Yves Bandet
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by pybfr »

I was under the assumption that there currently were no egress charges for o365.
Has that changed ? :?:
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by Regnor »

There are no egress fees for M365. mlumsden76 is talking about the Azure costs for the storage.

@mlumsden76: Doesn't sound expected to me. Did you get a response from support?
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by Polina » 1 person likes this post

Hi Michael,

Thanks for this heads-up. With such a difference in behavior between v6 and v7, it looks to me like a bug. I asked the support team to involve RnD in the investigation.
mlumsden76
Service Provider
Posts: 28
Liked: 9 times
Joined: Sep 13, 2018 3:00 pm
Full Name: Michael Lumsden
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by mlumsden76 » 2 people like this post

We've had a good rate of engagement from support, but they don't know what the cause is. I'm hoping a more senior resource can dig into this. Thank you for moving this up the chain @Polina.

I kicked off 3 repository retention policies manually by setting the date/time to immediate so I can observe them in a controlled way. The total size of the backups stored in the 3 repositories is about 2tb. The 3 repositories took about an 1h30m to complete their retention policy task and consumed about 85gb of egress. This is about 100x the egress use and much higher API call use compared to pre upgrade on VBO365 v6. It's like it's working much "harder" to apply the retention policy in v7 than it did in v6.

I compared the User guide documentation retention policy pages between v6 and v7 and I cannot find any documented differences.
I noticed in v7 they've now included the warning on the retention setting UI "applying a retention policy daily increases egress charges when using object storage".

I think this is rather uncommon. I can't find any posts on this forum, the Service Provider forum or 3rd party forums that sound similar.

One thing I've been wondering about daily retention policy vs weekly or monthly retention policy. Would a daily run retention policy use roughly the same egress as a monthly per instance? The monthly would occur less frequently, so overall it would use less egress, but would they use a comparable amount of egress each time the task runs?
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by JaySt »

Oh wow. Watching this one. Wondering if the (alledged)bug is Azure storage bound or more generic for other object stores as well
Veeam Certified Engineer
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by Mike Resseler » 2 people like this post

As you are aware (or at least I hope so :-)) the case has been escalated to RnD. We are looking into it as we speak, but for now it seems we cannot "recreate" the issue internally. Hopefully we will know more after todays tests and investigation
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by JaySt »

@Mike, are you able to share some findings in the mean time?
Veeam Certified Engineer
mlumsden76
Service Provider
Posts: 28
Liked: 9 times
Joined: Sep 13, 2018 3:00 pm
Full Name: Michael Lumsden
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by mlumsden76 » 4 people like this post

Hi. I have an update. Veeam support acknowledged that they found an issue. They created a .zip file for me to download that contains a new Veeam.Archiver.Repository.ObjectStorage.dll file. Stop the service, replace the file start the service update the repository install steps. I have not tried it yet as I have some other work tasks in front of this one, but I should get to it later today or tomorrow. Then I'll run a retention policy task and see if the problem is resolved.
Polina
Veeam Software
Posts: 2939
Liked: 681 times
Joined: Oct 21, 2011 11:22 am
Full Name: Polina Vasileva
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by Polina »

Michael,

Thanks for your update; let us know how it goes.
This fix will also be delivered in the next patch.
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by JaySt »

How did it go after deploying the fix?
Veeam Certified Engineer
jim.lowry
Veeam Software
Posts: 228
Liked: 62 times
Joined: Jul 12, 2018 4:45 pm
Full Name: Jim Lowry
Location: California
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by jim.lowry » 2 people like this post

I've spoken with @mlumsden76 a few times since he rolled it into production (we work together on a regular basis). As far as I am aware, there have been no issues at all post upgrade.
Jim Lowry
Sr. Systems Engineer
VCSP North America West
VMCE, VMCA, VCP-DC
mlumsden76
Service Provider
Posts: 28
Liked: 9 times
Joined: Sep 13, 2018 3:00 pm
Full Name: Michael Lumsden
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by mlumsden76 » 1 person likes this post

That's right. This patch resolved the issue.
svhelden
Novice
Posts: 3
Liked: never
Joined: Jun 05, 2016 5:20 am
Full Name: Stephan van Helden
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by svhelden »

Is this patch somehow publicly available?
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: VBO365 Upgrade v7 - Large Azure Egress Cost

Post by Mildur »

Hi Stephan

Yes, the fix is included in P20230512 (7.0.0.3604).
https://www.veeam.com/kb4425
Resolved Issues
When applying retention on Object Storage Repositories, the Retention Job downloads an excessive amount of metadata.
Best,
Fabian
Product Management Analyst @ Veeam Software
Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests