-
- Service Provider
- Posts: 37
- Liked: 10 times
- Joined: Sep 13, 2018 3:00 pm
- Full Name: Michael Lumsden
- Contact:
VBO365 Upgrade v7 - Large Azure Egress Cost
Veeam Support - Case # 06037449
We have 25 VBO365 v7 object storage repositories in Azure, with a total of about 65tb stored.
The repositories were configured to have their retention policy run nightly at 12am. This has been the setting going back to VBO365 v5.
We upgraded from VBO365 v6 to VBO365 v7 in mid-April. Since the upgrade, we have been getting hit with a 2.4tb daily egress charge. Azure bandwidth monitor shows the event begins at 12am, runs for about 12 hours and transfers data at a rate of about 500mbps - 800mbps to our on prem repository. I can observe the Veeam.Archiver.Proxy.exe process downloading data from Azure and writing to the C:\ProgramData\Veeam\Backup365\Logs folder on the repository.
Prior to v7, there was a nightly egress of about 45gb. What could the retention policy task be doing post upgrade that wasn't happening on v6?
This left us with a surprise $6k Azure bill for Egress traffic over 15 days before we noticed.
We moved the retention policy task to run weekly to keep this from happening each night. We don't mind running it weekly or even monthly, but we do want to understand why it is now downloading 2.4tb of data each time instead of 45gb prior to the upgrade.
We have 25 VBO365 v7 object storage repositories in Azure, with a total of about 65tb stored.
The repositories were configured to have their retention policy run nightly at 12am. This has been the setting going back to VBO365 v5.
We upgraded from VBO365 v6 to VBO365 v7 in mid-April. Since the upgrade, we have been getting hit with a 2.4tb daily egress charge. Azure bandwidth monitor shows the event begins at 12am, runs for about 12 hours and transfers data at a rate of about 500mbps - 800mbps to our on prem repository. I can observe the Veeam.Archiver.Proxy.exe process downloading data from Azure and writing to the C:\ProgramData\Veeam\Backup365\Logs folder on the repository.
Prior to v7, there was a nightly egress of about 45gb. What could the retention policy task be doing post upgrade that wasn't happening on v6?
This left us with a surprise $6k Azure bill for Egress traffic over 15 days before we noticed.
We moved the retention policy task to run weekly to keep this from happening each night. We don't mind running it weekly or even monthly, but we do want to understand why it is now downloading 2.4tb of data each time instead of 45gb prior to the upgrade.
-
- Veeam Software
- Posts: 174
- Liked: 24 times
- Joined: Sep 26, 2022 9:54 am
- Full Name: Pierre-Yves Bandet
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
I was under the assumption that there currently were no egress charges for o365.
Has that changed ?
Has that changed ?

-
- VeeaMVP
- Posts: 1017
- Liked: 317 times
- Joined: Jan 31, 2011 11:17 am
- Full Name: Max
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
There are no egress fees for M365. mlumsden76 is talking about the Azure costs for the storage.
@mlumsden76: Doesn't sound expected to me. Did you get a response from support?
@mlumsden76: Doesn't sound expected to me. Did you get a response from support?
-
- Veeam Software
- Posts: 3293
- Liked: 794 times
- Joined: Oct 21, 2011 11:22 am
- Full Name: Polina Vasileva
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
Hi Michael,
Thanks for this heads-up. With such a difference in behavior between v6 and v7, it looks to me like a bug. I asked the support team to involve RnD in the investigation.
Thanks for this heads-up. With such a difference in behavior between v6 and v7, it looks to me like a bug. I asked the support team to involve RnD in the investigation.
-
- Service Provider
- Posts: 37
- Liked: 10 times
- Joined: Sep 13, 2018 3:00 pm
- Full Name: Michael Lumsden
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
We've had a good rate of engagement from support, but they don't know what the cause is. I'm hoping a more senior resource can dig into this. Thank you for moving this up the chain @Polina.
I kicked off 3 repository retention policies manually by setting the date/time to immediate so I can observe them in a controlled way. The total size of the backups stored in the 3 repositories is about 2tb. The 3 repositories took about an 1h30m to complete their retention policy task and consumed about 85gb of egress. This is about 100x the egress use and much higher API call use compared to pre upgrade on VBO365 v6. It's like it's working much "harder" to apply the retention policy in v7 than it did in v6.
I compared the User guide documentation retention policy pages between v6 and v7 and I cannot find any documented differences.
I noticed in v7 they've now included the warning on the retention setting UI "applying a retention policy daily increases egress charges when using object storage".
I think this is rather uncommon. I can't find any posts on this forum, the Service Provider forum or 3rd party forums that sound similar.
One thing I've been wondering about daily retention policy vs weekly or monthly retention policy. Would a daily run retention policy use roughly the same egress as a monthly per instance? The monthly would occur less frequently, so overall it would use less egress, but would they use a comparable amount of egress each time the task runs?
I kicked off 3 repository retention policies manually by setting the date/time to immediate so I can observe them in a controlled way. The total size of the backups stored in the 3 repositories is about 2tb. The 3 repositories took about an 1h30m to complete their retention policy task and consumed about 85gb of egress. This is about 100x the egress use and much higher API call use compared to pre upgrade on VBO365 v6. It's like it's working much "harder" to apply the retention policy in v7 than it did in v6.
I compared the User guide documentation retention policy pages between v6 and v7 and I cannot find any documented differences.
I noticed in v7 they've now included the warning on the retention setting UI "applying a retention policy daily increases egress charges when using object storage".
I think this is rather uncommon. I can't find any posts on this forum, the Service Provider forum or 3rd party forums that sound similar.
One thing I've been wondering about daily retention policy vs weekly or monthly retention policy. Would a daily run retention policy use roughly the same egress as a monthly per instance? The monthly would occur less frequently, so overall it would use less egress, but would they use a comparable amount of egress each time the task runs?
-
- Service Provider
- Posts: 457
- Liked: 87 times
- Joined: Jun 09, 2015 7:08 pm
- Full Name: JaySt
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
Oh wow. Watching this one. Wondering if the (alledged)bug is Azure storage bound or more generic for other object stores as well
Veeam Certified Engineer
-
- Product Manager
- Posts: 8207
- Liked: 1327 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
As you are aware (or at least I hope so
) the case has been escalated to RnD. We are looking into it as we speak, but for now it seems we cannot "recreate" the issue internally. Hopefully we will know more after todays tests and investigation

-
- Service Provider
- Posts: 457
- Liked: 87 times
- Joined: Jun 09, 2015 7:08 pm
- Full Name: JaySt
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
@Mike, are you able to share some findings in the mean time?
Veeam Certified Engineer
-
- Service Provider
- Posts: 37
- Liked: 10 times
- Joined: Sep 13, 2018 3:00 pm
- Full Name: Michael Lumsden
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
Hi. I have an update. Veeam support acknowledged that they found an issue. They created a .zip file for me to download that contains a new Veeam.Archiver.Repository.ObjectStorage.dll file. Stop the service, replace the file start the service update the repository install steps. I have not tried it yet as I have some other work tasks in front of this one, but I should get to it later today or tomorrow. Then I'll run a retention policy task and see if the problem is resolved.
-
- Veeam Software
- Posts: 3293
- Liked: 794 times
- Joined: Oct 21, 2011 11:22 am
- Full Name: Polina Vasileva
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
Michael,
Thanks for your update; let us know how it goes.
This fix will also be delivered in the next patch.
Thanks for your update; let us know how it goes.
This fix will also be delivered in the next patch.
-
- Service Provider
- Posts: 457
- Liked: 87 times
- Joined: Jun 09, 2015 7:08 pm
- Full Name: JaySt
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
How did it go after deploying the fix?
Veeam Certified Engineer
-
- Veeam Software
- Posts: 245
- Liked: 68 times
- Joined: Jul 12, 2018 4:45 pm
- Full Name: Jim Lowry
- Location: California
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
I've spoken with @mlumsden76 a few times since he rolled it into production (we work together on a regular basis). As far as I am aware, there have been no issues at all post upgrade.
Jim Lowry
Sr. Systems Engineer
VCSP North America West
VMCE, VMCA, VCP-DC
Sr. Systems Engineer
VCSP North America West
VMCE, VMCA, VCP-DC
-
- Service Provider
- Posts: 37
- Liked: 10 times
- Joined: Sep 13, 2018 3:00 pm
- Full Name: Michael Lumsden
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
That's right. This patch resolved the issue.
-
- Novice
- Posts: 3
- Liked: never
- Joined: Jun 05, 2016 5:20 am
- Full Name: Stephan van Helden
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
Is this patch somehow publicly available?
-
- Product Manager
- Posts: 10099
- Liked: 2693 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: VBO365 Upgrade v7 - Large Azure Egress Cost
Hi Stephan
Yes, the fix is included in P20230512 (7.0.0.3604).
https://www.veeam.com/kb4425
Fabian
Yes, the fix is included in P20230512 (7.0.0.3604).
https://www.veeam.com/kb4425
Best,Resolved Issues
When applying retention on Object Storage Repositories, the Retention Job downloads an excessive amount of metadata.
Fabian
Product Management Analyst @ Veeam Software
Who is online
Users browsing this forum: No registered users and 3 guests