Public folder backup fails with: The remote server returned an error: (401) Unauthorized

[sbk]

Case #04726294
Veeam Software version: 5.0.1.179
Authentication method: Modern authentication & Allow for using legacy authentication protocols
The backup job is setup to process all Mail/archive for the organization

Problem: We recently (about a month ago) migrated public folders from Exchange on-prem to Exchange Online and ever since the move VBO365 has not been able to backup public folders (all other email in O365 is backed up correctly without a problem). The error given in the console is: Processing mailbox Mailbox1_*********@****.onmicrosoft.com failed with error: The remote server returned an error: (401) Unauthorized. We have 54 public folder mailboxes, 55k + folders within those mailboxes and over 2.6 TB of data. The backup job only attempts to process the first mailbox (Mailbox1) and that is the primary hierarchy mailbox, just not sure if the job (when working correctly) is supposed to process all mailboxes or just the one. Veeam attempts to process that mailbox for a long time before failing each time (over an hour).

The service account that is doing the backup has owner rights to all public folders.
I have edited the proxy.xml and added the line: <Source EwsImpersonatePublics="False" />

From the logs:

Code: Select all

Processing mailbox: Mailbox1_*********@****.onmicrosoft.com...
   66 (8432) Connecting to Exchange Web Services (server: outlook.office365.com, account: <backup_Account>@domain.com, ID: dc85582b-0bf1-4961-8be7-16b67a4b963a)...
   92 (4944) Adding 1 objects to processing...
   66 (8432) Microsoft Exchange Server version: 15.20.4042.16
   66 (8432) Preparing folders for items processing...
    9 (6068) Starting database contexts clearing...
    9 (6068)  Clearing old contexts in all databases...
    9 (6068)   Database contexts closed: 0
   66 (8432) Warning: Exchange Web Services request failed
   66 (8432) Warning: Failed to get folders
   66 (8432) Warning: SOAP header Action was not understood.
   66 (8432) Retry count: 0
   66 (8432)  Waiting before next processing attempt: 00:00:10...
   66 (8432)  Trying to resume...
    9 (6068) Starting database contexts clearing...
    9 (6068)  Clearing old contexts in all databases...
    9 (6068)   Database contexts closed: 0
   66 (8432)  55874 folders found
   66 (8432) Requesting folder hierarchy changes...
   66 (8432)  No folder hierarchy changes
   66 (8432) Processing mailbox: Mailbox1_*********@****.onmicrosoft.com...
   51 (5820) Processing mailbox: Mailbox1_*********@****.onmicrosoft.com...
   93 (4932) Processing mailbox: Mailbox1_*********@****.onmicrosoft.com...
   82 (7532) Processing mailbox: Mailbox1_*********@****.onmicrosoft.com...
   86 (7336) Processing mailbox: Mailbox1_*********@****.onmicrosoft.com...
   51 (5820) Connecting to Exchange Web Services (server: outlook.office365.com, account: <backup_Account>@domain.com, ID: 142c08ff-501f-45a0-bec4-7870b6f8ddfa)...
   86 (7336) Connecting to Exchange Web Services (server: outlook.office365.com, account: <backup_Account>@domain.com, ID: 2be973d6-b8f5-4954-ab4c-494903b4dea3)...
   82 (7532) Connecting to Exchange Web Services (server: outlook.office365.com, account: <backup_Account>@domain.com, ID: f174120f-8a12-4ec2-8241-b8ef11c79edd)...
   93 (4932) Connecting to Exchange Web Services (server: outlook.office365.com, account: <backup_Account>@domain.com, ID: 165dfd1a-7744-4752-a243-075880aa07da)...
   93 (4932) Exchange CAS autodisovery failed: Check the required permissions
   66 (8432) Exchange CAS autodisovery failed: Check the required permissions
   93 (4932) Error: The remote server returned an error: (401) Unauthorized.
   93 (4932) Type: System.Net.WebException
   93 (4932) Stack:
   66 (8432) Error: The remote server returned an error: (401) Unauthorized.
   66 (8432) Type: System.Net.WebException
   66 (8432) Stack:
   66 (8432)    at System.Net.HttpWebRequest.GetResponse()

I can see that something seems to be failing with auto discovery but I don't know how that process is supposed to work in a successful PF backup. Is auto discovery suppose to work for the primary hierarchy public folder mailbox? As far as public folders go for the organization, everything is working as expected from a user perspective. I also confirmed that logging into Outlook as the Veeam service account running the backup, I can see all the public folders, that account has full rights (owner) and can do and see everything for all public folders.

My ticket has been "escalated" for almost two weeks now with no responses from tier 2 (except one response from the Sydney, Australia team after I hit the talk to a manager button due to no ticket response or updates. Then my ticket was moved back to north America queue where it sits idle).

Any help or advice is appreciated. Given the amount of data that is not being backed up and the amount of time this ticket has been open I really need a solution. I'm sure its a configuration issue or something on our end but I need support to help pinpoint the precise issue.

[Polina]

Hi Shawn,

For public folders, your backup account must be licensed in Exchange Online and also must have an active mailbox. Can you check if this requirement is met?

And I'll try to get an update status on your support case. Stay tuned.

Thanks!

[sbk]

Thank you for the reply. Yes, the account is licensed and has an active mailbox.

[sbk]

Thank you Polina! You must have done something to get my ticket looked at and I appreciate it!
So far it seems that my problem is resolved now, I just had to allow basic authentication in the authentication policy that the service account was using. To do so, you just execute the following in an Exchange Online PowerShell session: "Set-AuthenticationPolicy -Identity "Allow Basic Auth" -AllowBasicAuthAutodiscover". I'm not sure if we just missed that during the initial setup that happened when we first started using Office 365 & VBO365 or if it was added sometime after but in either case you can find the reference to this being needed here: https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=50 (very last line in the article).

[Polina]

Great news, Shawn! I'm glad that your issue is now resolved )