Some excluded objects were not resolved

[AlexLeadingEdge]

Firstly, I cannot open a ticket because Veeam 365 is not tied to my account, and because I cannot open a ticket I cannot call Veeam to tell them there is an issue. I cannot open a forum post on this forum because I don't have a current case number (?!) Your whole ticketing system is **** and pisses me off every time I need assistance :@ I would like someone to contact me to tell me how this could be improved so I don't waste an hour trying to contact Veeam when things have fallen over. Anyway, I'm going to ignore your request for an support case ID because I cannot generate one with my support ID, so here is my issue...

For some reason all Microsoft 365 backups have failed today for one client.

The backup seems to sit on "Connecting to organization..." for 7 minutes and 23 seconds, then says this after a further 6 minutes and 45 seconds:

24/07/2024 7:11:36 AM :: Some excluded objects were not resolved. Failed to resolve user (name: Discovery Search Mailbox, ID: REDACTED:00000000-0000-0000-0000-000000000000) :: 0:06:45

I tried to "Edit org" and re-connect using a new self-signed certificate, assuming that perhaps the certificate has expired, but it sits on "initializing..." for a long time and then finally allows me to log in with the device login code. It then seems to sit on "Waiting for user logon" which I've already done and closed my broswer. After several minutes it then says it has authenticated. It then hangs again on "Connect to Microsoft Graph" for 7 minutes and 43 seconds, and then hangs on "Connect to EWS" for 1 minute and 27 seconds, and then hangs on "Connect to Sharepoint" for another 4 minutes and 35 seconds, and then "Connect to PowerShell" for another 6 minutes and 45 seconds then fails with this error message:

24/07/2024 11:11:05 AM :: Connect to PowerShell: Case when Message contains:REDACTED Invalid scope. :: 0:06:45



I assume some sort of timeout is occurring?

I'm not sure if this is related to the CrowdStrike issue over the weekend, but I can't see any of my other clients having issues so have to assume it is something for this one client (?) I also cannot see anyone else on this forum having an issue.

[AlexLeadingEdge]

Just FYI...

Veeam Support - Escalation # ESC0066398 - General Case Escalation

[AlexLeadingEdge]

The same client has just reported that Mailstore (which is connected to 365) is also not working, which I have confirmed.

They get the error message "Mailstore Server was unable to retrieve a list of users from Directory Services. Request to the endpoint timed out."

[AlexLeadingEdge]

Just FYI...

Veeam Support - Escalation # ESC0066406 - Licensing Case Escalation

[AlexLeadingEdge]

Veeam Support - Case # 07351711 - Cannot open tickets for critical outage

[Mildur]

Hi Alex

I would like someone to contact me to tell me how this could be improved so I don't waste an hour trying to contact Veeam when things have fallen over.

Prepare your case administrator accounts before you have an emergency: https://www.veeam.com/kb2211
There is also a KB for Service Provider: https://www.veeam.com/kb4461

You need to do the same if it‘s about a license from a client of yours. The customer has to assign you as a case administrator.

Both kb also describes how to assign a new license administrator account in case you have lost it.

If case administrator still not works for you, provide feedback through a general case as you have probably done with your 3 cases today. Product management has no access to the ESC cases, so lets wait for them to answer back with how you can fix the case administrator solution.

About your error. Are you still using legacy/basic auth? I see that the wizard tried to connect through PowerShell. This should not happen with ModernApp only auth. I assume Microsoft disabled Basic auth in your clients tenants as they have done in millions other tenant.
Please change to ModernApp only and try again: https://www.veeam.com/kb4365

Basic/Legacy auth cannot be used anymore and will be removed from the backup console UI soon.

Best,
Fabian

PS:
Please register yourself in our service provider user group if you are a service provider. This gives you access to our hidden service provider sub forums where service provider products are discussed: Apply for the Cloud and Service Providers user group

[AlexLeadingEdge]

Thanks, will look into that when I'm not trying to fight fires. 24 hours later and I haven't got an email from the Veeam Licensing staff. I have just got this automated email:



When I logged into my portal I can see they've replied, but I have received no emails to notify me that I had a new email. The automated email even fails to show the reply it claims was "sent to you below".

They've made me the license administrator, but when I try to add a Support ID to myself it says I can only add Support IDs to those in the predefined list, which is completely useless to me, and I STILL can't open a support ticket.






As for the PowerShell, I am using the Modern Authentication. I have tried to reconnect and got the same PowerShell error:






I see that if I ping graph.microsoft.com from anywhere other than the client that has issues it goes to www.tm.prd.ags.akadns.net [20.190.142.40] but if I do it from the client it goes to www.tm.prd.ags.trafficmanager.net [26031400:68::81] and all pings are lost.

I have flushed the DNS and now it goes to www.tm.prd.ags.akadns.net [26031400:68::82] but still won't ping.

When I force the ping to only IPv4 it works (www.tm.prd.ags.akadns.net [20.190.142.40]).

[AlexLeadingEdge]

I called Veeam Support using my case number for the license issue, they confirmed that the license is under my company, not the client because we are a VCSP.

I was able to bring up Veeam Agent for Microsoft 365 under my company, but I cannot click the Submit button. I screen shared with the Veeam Teachnician and they confirmed there is a bug in the submission page and they will send a message to the devs. They then suggested that I open a ticket under Backup & Replication instead just to get the ticket open.

They said that someone from Veeam will call to have a look at the issue, but they suspect as I do that it is a Microsoft issue and Veeam won't be able to help.

[Mildur]

Hi Alex

I can see case 07353581 with severity 1 was opened for this issue 27 hours ago.
Unfortunately I don't see your answer yet in the case itself. Logs are also missing. Do I have the correct case number?

If the issue is with our RnD team, I want to check it's status in our internal RnD system.

Best,
Fabian

[AlexLeadingEdge]

It is (was) the correct case number, but I never received an email to tell me that someone from Veeam Support had replied. Five days later I've received an automated email to say that Veeam hasn't heard back from me and Veeam was waiting for my response.

I've closed the case, it turned out to be a Managed Switch was corrupting the data stream to Microsoft and the Firewall was blocking all the traffic as spoofed. A simple power cycle of the Managed Switch has fixed the issue. It took me three days to isolate the issue, ruling out Veeam365 (given Mailstore stopped working it was a longshot that Veeam365 was the cause), the Server, the Firewall, and finally looked as why the Firewall was blocking IPv6 traffic to Microsoft servers (graph.microsoft.com).

As for the licensing issue, it turned out that the issue wasn't actually an issue, the client is managed through a bulk VCSP Products Bundle so came under Leading Edge (my company) not the client. My confusion came about because I was getting the support code from the client's Veeam365, but had no way to realise that the support code was for my company, not the client.