VBO v6 - "does not have a valid Microsoft 365 license with SharePoint plan enabled"

[warnox]

Hello,

After recently upgrading our VBO installation from v5 to v6, we started seeing a lot of warnings in backup jobs with the error "...does not have a valid Microsoft 365 license with SharePoint plan enabled". Veeam support (Case ID 05363319) advised this is due to the below.

If these Personal sites belong to a user, who no longer has an SPO license, then VBO 365 [v5] would still back up the personal site without assigning a VBO license to a personal site owner.
In version 6, this behaviour has changed in order to comply with our licensing policy:
https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=60
And now, we do not back up these personal sites of non-licensed users.
Instead, we will throw the following warning:
"User does not have a valid Microsoft 365 license with SharePoint plan enabled"
And will not back up such a personal site.

While I understand aligning features with a licensing policy, Veeam should have really provided an option to do the following.
1. Automatically exclude all disabled or unlicensed users from backup jobs
2. Automatically exclude OneDrive backups for Shared mailboxes

Without the two options above, whenever a user if off-boarded, backup job exclusion lists now have to be modified. If you have a big customer base, this causes a significant impact.

I'm not sure whether I've missed something with the above, but I would like to know how other VBO users handle this and whether Veeam has any plans to provide an easy way to address these warnings.

[bmaster001]

I am struggling with the same problem at this very moment. I know there's a powershell script to exclude deleted users from a job, but excluding disabled and unlicensed users would be great too. And indeed, it would be even better if this would be a setting in Veeam instead of a powershell script that has to be run manually.
PS: We have this issue because when a user leaves the company, we convert it to a shared mailbox, and that user is left behind with no licenses...

[Polina]

Hi All,

I fully understand your concerns, but let me please explain why we've decided not to go with any automation in v6. Technically, it's a compromise between performance and licensing compliance. License/plan check on the M365 side requires additional requests that significantly affect the speed of enumerating/populating objects when you create a backup job. It'd take hours to build the list of available objects. And this is why we had to compromise the ease of managing such objects until there's a better solution. We'll keep it on our radar and if there're any new APIs from Microsoft that will make our life easier, we'll definitely look into it.

Thanks!

[wasc]

I am struggling with the same problem at this very moment. I know there's a powershell script to exclude deleted users from a job, but excluding disabled and unlicensed users would be great too. And indeed, it would be even better if this would be a setting in Veeam instead of a powershell script that has to be run manually.
PS: We have this issue because when a user leaves the company, we convert it to a shared mailbox, and that user is left behind with no licenses...

Hi

Do you know where this powershell script is? It sounds useful?

[Polina]

Ceck out this: https://github.com/VeeamHub/powershell/ ... ersFromJob

[stevepogue]

After upgrading to V6 we are now having the same problem. I understand this: " License/plan check on the M365 side requires additional requests that significantly affect the speed of enumerating/populating objects when you create a backup job. It'd take hours to build the list of available objects."

I have to monitor hundreds of daily jobs for success (Green). How about giving us an option within the product to simply ignore certain messages as Informational so the job still appears successful (Green)

[Polina]

Hi Steve,

There is a similar discussion on forums already, so it's a known nice-to-have.
Unfortunately, I can't promise it soon (it's all about setting the priorities in case of limited resources...), but we keep this request in mind and I do hope it'll become part of one of the next releases.

[AlexHeylin]

I'm so disappointed and frustrated to have to write this.

Guys - here's my £0.02... you made a half-arsed change that now causes your partners additional toil and false positives in monitoring / backup result reporting.
Complying with your own license is one thing - rushing out a change that's badly thought out in terms of impact is not something anyone here is happy about. V6 seems to have been slung out the door with a number of changes that are good for Veeam and bad for customers and particularly VCSP. Changes I'm surprised got made like this, and I'm even more surprised they made it through an informed UAT process. When you create a problem for customers, you create (tenant count) x (that problem) for VCSP. VCSPs cannot practically update the job each time a user has their license changed. It would be cheaper to rip VBO out and replace it with another solution than to add that TOIL! Don't say the update is the tenant's responsibility - in our case it's ours as MSP offering fully managed backups. Please get your head around this use case and stop treating it as a use case you can ignore when making changes / designing systems.

Can we just be clear what you're saying - if the user had an SPO license and data was created in their site (OneDrive for Biz typically), then gets downgraded to a non-SPO license (such as EOP1) then VBO will generate this warning on every backup and the ONLY way to resolve it is to exclude the user from backup ENTIRELY, or upgrade their license again?

If you're going to mention hacking the proxy.xml AGAIN - that's not an acceptable way you should have implemented anything. Not least there's multi-tenancy to consider, so one size might not fit all.

I wish we'd never upgraded to v6. I see the latest patch went out and first in the change log it mysteriously includes a fix for something I was told was a documentation issue after I reported it. Now I'm starting to wonder if we can believe what we're told - and if we should ever upgrade any Veeam product without a very good reason because apparently Veeam are going to have "fixed" things from their point of view, which looks very much like "broken" things from our point of view.

Case #05508116

[Polina]

Hi Alex,

That's a pity that you're so disappointed with the v6 release, but I'm afraid there's not much we can do to fix it at this point.
From my perspective, if a user account has been explicitly selected to be protected, the product must report on any issues that occurred during processing. And silent removal of problematic objects from jobs would be unacceptable.

[AlexHeylin]

Hi @Polina,

Being included in a "backup everything" selection is not "explicitly selected", it's "implicitly included" - thus your logic doesn't apply.

When dealing with VBO365, the obvious logic is (and has been to date) that the VBO365 admin knows that "backup everything" means "backup everything with a license and don't tell me about the things that don't have a license - because *I* don't care - that's someone else's problem" not "backup everything with a license and please annoy me (with a message that I can't silence) about each individual user that doesn't have a license - because I have nothing better to do than modify backup jobs every time someone else in another team (or possibly the client or another vendor etc) removes the license from a user, probably because they've left".

Note, that's different from "We detected user x has a valid license [for some part of O365] but that license doesn't include SPO, yet we see SPO data for them - which we're not going to back up" That situation is one in which the warning might be useful - though it still needs a way to shut it up - because the user CANNOT be removed from the backup job, because valid data is still being backed up from sources other than SPO.

However, that's not the situation we face. We're now being nagged about users who have been delicensed in O365 while data remains in SPO. As the MSP running their backups, I neither care that this happened, nor why it happened. I care about my automated backup checks passing, not being screwed up by "false" warnings that Veeam decided to roll out without ANY way to silence them.

If you think I'm being unfair - please let me know where.

Thanks

Alex

[AlexHeylin]

I'm afraid there's not much we can do to fix it at this point.

I'm sorry but we both know there is a lot could be done to fix it. Unless Veeam is never going to release any patches or newer versions of VBO ever. At the VERY least, using the same "option in the proxy.xml file" approach as some of the other messages would be better than the "absolutely nothing" approach Veeam have currently coded into the product.

From my perspective, if a user account has been explicitly selected to be protected, the product must report on any issues that occurred during processing.

See that's where we differ - this isn't an issue. It's an expected situation.

silent removal of problematic objects from jobs would be unacceptable.

I agree - but that's not what we're discussing. We're discussing not creating unsilenceable warnings for an expected and legitimate situation. A situation over which the VBO admin may have no control or influence. Expecting the VBO admin to suddenly start spending time every day excluding users from their jobs (think MSP / VCSP scale) to avoid this message is ridiculous.

It's interesting to note that semi-automated "removal of problematic objects from jobs" (via powershell) is exactly what support suggested. So it looks like support know that's not a very strong position to take and that at scale some means of dealing with these messages is required. I disagree that editing the job is the correct solution, it's the only one they have because this message was apparently designed without any way to stop it happening - even in completely legitimate circumstances.

Please convey to your product management / design teams that when running Veeam products at scale, particularly VBO which is claimed to be multi-tenant, but we all know isn't really - that one size fits all thinking can cause VCSP and partners (particularly MSPs) significant problems which a little additional thought (or a good deal less assumption) would have avoided.

Thanks

Alex

[savanki1]

Hi ,

Im having the same problem, i have tried to disable from Veem to not backup the user sharepoint/OneDrive/mail. (created an exclusion).

Also removed the user from doing the backup and i still got warnings.

Any other method?

[chi-ltd]

Same issue for us.

[Polina]

@AlexHeylin

We did another check after your post and you're right, when backing up an entire organization, these warnings are excessive and not consistent with how we work with other types of data. In the next update, this issue will be fixed.

@savanki1
Could you please add for me steps to reproduce your issue, because I'm not sure how you possibly can be notified about a user who's even not in a job?

[AlexHeylin]

Thanks!

I think @savanki1 was saying they excluded the sharepoint and onedrive (only) for that user from the backup job, but left their Exchange etc in the backup job. This results in the error. Logically I would expect that, given what we've seen.

[Polina]

Hi Alex,

In my quick tests, this issue is not confirmed. If after upgrading to the latest patch (see the https://www.veeam.com/kb4285) this problem with exclusions remains, please open a support case and post its ID here so that I could follow it up.

Thanks!

[AlexHeylin]

Hi,

I don't understand why when this was already a confirmed issue - it suddenly became an unconfirmed issue. If it was resolved in 6.0.0.400, there's a whole new load of pain in 6.1.0.222 that's related to this issue.

It seems on 6.1.0.222 that this issue has got worse and now for every O365 user with a personal SPO site or OneDrive but no O365 license (a perfectly valid scenario that does happen legitimately) VBO now throws two big red errors.

Failed to process OneDrive: ##REDACTED##. The underlying connection was closed: An unexpected error occurred on a send.. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. An existing connection was forcibly closed by the remote host
Failed to process site: ##REDACTED##. The underlying connection was closed: An unexpected error occurred on a send.. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. An existing connection was forcibly closed by the remote host

I don't have the words to express my disappointment and frustration at this, given I though we'd agreed that the approach originally taken in v6 GA was "highly suboptimal at scale, and where the backup is operated by a different team / org / company to the O365" and that it's not reasonable to expect VBO admins to constantly edit their jobs to avoid this.

Please can someone jump on this before many more people install this upgrade and end up worse off than they are now?

While the new features and improvements are very welcome, I think all the v6 releases needed more thinking about and certainly more testing at scale before they were made available or pushed to customers. The QA seems to have been disappointing on all v6 releases. Each time we've upgraded to get away from one problem we've been hit by another, which might be worse.

Case # 05588000

[pesos]

subscribing with interest

[AlexHeylin]

Support have confirmed that 6.1.0.222 generates error messages under certain circumstances - one of those circumstances appears to be because having no O365 license on the user causes the "createdDateTime" property for the user to be null, which VBO tries to cast into a System.DateTime without checking for null value first - so an error is thrown.

My personal recommendation is don't upgrade to 6.1.0.222, and if you're on a version earlier than 6 - I'd stay there unless you have a compelling reason to upgrade because so far every release of 6 has caused us new problems.

PM's - please give serious consideration to if the QA & beta testing for the whole of the 6 release has been appropriate. I suggest there is scope for improvement there.

[Polina]

Hi Alex,

I'm following your case together with support engineers and QA. According to the preliminary research, the issue is related to a known issue recently found. The hotfix is already available via support and we're working on the patch that should be rolled out shortly.

This issue only occurs in very rare cases and unfortunately could not be caught/confirmed in our QA labs - VB365 receives an unexpected response from Microsoft. We also work together with Microsoft to better understand the conditions that trigger this issue in Microsoft 365. More information should come soon.

And my apologies for this inconvenience.

[K.KB]

Also subscribed with interest.
In our multi tenant installation we have a ton of Jobs finishing with "Warning" because of things like this:

Processing mailbox XXXXXXXXXXXX completed with warning: Mailbox does not have a valid Microsoft 365 license
Processing site https://XXXXXX.sharepoint.com/personal/XXXXXXX finished with warning: User XXXXXXX does not have a valid Microsoft 365 license with SharePoint plan enabled

If those "Warnings", which are not legitimate in our eyes, can be silenced like AlexHeylin said on » Jun 30, 2022 8:51 pm, We would finally have valid results in our morning checks.
Looking forwards towards the solution!

[Polina]

Hi Koen,

Displaying/hiding these warnings depends on your backup jobs configuration. If you're processing an entire tenant organization, they will be suppressed (the mistakenly displayed warnings for personal sites are now fixed in the latest v6a). However, when specific users are selected for backup, we'll show these warnings to let you know that part of the object that you want to protect cannot be protected.

At the same time, I hear you loud and clear, for service providers this behavior might be not optimal and we'll think about how to adapt it.

Thanks!

[AlexHeylin]

Thanks @polina - however the hot fix from support made the situation much worse, not better.

[Polina]

Hi Alex,

From what I know, there was a different reason, not the consequence of the hotfix. But I'll ask the team for an update on your case.

[AlexHeylin]

Hi @Polina
We installed the hot fix and the backup job just errored out and quit without backing up anything. We removed the hotfix and it went back to backing up the licensed users and throwing connection errors about all the unlicensed users. To me that's pretty compelling that the hotfix made the situation much worse.
Thanks

[rm@kompetera.dk]

we also have a MSP setup with full managed backup and get a lot of these "does not have a valid Microsoft 365 license" which make it a pain to check every day and then you end up checking that it is a warning like yesterday and say that is ok then until at some point another warnings comes which is in our case meant no backup (how that was only a warning I don't get) which left the customer without a backup for 14 days before it got caught because the customer wanted a restore which was pretty hard. That customer is luckily still with us but they could just as well have left the building and with it Veeam backup so I really hope that there at least will come some sort of checkmark or something where you can say it doesn't count for a warning or only the first day and after that no warning but in the last case if some leaves the company everyday you will be fucked regardless but would have been a nice one for normal backup as that would cover many of them and then the checkmark to turn it off completely also.

I really hope this will be covered in a future update.

[K.KB]

Hi Koen,

Displaying/hiding these warnings depends on your backup jobs configuration. If you're processing an entire tenant organization, they will be suppressed (the mistakenly displayed warnings for personal sites are now fixed in the latest v6a). However, when specific users are selected for backup, we'll show these warnings to let you know that part of the object that you want to protect cannot be protected.

At the same time, I hear you loud and clear, for service providers this behavior might be not optimal and we'll think about how to adapt it.

Thanks!

Hi Polina,

We do have 6a P20220825 (build: 6.1.0.254, have all our jobs configured as entire tenant organization (split into 4 jobs, a job for Mail+Archive, OneDrive, Sharepoint and a job for Teams) but we are still getting the warnings about missing SPO license or missing Microsoft365 license. Is there a hotfix other than the one Alex Heylin got that does not break the functionality? As a SP we would love to finally get rid of the "warnings".

Greeting Koen

[Polina]

Hi Koen,

In 6.1.0.254, such behavior is not expected; please open a support case to investigate. All the excessive warnings have been removed and the latest build should already include all the fixes.

Thanks!

[ks.any.cloud]

@Polina we see the same issue on our v5->v6a upgrade, but only 4-5 tenants. just to confirm that the error is present.

[Polina]

Hi Kim,

I asked support engineers to contact you on this case. Please share logs and additional details with them so that we could troubleshoot it.

Thanks!