After upgrade to 6.1.0.254 sharepoint job stopped processing all sites

[sumeet]

Hello,

In the past I have asked few questions and when mentioned that we are still using version 5 + latest patches, we were told why in the age of clouds we are still behind in version updates -- exactly for this reason. Never has our experience with latest major versions been good. Same was with v5

Case sev 1 - 05662195

Last week we upgraded few our clients setups to 6.1.0.254 and observed that sharepoint job was only processing 12 sites/objects instead of 2400+ sites.
The site count varied for each client, but this is just so that PMs understand our experience, hence mentioning the count for a single client.

When opened a sev 1 case, I was given a HF, after applying that HF, the Sharepoint backup seem to be running better, than finishing up in 3-4mins with 12 sites. It seemed like we solved the issue, but the next day when I checked the reports, the total site counts did not match with the count that was processed with version 5

I was asked to apply the latest patch 6.1.0.423, but no change.

Has anyone experienced similar behavior?

Thanks,
-Sumeet.

[sumeet]

Forgot to mention that on a different setup, upgraded to 6.1 latest patch, all sharepoint backup runs transfers 360+ GB of data.

1. v5 - 4379 objects
2. v6.1.0.254 - 2848 objects with 368 GB of data
3. v6.1.0.423 - 4072 objects with 366 GB of data
- Strangely all three last backups with 6.1.0.423 has processed 360.8 GB, 361.4 GB and 366.7 GB of data
If I check the last few backups with v5, all backups had 4379 objects with data processed in 1.4, 1.6, 1.9 and 1.3 GB of data
Post v6, all backups processing 360+ GB of data is suspicious

Verified last nights backup that it has processed 4199 objects with 361 GB of data transferred.

The object processed is less than what was with v5, but not as much less as other clients, but strangely each backup run has 360+ GB of data.

[Mike Resseler]

Sumeet,

I noticed in the case you are working with our tech support. Please keep working with them and hopefully they find out what is going on. I believe we did some changes in counting the objects (objects are not the same as sites or something like that) but only 12 objects seems like very low number.

Question: If you start the Veeam Explorer for SharePoint, can you see the latest data in that explorer? (For example, upload a new document to 1 site, take the backup, and see if it is available in that restore point)

[sumeet]

Mike,

I do not have access to the front end to upload documents to sharepoint and then verify the backups.

Will appreciate if you can assist any further with this case.

It has been long since this sev 1 case is raised and the only thing support ask me to do is add backup application. For one of the clients there were no backup applications and throttling was high. I agree that throttling should be low, but all sites were processed with version 5.
Not sure how after upgrade to v6, the throttling can cause the sites to get skipped. In all these years of using VBO, I know that throttling can cause the backups to run slow or fail. But skipping objects/sites is new to me.
Anyways, being desperate I agreed to support and added 10 backup applications. Now the throttling is in single/double digits (pasted below) and the object count is still the same as observed after version 6 upgrade.
Now support wants me to add another batch of 10 backup applications. Does not make any sense to me????

And is insisting I do the same with rest of my clients. Not sure how if it does not work with one client, how will it work with others.

Further, in all scenarios we do not have full access to our customers tenancy. And asking them to add backup applications is not a easy process.



[18.10.2022 21:42:07] 60 (15580) Accounts throttling statistics:
[18.10.2022 21:42:07] 60 (15580) cf19fa3d-2755-4fbb-aa1a-0ad51d40d***: requests 19442, throttled 3
[18.10.2022 21:42:07] 60 (15580) 46e57678-c7b3-4dc6-b4b8-61233cab7d: requests 15314, throttled 1
[18.10.2022 21:42:07] 60 (15580) 36165d30-39e9-499b-a52c-515ef11fc: requests 17837, throttled 2
[18.10.2022 21:42:07] 60 (15580) 6bef1c1e-fc9b-49b5-9723-6dcfbe632: requests 17173, throttled 4
[18.10.2022 21:42:07] 60 (15580) e9bb0b7c-d2af-4338-9b7c-d7da53bd4bac: requests 15040, throttled 2
[18.10.2022 21:42:07] 60 (15580) f5c0d52f-4e44-402f-8662-e699885be: requests 13531, throttled 1
[18.10.2022 21:42:07] 60 (15580) beb73106-089a-4d3e-a8f4-24e15aa80: requests 16933, throttled 4
[18.10.2022 21:42:07] 60 (15580) 65a7189c-3726-4b6f-bf8e-ad708068d: requests 17001, throttled 1
[18.10.2022 21:42:07] 60 (15580) 947ccea1-abef-4067-b0cd-062e5fc8a6: requests 15141, throttled 1
[18.10.2022 21:42:07] 60 (15580) 7859cd8b-8502-4937-b39f-a3c3643b7d: requests 23409, throttled 82

Note: I have removed few characters from app ID above for security purpose.

[Polina]

Hi Sumeet,

Adding another 10 apps doesn't make sense to me either. Let me get the details on your case from support to see how we can proceed with resolving the issue.

Thanks!

[sumeet]

Thanks Polina,

Yesterday support asked me to add 6 backup apps, which I denied.

Today I was asked to provide list of few sites that are no longer discovered. I already did provide this information last week.
Further as requested by support, I have attached four sharepoint backup reports. Details below of the each report attached.

1. First report listing 1937 objects/sites that ran on the night of 4th Oct with version 5
2. Second report listing 7 objects/sites that I ran on 5th Oct after version 6.1.0.254 upgrade
3. Third report listing 7 objects/sites that was scheduled run on the night of 6th Oct
4. Forth report listing 1445 objects/sites that I manually started on Friday 7th Oct after applying the HF provided by the earlier support person. The HF was applied on version 6.1.0.254


As you see in the reports attached to the case, after 6.1.0.254 upgrade, the backup only processed 7 sites out of 1937 sites with version 5.
After applying HF provided by support on 7th Oct, the processing improved from 7 sites to 1445 sites, but they still do not process all 1937 sites.

There is definitely some change in VBM, that caused the backup to only process 7 sites instead of 1937. The HF did resolve the issue partially but not complete.

MOD removed link to hotfix as these are case specific

A sev 1 case opened on 7th Oct, and going on. Until two days ago the only thing I feel support did was to ask keep pushing me on adding more backup apps.

-Sumeet.

[sumeet]

When I opened the case, I mentioned to support that is this issue of sharepoint sites being skipped specific to upgraded setup. I was informed that it is only observed for upgrade scenario.
The reason I asked is because for a fresh 6.1.0.254 deployment for a new client, the first full backup only processed 27 sites, whereas they are a large tenancy.
I did check with our client if he can provide me the total site list, but never got a response.
I applied the 6.1.0.423 just now and started the sharepoint backup and now in 20mins I see the job is processing 20 objects of 523.
From 27 sites to 523 and counting.

My mistake, I should have applied the 423 patch last week.

I have uploaded the logs and hopefully it helps debug the scenario.

Thanks,
-Sumeet.

[Polina]

Hi Sumeet,

Thanks for getting back on this. I'm following up on the investigation together with our support engineers and the QA team.
The lits with the total number of sites from your customer would be really needed to verify the current state of production vs backup.

[sumeet]

The following update we got from support on 9th Nov

=======================
We found that the remaining objects that are not discovered, are for O365 users that are not licensed or deleted.

In v5 Veeam was backing up everything in SharePoint, included deleted users, or unlicensed 365 users, while in v6 it only backs up objects for licensed users.

This means that to backup all user sites, will need to assign licenses to all of them.
=======================

Not sure why such a change was not highlighted by Veeam in release notes?
Not sure why no one else using VBO has observed this and that the total objects being backed-up has changed/reduced after the patch?

The most concerning thing is that being able to backup everything in a Org irrespective of the M365 license was a big part of Veeam's sales value proposition. Now this changes everything.
Now adding a new customer backup, do we have to explicitly call out to our customer that the day VBO backup starts, only the users that have M365 license will be backed up. All users that do not have license on the day will not be backed-up.

[sumeet]

In another week it will be 2 months since I opened this sev 1 case. Still waiting for more details Veeam support to understand the primary reason for this unexpected behavior of the product.

[Mike Resseler]

Sumeet,

Those remaining objects. Are those SharePoint sites or personal SharePoint sites? If it is SharePoint sites, then the remark from support does not make any sense. If they are indeed personal SharePoint sites (or OneDrive for Business) and belong to deleted / unlicensed users, then it is normal. That never worked before in any version. Just want to make sure that I understand exactly what sites you are missing?

[sumeet]

Mike,

They are sharepoint sites.
Let me add more details here. I patched a setup on 1st Dec from version 6.1.0.254 to 6.1.0.423.

I cannot put screenshots here, so I will copy paste snippets from the Veeam report.
Prior to the upgrade - SharePoint_01 (278 objects), 72 issues
Post this upgrade - SharePoint_01 (414 objects), 86 issues

To be honest, it is not easy to match a veeam report across versions 6.1.0.254 with 6.1.0.423 to find exact difference. But I looked at the data size column and picked up few where the data is in GBs.
Found a site that is success in version 6.1.0.423. Pasted snippet below, but masked out few details for privacy.

Web https://****************.sharepoint.com/sites/DDAct******** Success 12:07:49 PM 12:45:18 PM 5712 6.5 GB

When I search this site DDAct in the old report, it is not listed.
This is not a personal share point site.

Another fresh deployment that I mentioned earlier in the thread.
With version 6.1.0.254 sharepoint backup report has this -- [Warning] *********.onmicrosoft.com - SharePoint_01 (27 objects), 15 issues
After patching to 423 the backup ran for 10+ days with 1.3 TB of data and had this in the report -- [Warning] ********.onmicrosoft.com - SharePoint_01 (5518 objects), 2120 issues
Jump from 27 objects to 5518 objects.

Definitely cannot be just the personal sharepoint users when objects have increased from 27 to 5518 objects. Not sure even with version 423 if there are sites missing.


If you check the case and send me an email, I can discuss these in a remote share sessions and show these reports to you.

Thanks,
-Sumeet.

[sumeet]

Mike,

Missed adding another the following in my above reply.
The setup that I patched from 6.1.0.254 with 6.1.0.423, when this setup was at version 6.0.0.385, it processed SharePoint_01 (408 objects), 85 issues.
This setup was patched on 4th Aug 2022 from 6.0.0.385 to 6.0.0.400 and that nights sharepoint sites backup report has - SharePoint_01 (277 objects), 68 issues

So something changed between patches 6.0.0.385 to 6.0.0.400 and it remained so until 6.1.0.254. Then again something changed with version 6.1.0.423, making the backup jobs to not skip sites.

Will request you to check with your R&D to perform a code diff between these two patches and hopefully this helps to find what is causing the issues.

Again, I have these reports with me, if you or your teams wants to take a look, please check the case and reach out to me through the case.

Thanks,
-Sumeet.