Major issue with incomplete SharePoint backups, files are missing and lost

[rairiau-omr]

Hello,

This topic to inform you that we encounter a major issue with Veeam Backup for Office 365 backups for SharePoint, unable to restore some files, the backups were incomplete and now files are definitively lost.
You have to manually run Full Sync backup on jobs with last patch deployed to have all files in backups, but too late in our case.

Veeam support case #05160244.

Some details from support :
The issue at hand is a known issue that was found on previous version of VBO365 which affects all backup repositories containing SharePoint/OneDrive/Teams data.
The root cause for the missing data was due to incorrect representation of the changes from the SharePoint API side.
Veeam RnD team performed an investigation and found that sometimes the SharePoint API mechanism of tracking changes did not track the changes inside the Child files or Folders inside the Site`s list.
For example, a child folder is created within the folder that is already backed up, or a folder appears in the root of the list - but the incremental backup is not backing up these folders because they are not indicated as a change by the change tracking mechanism.
As a result, some folders and subfolders were missing from the backup, even after the incremental backup job runs.
This change tracking information received from the SharePoint API is then stored inside Veeam Backup Repository for the incremental runs.

The Veeam RnD team developed a fix that improved the way how VBO365 works with change tracking of the SharePoint API, but the fix only changes the way how VBO365 requests and uses the Change Tracking information with SharePoint API, but it does not fix the already affected backup repositories.

To resolve the issue inside Backup Repositories we need to refresh the change tracking information which is stored inside the Veeam Backup repository by running the Full Sync backup for each backup job that is configured to use the affected repository.

[karsten123]

really? Is there official documentation for this bug?

[rairiau-omr]

Yes it's real in our case, as well the backup job concerned all the SharePoint sites, and there was successful, some files are missing in backups.
Our client have an issue on SharePoint storage due to Onedrive sync bug, and try to restore files, but some of these files were not backup-ed by Veeam vbo due to issue explained, and so, there are lost definitively.

The last patch, KB4222, explained that there was an issue with repository on object storage, but in our case we use Windows Server repository (only one vbo server with all roles).
Actually Veeam team is re-checking the situation internally with QA/RND and Product Management teams to define what is needed for next steps.

[nielsengelen]

Hi,

Correct, this case is under active investigation together with R&D and we hope to provide an update as soon as we get more details.

[karsten123]

Oh my god. That sounds very bad.
Niels, please give us an update asap.
Tia

[nielsengelen]

This case is still being investigated as we speak. Once we get to a resolution, we can update this topic.

[MarkG-ZA]

Hi all, any feedback regarding this case? We are also having a similar issue with a client, the sites backup without errors but some folders and files appear to be missing when browsing for them in Veeam Explorer.

Regards
Mark

[Polina]

Hi Mark,

Romain's case turned out to be very specific and it's still in troubleshooting.
Even if you see similar symptoms, the root cause may differ. Please create a support case on your issue and post your case ID here for the follow-up.

Thanks!

[MarkG-ZA]

Good day Polina, thank you for the information.

We have logged the case. Site not Displaying in Explorer - Case #05240929

Thanks!

[dasfliege]

We've been informed by one of our customers that he's not able to see all folders inside his sharepoint backups. We've created a case (07527751) and according to the current state of investigation it seems like this data has never been backed up even though the jobs were successfull all the time. Veeam support pointed me towards the following kb which may could have lead to such a behavior: https://www.veeam.com/kb4447

As i see now, this issue has already been brought up quite some time ago. But there is still no mechanism in veeam to check if really all data has been saved which is selected for backup. We now have to explain to our customer that his data has never been saved correctly. That's unaccaptable behavior for a backup solution. I understand that issues like this can happen but there should at least be some kind of warning or something inside veeam, which reminds you to perform full backup of all jobs which could potentially be affected by this issue.

[dasfliege]

As we are performing yearly extracts of VBM365 data for our customers, we’ve just received feedback from another customer who is missing data, but this time it’s missing in Teams.
I performed the same steps as i did for the sharepoint backup which was missing data (Temporary job with full sync) and the data was visible afterwards.

This is an absolute disaster scenario, as we obviously have no idea how many of our customers have not backed up all their data over the past few years and veeam gives no indication that there is missing data. Also we have not been informed in any way that such a major issue exists. We need to communicate this to our customers and want an official statement from Veeam about what caused this error. We will also involve our key account manager to get the required statement.

[karsten123]

is it recommended to start a full sync (kb4447) as a precaution for all jobs?

[dasfliege]

is it recommended to start a full sync (kb4447) as a precaution for all jobs?

That's what i assume from our observations. But i want an official statement from veeam.
Probably a lot more customers are experiencing the same issues but are not aware of it as they don't had to perform restores of missing data yet.

[karsten123]

I just started a full sync of all of my tenants. Better safe than sorry. Hopefully

[dasfliege]

@Polina
Any chance to get a statement from Veeam regarding this issue?
If there is a risk of potential data loss for many many customers of yours, i wouldn't rely on the assumption that one previously analysed case was only caused by "very specific" circumstances.

[bbuckles]

We just ran into this with one of our customers who needed a restore of a OneDrive folder that was not being backed up. This has caused quite a stir that they're missing backups for multiple years. I would also like an official statement regarding this as this is really unacceptable. There has to be a way to validate that Veeam for M365 is actually backing up what it says it's backing up without having to manually check the backups and compare to what's out there in M365. Case #07563434

[Polina]

Hi bbuckles,

Before providing any comments, let me please review your support case and get all the details to better understand the issue itself and its root cause.
I'll get back shortly; thank you for your understanding.

[DaStivi]

I just started a full sync of all of my tenants. Better safe than sorry. Hopefully

what did you do exactly?? just "start-vbojob -full" ??

i've tested this with a few small customers now and for me it looks like this is running a full full-Backup... propably it was missing that many items... but i feel its not a just a resync and backing up missing items.. more like its taking another "full snapshot" of all data?? and hence its taking double the amount of "full-backup-space" then!

[karsten123]

kb4447 but for all jobs

get-vbojob | start-vbojob -runasync -full

the job is shown as full in job history, but it transfers only the inc.

[DaStivi]

Okey... I've tested with some jobs.. and I see a lot more transferred data on the full job compared to the incremental runs the days before...

I also test with a recently created job... Where the initial backup is just 1 or 2 weeks.. and even there I'll see in the additionally "full" more transferred then in the incrementals. ... So either the initial issue, that not everything is backuped is still there and the patch back in may 2023 didn't helped or there is a lot more backed up in this full run... Not only "increments" but I am also pretty sure it's no "active full" (according to VBR language)


Else: if the full cycle really transfers that much data, for example, 8GB compared to only a few 100mb or at some days max. 2GB , this means there is a lot of data missing . We don't have a relaxed weekend as service providers taking care of multiple 100TB of m365 Data if this gonna get true it's a nightmare!

[edh]

Hi,
Case #07572674

One customer reported some Sharepoint folder where missed from backups. Via SelfService restore portal show as "Acess Denied" , at Explorer show as empty. Customer is 100% sure that files exist in the folder.
Checking this post comments looks like this a know shadow bug? (https://www.veeam.com/kb4447)

[DaStivi]

did anyone got some new informations? i'm working through my backup jobs, running "full" syncs... and comparing to the incremental runs on all jobs its disproportionate a lot of data that is transferred!
for example:

~20GB instead of max 2GB increments
1.5GB instead of max 500MB increments
15GB instead of a few hundred MB!
3.2GB on a very new job that has only been run since 2 weeks, so first full isn't that long ago... but the increments before and are also only a few 100MB!!!
11GB instead a max 1GB...

and its continuing this way , having 77Jobs and its more then disturbing if the numbers speak truth!

[Mike Resseler]

@DaStivi Can you create a support call? That a full sync will have some more data download (some of which will be "disregarded" after that) but this seems like a huge difference. Could you create a support ticket so we can investigate?

[edh]

@DaStivi Can you create a support call? That a full sync will have some more data download (some of which will be "disregarded" after that) but this seems like a huge difference. Could you create a support ticket so we can investigate?

Just a question the start-job -runasync -full , the full param will create a FULL downloading again all objects duplicating the space used or just check all object from origin to destination and downloading the difference/missing?

[DaStivi]

no the "full" parameter definitely not download an "active-full" !

[bbuckles]

So the question again is, how do we know that we're actually backing up all the data that the backup job is supposed to be creating? We can't continue to sell this to customers if we can't guarantee their data is being backed up? It's a huge liability. Comparing users files/folders to what shows up in the backup explorer is what was suggested to me, but that is not a valid process.

[DaStivi]

@DaStivi Can you create a support call? That a full sync will have some more data download (some of which will be "disregarded" after that) but this seems like a huge difference. Could you create a support ticket so we can investigate?

i've created a case today: # 07584451

still running full jobs, take some time because i start them only one by one... but the high amounts of transferred data continue to show up on the next jobs too... another thing i did saw, the next incremental run is also much more data then the other incrementals.. but probably this is only because the full job runs longer then my normal schedule and hence this there is more data changed ...

[msobm]

I just discovered that a Sharepoint Backup (to local Disk) is nearly empty, but the available Files are different when using different snapshots in Time.
Also, it seems to only affect our local Storage, the Backup to Wasabi seems to have all files (as far as I can tell).
Both Repos are Snapshot based.
There are exclusions set on both Jobs with PowerShell (same exclusion).
I created a case and keep you posted. Case # 07589291

[FrancWest]

Case # 07514638

We had the same issue with missing files and folders for OneDrive backups. Support didn’t find any cause and advised us to recreate the backup job. We’ve ended up loosing all our previous backups. This week we received an email about v8.1 that the missing files and folders is a bug and is fixed in version 8.0.4.29.

https://www.veeam.com/kb4704

Why doesn’t v8.1 do an automatic full sync on first run to fix the issue by itself instead of relying on the customer to do this manually using PowerShell?

Why isn’t Veeam giving this serious bug much more attention? Many people might be affected by this bug without even knowing they are. I only noticed it by viewing my own OneDrive backups which showed way less files than that where actually present in my OneDrive.

[FrancWest]

Also, I’ve ran the full backup command mentioned in the KB article, but when testing the repository afterwards it still reports we have items which don’t have a parent. I assume this is because older restore-points still have the issue, but does this mean we we’ll see this error indefinitely?

With this issue I’ve lost confidence in the reliability of the Backup for Microsoft 365 product.