Major issue with incomplete SharePoint backups, files are missing and lost

[rairiau-omr]

Hello,

This topic to inform you that we encounter a major issue with Veeam Backup for Office 365 backups for SharePoint, unable to restore some files, the backups were incomplete and now files are definitively lost.
You have to manually run Full Sync backup on jobs with last patch deployed to have all files in backups, but too late in our case.

Veeam support case #05160244.

Some details from support :
The issue at hand is a known issue that was found on previous version of VBO365 which affects all backup repositories containing SharePoint/OneDrive/Teams data.
The root cause for the missing data was due to incorrect representation of the changes from the SharePoint API side.
Veeam RnD team performed an investigation and found that sometimes the SharePoint API mechanism of tracking changes did not track the changes inside the Child files or Folders inside the Site`s list.
For example, a child folder is created within the folder that is already backed up, or a folder appears in the root of the list - but the incremental backup is not backing up these folders because they are not indicated as a change by the change tracking mechanism.
As a result, some folders and subfolders were missing from the backup, even after the incremental backup job runs.
This change tracking information received from the SharePoint API is then stored inside Veeam Backup Repository for the incremental runs.

The Veeam RnD team developed a fix that improved the way how VBO365 works with change tracking of the SharePoint API, but the fix only changes the way how VBO365 requests and uses the Change Tracking information with SharePoint API, but it does not fix the already affected backup repositories.

To resolve the issue inside Backup Repositories we need to refresh the change tracking information which is stored inside the Veeam Backup repository by running the Full Sync backup for each backup job that is configured to use the affected repository.

[karsten123]

really? Is there official documentation for this bug?

[rairiau-omr]

Yes it's real in our case, as well the backup job concerned all the SharePoint sites, and there was successful, some files are missing in backups.
Our client have an issue on SharePoint storage due to Onedrive sync bug, and try to restore files, but some of these files were not backup-ed by Veeam vbo due to issue explained, and so, there are lost definitively.

The last patch, KB4222, explained that there was an issue with repository on object storage, but in our case we use Windows Server repository (only one vbo server with all roles).
Actually Veeam team is re-checking the situation internally with QA/RND and Product Management teams to define what is needed for next steps.

[nielsengelen]

Hi,

Correct, this case is under active investigation together with R&D and we hope to provide an update as soon as we get more details.

[karsten123]

Oh my god. That sounds very bad.
Niels, please give us an update asap.
Tia

[nielsengelen]

This case is still being investigated as we speak. Once we get to a resolution, we can update this topic.

[MarkG-ZA]

Hi all, any feedback regarding this case? We are also having a similar issue with a client, the sites backup without errors but some folders and files appear to be missing when browsing for them in Veeam Explorer.

Regards
Mark

[Polina]

Hi Mark,

Romain's case turned out to be very specific and it's still in troubleshooting.
Even if you see similar symptoms, the root cause may differ. Please create a support case on your issue and post your case ID here for the follow-up.

Thanks!

[MarkG-ZA]

Good day Polina, thank you for the information.

We have logged the case. Site not Displaying in Explorer - Case #05240929

Thanks!

[dasfliege]

We've been informed by one of our customers that he's not able to see all folders inside his sharepoint backups. We've created a case (07527751) and according to the current state of investigation it seems like this data has never been backed up even though the jobs were successfull all the time. Veeam support pointed me towards the following kb which may could have lead to such a behavior: https://www.veeam.com/kb4447

As i see now, this issue has already been brought up quite some time ago. But there is still no mechanism in veeam to check if really all data has been saved which is selected for backup. We now have to explain to our customer that his data has never been saved correctly. That's unaccaptable behavior for a backup solution. I understand that issues like this can happen but there should at least be some kind of warning or something inside veeam, which reminds you to perform full backup of all jobs which could potentially be affected by this issue.