Major issue with incomplete SharePoint backups, files are missing and lost

[rairiau-omr]

Hello,

This topic to inform you that we encounter a major issue with Veeam Backup for Office 365 backups for SharePoint, unable to restore some files, the backups were incomplete and now files are definitively lost.
You have to manually run Full Sync backup on jobs with last patch deployed to have all files in backups, but too late in our case.

Veeam support case #05160244.

Some details from support :
The issue at hand is a known issue that was found on previous version of VBO365 which affects all backup repositories containing SharePoint/OneDrive/Teams data.
The root cause for the missing data was due to incorrect representation of the changes from the SharePoint API side.
Veeam RnD team performed an investigation and found that sometimes the SharePoint API mechanism of tracking changes did not track the changes inside the Child files or Folders inside the Site`s list.
For example, a child folder is created within the folder that is already backed up, or a folder appears in the root of the list - but the incremental backup is not backing up these folders because they are not indicated as a change by the change tracking mechanism.
As a result, some folders and subfolders were missing from the backup, even after the incremental backup job runs.
This change tracking information received from the SharePoint API is then stored inside Veeam Backup Repository for the incremental runs.

The Veeam RnD team developed a fix that improved the way how VBO365 works with change tracking of the SharePoint API, but the fix only changes the way how VBO365 requests and uses the Change Tracking information with SharePoint API, but it does not fix the already affected backup repositories.

To resolve the issue inside Backup Repositories we need to refresh the change tracking information which is stored inside the Veeam Backup repository by running the Full Sync backup for each backup job that is configured to use the affected repository.

[karsten123]

really? Is there official documentation for this bug?

[rairiau-omr]

Yes it's real in our case, as well the backup job concerned all the SharePoint sites, and there was successful, some files are missing in backups.
Our client have an issue on SharePoint storage due to Onedrive sync bug, and try to restore files, but some of these files were not backup-ed by Veeam vbo due to issue explained, and so, there are lost definitively.

The last patch, KB4222, explained that there was an issue with repository on object storage, but in our case we use Windows Server repository (only one vbo server with all roles).
Actually Veeam team is re-checking the situation internally with QA/RND and Product Management teams to define what is needed for next steps.

[nielsengelen]

Hi,

Correct, this case is under active investigation together with R&D and we hope to provide an update as soon as we get more details.

[karsten123]

Oh my god. That sounds very bad.
Niels, please give us an update asap.
Tia

[nielsengelen]

This case is still being investigated as we speak. Once we get to a resolution, we can update this topic.

[MarkG-ZA]

Hi all, any feedback regarding this case? We are also having a similar issue with a client, the sites backup without errors but some folders and files appear to be missing when browsing for them in Veeam Explorer.

Regards
Mark

[Polina]

Hi Mark,

Romain's case turned out to be very specific and it's still in troubleshooting.
Even if you see similar symptoms, the root cause may differ. Please create a support case on your issue and post your case ID here for the follow-up.

Thanks!

[MarkG-ZA]

Good day Polina, thank you for the information.

We have logged the case. Site not Displaying in Explorer - Case #05240929

Thanks!