Major issue with incomplete SharePoint backups, files are missing and lost

[msobm]

We have multiple Backup-Servers affected by this Bug after checking some Servers.
Veeam should (have) communicated this Bug proactively to partners and registered Accounts.

[DaStivi]

I just guess these two bugs are somehow related and V8 upgrade made it even worse looks like ...

Trust is to the product is pretty gone... Not sure who to blame more... Veeam having this issues or Microsoft causing it with all their changes they introduce all the time , making vendors like Veeam a hard time keeping their product up2date, functional and verify what data you can get... Probably one of the biggest issues that there is no real backup API , like snapshots or vss available

[dasfliege]

I'm the one who rehabilitated this thread after some time as two of our customers became aware that data is missing in the yearly exports we perform for them. Was on vacation for a month now, but worked on the case (07527751) with veeam support during that time.
What we've found out for our two specific issues is the following:

Data missing in Sharepoint Online backup:
- The data was actually backed up but not displayed in explorer. I have received a fix which now displays the data in explorer but in a really random structure. It's pretty flat and most of the folder hirarchy is gone. But as far as i understood, there will be a fix for that in a future version.
- Performing full sync fixed the issue for all restorepoints AFTER the full sync.
- Performing full sync copied a lot more data then previous incrementals. Like most of you, i have no idea why, because it seems data was backed up before already.

Data missing in Teams Files explorer:
- Data was actually backed up but not displayed in explorer. Data was visible in Sharepoint eplorer though, but of course not in the original structure as you would expect it to see in Teams Explorer. So for a customer performing self restore, this is not a very convenient way and also we can't do our yearly exports of Teams data if it is missing from Teams explorer.
- Performing full sync fixed the issue for all restorepoints AFTER the full sync.
- The issue is (according to support) related to a known limitation described in the following article. I was absoutely not aware of this limitation and i guess most of you aren't. I have been told that performing a full sync prior to our yearly exports "should" fix this issue and we "should" be secure that all saved data is beeing exported.

"If a user never opened the Files tab of a team channel in Microsoft Teams before data backup, files from this tab are not displayed in Veeam Explorer for Microsoft Teams."
https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=80


However, there were a lot of "shoulds" in the conversation with support and i lost quite a lot of trust into the product. Also response from product managers in this threat is expremely slow and not very helpful yet, especially for service providers who have to guarantee a reliable backup solution for their customers. I asked for an official statement by veeam reprentatives on the 17th of december. Nothing yet....

[edh]

Florin, just based in your experience with your cases. Its recomended perform a sync of all jobs?

[DaStivi]

Maybe scheduling (scripting) a full sync once a week isn't a bad idea? Biggest issue would be large user counts .... Where this full job is running even longer then normal...

I haven't run this for my biggest customers but I have Todo soon...

[dasfliege]

@eth
There's no way around it if you want to be mostly sure that all data is restorable in a convenient way.
We haven't performed it yet for all customers as we charge our customers for license count AND storage volume. As long as Veeam does not provide a final statement about why a full sync transfers more data then a incremental, even though the data was already backed up before, i am not confident to perform it for all our customers. Our biggest customer has 50TB of backup storage. Don't know how long the job will run for them.

[DaStivi]

a "smaller" customer from us, with only 30TB, 1 teams/SharePoint but 4 User jobs (distributed through dynamicgroups) where i ran full-sync now, resulted in 3x170GB transferred and 1x 230GB in transferred data for the User jobs! the teams/SharePoint resulted in only 10GB

[edh]

Only for the people who are reading this thread. Do yourself a favor and launch the works with a full . In our first clients we have seen "increments" of 500GB and it only takes 40% of the complete synchronization.Meaning that a lot of files where missed by some random bug whatever caused by MS API or Veeam.

[aeph]

@edh
Is this only relevant for organizations that were already onboarded from https://www.veeam.com/kb4447 before the patch? So customers who were only later onboarded are not affected?

[edh]

I really dont know this customer got 14154 objects, the only thing we know that its downloading data.... that i suppose is missed someway. Its a pain to do a full sync in timing but, i prefer waste time now ...

We created this small powershell to run max 12 full jobs sync currently to solve this in some way to all customers

Code: Select all

$MaxJobs = 12
$PendingJobs = Get-VBOJob
$totalJobs   = $PendingJobs.Count

Write-Host "Total de trabajos pendientes: $totalJobs`r`n"

for ($i = 0; $i -lt $totalJobs; $i++) {
    $job = $PendingJobs[$i]

    # Mientras haya 4 o más trabajos corriendo, esperamos
    while ((Get-VBOJobSession -Status Running).count -ge $MaxJobs) {
        Start-Sleep -Seconds 300
    }

    # Calculamos cuántos trabajos quedan después de éste
    $quedan = $totalJobs - ($i + 1)

    Write-Host "Iniciando trabajo [$($i + 1)] de [$totalJobs]: $($job.Name). Quedan $quedan trabajos por lanzar..."

    # Arrancamos el trabajo
    $job | Start-VBOJob -RunAsync  -full
}

[DaStivi]

I've run multiple full-sync on customers now... had a support session with veeam together... but overall, as there is no real hint of missing data and its hard to "prove" something is going wrong!

its also very hard to tell because you need to differentiate between tenants/Job that has been added before there was the code change that should have fix this particular issue back in 7.0version...
so in theory if a customer has been created back then, and just in case the particular issue had happend (quickly added/remove and readded files to sharepoint/onedrive) you just need todo the full-sync to maintain this data.. so in theory it still could be today that there is some missing data... and is fixed by running the full-sync as suggested in the KB.

after the fix (fix was available around may 2023, of course it still depends when the fix was installed!) is in place it shouldn't be that issue any longer...

so what could cause the "huge" data transfer now?
its really hard to tell but these full jobs also has way longer run times, the amount that gets transferred could be totally normal. in our case we do incremental backups each 2h for our customers... and for example if i have 6 incremental runs, that causes like 10gig of change rate together in that 12hours... and if the full sync for example also runs this 12hours, it should an will produce the same amount of 10g change rate! you just can't compare it to a single incremental run, you need to take the runtime into this consideration too! and this also explains why a subsequent incremental run also might transfer way more data, because the changed amount of data in the time of "no-backup" was just longer!

additionally, the amount of data that's really changing depends on the user workload too! so maybe a fair comparison between 12hours of incremental runs and like ah 12h full sync could be done on the weekends where less or almost non changes are made to office 365!

another good hint could be the data-usage information of microsoft 365 it self! https://admin.cloud.microsoft/#/reportsUsage
still somewhat fiddley ... but the change rates displayed there should also reflect the change rates of the vb365 jobs...

[ReKe]

Looks like other vendors has identical issues.

Here from the latest release notes from Synology Active Backup for M365.

Fixed an issue where only first-layer folders within copied SharePoint folders could be backed up.

[m.novelli]

After seeing this scary post I've started updating all customers to v8.1 and running a full sync

Marco

[m.novelli]

Next version 8.2 should have a flag during setup "due to bla bla bla issue we recommend tu run a full sync after upgrade, please click this flag" and then VBO do the magic

Marco