Files missing from incremental backups

[sooperfly]

Good morning,
Firstly, we have an existing case. #06036422
We have an issue with a client agent backup on a Windows Server Essentials 2016 install. During a routine file recovery, the admin was unable to recover some files from a specific restore point.
We investigated and confirmed that a few specific folders which had been created some weeks back, contained files. Some files were missing. In all of the restore points, neither the folder nor the files were present. This led to more testing, us creating files and folders, none of which are showing up.
Veeam Support noticed that this was a forever incremental with no full scheduled (config error). When we perform a full back up, then and only then do all new files show up in restore points however, any subsequent file/folder changes during the following incremental continue to fail to appear when attempting to restore. It has been a few weeks now and client is getting antsy about his backups. Something worth noting is that we see this issue for two backup jobs on this server.
One onsite to a local NAS and another offsite to our VCSP Repository. We also don't see this issue on another server which is backed up at this client.
My instinct tells me there is something amiss in the OS, but I am not sure exactly what it could be. (NTFS or some partition issue? Maybe a missing server update?) If anyone has dealt with similar issues in the past, I would greatly appreciate any insight you may be able to provide.

[sooperfly]

Another log discovered by Veeam support pointed to an error in the MFT. An offline checkdsk of the volume with fix enabled has led to an incremental which appears to be working.

[BackupBytesTim]

Good to see you got it fixed, I was hoping when you first posted that it'd turn out to be something more Veeam specific. We have a similar issue, but with a Mac, so unfortunately the fix doesn't really apply here. So far all we've gotten is that the problem started with the most recent major update from version 1 to version 2 of the Mac agent, files that were meant to be included in the backup simply were not getting included, despite their parent folder being included and there being no errors or warning messages, so we're thinking downgrading back to version 1 is the only solution for now. Of course we only noticed a problem when the customer emailed because couldn't figure out how to get data recovered from the folder that's empty in the backup, so there's no recent backup of the customer's missing data, needless to say they're unhappy.

Did your issue by chance start with the latest Windows agent upgrade from version 5 to version 6?

[sooperfly]

Sorry, I forgot to respond to this.
We don't know if the issue started with the agent update as we tend to keep them pretty up to date. I do believe we performed the latest update a troubleshooting measure without result.
From what we can tell, the MFT was broken in such a way that the agent couldn't track changed block/files. So you may investigate similar tracks related to MacOS file system. fsck and the like perhaps?

[BackupBytesTim]

Possibly, though my understanding is that on macOS Veeam will download the list of files and modified dates from the repository each time the backup runs to compare against the local filesystem to find what's been changed, such that it doesn't keep any local information or rely on any information kept by the OS locally. For our case downgrading the agent version seems to be a temporary workaround and Veeam is looking into it further to determine what the problem is.

We're also troubleshooting issues where the data Veeam is downloading from the repository to compare those file changes seems to be a thousand times bigger than it needs to be, downloading nearly twice the size of the previous incremental backup file is not uncommon. So we have a few different issues that may or may not be related.

[BackupBytesTim]

Update on the missing files, Veeam has identified an issue, but didn't go into detail on what the issue is, and is working on a patch to be released with the next agent update, in the meantime we're just keeping with the previous agent version. Again, on macOS, so might not be related to your issue. They did confirm the agent downloads the files list from the repository each time to compare changes so it's unlikely to be a local disk issue that broke the change tracking.