Backup copy job to external SMB no longer works for incrementals

[wires]

I was running the 11 version for some time with a portal USB HDD attached to the NAS that is being used as the main repository via SMB.

For 1/2 year, that infrastructure has been flawless. Two weeks ago I upgraded to 11.0.1.1261_20211211. Now, only the full backups to the SMB attached storage will succeed. Every incremental fails with this error on the dashboard "Failed to reload metadata bank. Declared and actual CRC are different for all bank snapshots."

I've tried different USB HDDs, deleted the copy job, deleted the BMR job that it pulls from, uninstalled from the BMR, did a re-install and full of the BMR, Nothing seems to fix this incremental failure.

As a test, I attached the USB HDD in passthrough as a repo, and that works fine (except Veeam B&R won't back up a VM using passthrough). It seems to be something with the SMB repositories not working.

Anyone else having this issue? I opened Case 05249815 for free product.

[HannesK]

Hello,
that sounds more like data corruption on hardware level (meaning the upgrade is a coincidence, but not the cause). NAS systems are well known as number one root cause in Veeam support for data corruption.

A new full backup does not rely on old data, so data corruption on old backups is irrelevant for the new backup.

Best regards,
Hannes

[wires]

The part that is confusing me is the NAS backups are fine (full and incremental), it's just the backup copy job. I have three portable USB HDD's used as rotating drives, connected to a Synology DS1019+ NAS mapped in SMB mode.

It doesn't matter which USB HDD I use. The full backup copy will succeed, and then every subsequent incremental will fail in the same metadata failure mode which happened the exact day I did the 11.0.1.1261_20211211 upgrade. If I delete the backup copy from the console, the next backup will be forced to full, which works, and then the next evening I will get my 3 failure emails that the incrementals did not succeed..

[soncscy]

Heya Lyle,

I don't know if this is your issue, but it's VERY similar to an issue I had with a client years ago, and the fault there was the NAS' handling of the NFTS driver. Exact same setup, exact same symptoms; Fulls worked, incremental did not.

Switching to XFS (and ext4 for a few drives) helped and they never had an issue afterwards.

I'd place the fault on your NAS OS. If you are in a position to wipe one of the drives and use a linux native filesystem, see if it works.

[wires]

Thanks for that insight. I did try EXT4, and that worked. The Synology DS1019 won't format NTFS, but it will R/W it. Its just super weird (could be a full red herring though) that it happened right after the upgrade. Might just be how Veeam verifies now breaks something that was hidden and broken prior.

I ended up moving the backup USB to a mounted drive on the BMR as a "Windows Direct Repo". My concern about EXT4 was around doing a BMR recovery on a Windows platform, so keeping it NTFS reduces that extra level of stress during an already stressful recovery scenario.

As a side note, I did encounter a bug with the Veeam console and using the USB HDD mounted that way. I ran into a permissions issue on the USB share, so when Veeam started the copy, it would fail with "Access Denied". Once I fixed the permission, every backup copy continued to fail with "access denied". I went to the backup repositories tab, saw that repo was tagged as "unavailable". I ran through the wizard for the repo, made sure it populated the size (if it did not, then Veeam would fail the copy due to permissions). But once that was done, the drive did not move from the "Unavailable" repo tab to the active tab. Only way to bring it online and allow the copy to proceed was to reboot the PC running Veeam console.

[soncscy]

Heya Lyle,

I'm glad to hear it worked. I still point the finger at the Synology NFTS handling; likely it's ntfs-3g but who knows. I am positive it's Synology here as my client had the exactly same thing, but this was many Veeam versions ago (I think like v9 update 2? It was very old).

As Applications can't make any decision on the file system driver, it has to be coming from the NAS box and the driver in my opinion.

I get your point on recoveries, so completely understood on not sticking with ext4; I would just avoid the Synology USB passthru altogether, it's really flaky from my experience for any real copy operations.

[Regnor]

While i can't speak for Synology, I know other cases where the ntfs-3g had problems with external media. For example if the file/folder structure was too big, it would just time out; so the read/writes never completed.