btrfs balance Leads to Large Incremental Backups

[netyard-mh]

Hey everyone,

I'm currently having trouble with the following setup:

- Veeam Agent for Linux installed on an Ubuntu 20.04 VM
- Samba mount /mnt/Nextcloud to a bare metal Ubuntu 22.04.1
- File level backup (directly from live file system) of /mnt/Nextcloud
- The file system on the bare metal Ubuntu is BTRFS

Prior to extending said BTRFS volume I ran a

Code: Select all

btrfs balance start

.
Then I extended it by 4TB.

Now Veeam seems to backup everything in that mounted Samba folder again and I don't get why.
The data itself hasn't changed, just the inode the file resides in.
mtime and ctime are also still the same:

Live file system

Code: Select all

stat /mnt/Nextcloud-Data-SMB-Mount/USER-ID/files/_Daten/VHS Kassetten/Band\ 10\ Teil\ 3\ 1997.07.30\ Nähen.mp4

  Size: 47351603        Blocks: 92488      IO Block: 1048576 regular file
Device: 37h/55d Inode: 145530      Links: 1
Access: (0770/-rwxrwx---)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2022-09-07 08:13:28.956192500 +0200
Modify: 2022-03-07 22:55:45.000000000 +0100
Change: 2022-03-07 22:55:45.000000000 +0100
 Birth: -

From the latest incremental backup before the btrfs balance

Code: Select all

stat /tmp/Veeam.Mount.FS.efdd3afb-3602-4318-94ea-f62dbe339d5a/FileLevelBackup_0/mnt/Nextcloud-Data-SMB-Mount/USER-ID/files/_Daten/VHS\ Kassetten/Band\ 10\ Teil\ 3\ 1997.07.30\ Nähen.mp4

  Size: 47351603        Blocks: 92496      IO Block: 4096   regular file
Device: 706h/1798d      Inode: 1276        Links: 1
Access: (0770/-rwxrwx---)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2022-07-07 01:47:49.000000000 +0200
Modify: 2022-03-07 22:55:45.000000000 +0100
Change: 2022-03-07 22:55:45.000000000 +0100
 Birth: 1970-01-01 01:00:00.000000000 +0100

A

Code: Select all

cmp

on the two files shows no differences either.

What can I do to prevent everything from being backed up again?
We are talking about several TB and I cannot have an incremental backup that large.
As I do a remote Copy Job, too I also cannot just start a new backup chain as otherwise Veeam will transfer the new chain via WAN to a remote site, too.
That would take weeks.

Cheers,
Mark

PS: The only other change I made was to enable FIPS compliance as I needed a pre-script on another job and the SSH connection was failing.
However, I don't see any relation to this here.

[HannesK]

Hello,
and welcome to the forums.

Agree, file-based backup should not be influenced by filesystem extension. Note: also changes to permissions or extended attributes result in backing up the file again.

Where / how did you enable FIPS compliance? I guess on Ubuntu 20.04, because for 22.04 there is no FIPS compliance yet. Agree, I also cannot see any reason for it to fail... I'm just guessing (support might find something in the logs, that might be worth a try).

The only other test scenario I have would be trying NAS backup and see whether this has the same result.

Best regards,
Hannes

[netyard-mh]

Thanks, Hannes.

I did actually enable FIPS compliance in Veeam Backup and Replication (there I manage the Linux backup agents).
-> General Options -> Security -> Enable FIPS-compliant operation mode

Thanks for mentioning the NAS backups.
However, shortly after it was released in v10 I did try it for our customers (I work in an IT company) and it had some severe flaws that were unacceptable back then.
For example when the backup failed for whatever reason (network hickup) after backing up say 1TB, the system would completely forget that it had already backed up data and would backup everything again. Even when it finished successfully and did a health check, it would not reduce the size again despite the fact that there was only one working backup snapshot.
I opened several support tickets (#04738596, #04751253 & #04776937) back then, but all of them were not answered and simply closed. So I don't really want to give that a try again. Maybe if Veeam confirms that this bug is gone. But even then I don't like the blob-file style backup where I cannot simply remove increments when there is an issue.

Cheers, Mark

[HannesK]

Hello,
ah okay... I thought FIPS on Ubuntu.

Customers without support contract have "best effort" support. That's the reason, why the cases were closed after nobody had time on "best effort". Logs from that time (April 2021) are deleted due to compliance reasons. So I cannot ask anyone to investigate it.

Best regards,
Hannes

[netyard-mh]

Thanks, Hannes.
I have just disabled FIPS-compliant operation mode again, but it doesn't make any difference.

Is there any way to find out why exactly Veeam is backing up the files and what has changed?
As mentioned in my first post I really don't get it.
All the relevant (at least visible to me) metadata seems to be the same and being a file level backup the actual inode or location within the file system shouldn't matter, right?

I have opened another free support case:
#05616004

We do have several paid subscriptions for us and our customers, but as this really is my personal free setup, I'd like to keep it separate.
Still, I'm guessing this is some kind of bug or at least Veeam uses some way to compare the files that is not documented.

[HannesK]

Hello,
yes, FIPS compliance in the product should not make any difference.

Log files are missing in the case. Please add full VBR log bundle plus agent logs https://helpcenter.veeam.com/docs/agent ... tml?ver=50

Then I can ask support whether they might someone to have a look at it.

And just to be 100% sure: you use the latest version of everything? Otherwise I cannot ask anyone on a free case to check it.

Best regards,
Hannes

[netyard-mh]

Hey,

I have decided to just create a new backup and drive to my offsite location to seed its copy via HDD.
Everything else is too much effort for me and I cannot wait for support to fix this.

You may close the support case.
However, you may also want to bring this to the attention of the developers as this is clearly a bug of some sort.
When live file level backup is used the location of the files in the file system must not matter in my opinion.

Cheers,
Mark

[HannesK]

Hello,
I just tried to reproduce the issue you had and it works fine for me.

I mounted an SMB share running on BTRFS on Ubuntu 22.04. Then I did my first backup. Then an incremental backup. Then I enabled FIPS mode, did the rebalance and extended my BTRFS. Then I ran the next backup. No new full. As expected.

So it seems to be specific to your environment and not a general issue.

Best regards,
Hannes