Veeam service on repo not removing immutable attribute on older files

[joebranca]

I have had a case open Case #07148096 on this issue. So far I have had no useful suggestions, nor does this seem to be a known issue. Around mid to late January I updated our Veeam B&R environment from 12 to 12.1, and also had our Linux sysadmin reboot the Linux hardened repos to apply kernel updates.

It seems that since then, Veeam jobs will end in a failure status when attempting to apply the 42 restore point retention policy, because the oldest vbk and vib files in the backup chain are still immutable. This means that for some reason the 14 most recent backups immutable policy is not being enforced when it comes to *removing* the immutability attribute.

I have tried starting new backup chains w/ new fulls on different repositories, but the issue persists across all repositories. I have had to work with the sysadmin to get root access to manually remove immutability from older files using chattr -i, so that Veeam can apply some retention policies and our repo capacities are under control.

Does this problem sound familiar to anyone? I don't even know where the breakage has happened between the VBR server issuing the info on immutable policies, and the local lock files/metadata that is supposed to be tracking immutability expirations.

According to this white paper on pg 17-19:
https://contenthub.gmevents.ae/wp-conte ... ackups.pdf

the Veeam service which sets and unsets extended attributes veeamimmureposvsc runs as root. Presumably that service is operational in terms of having root access, since it can clearly set the immutable attribute on newly created files. But something is preventing it from removing the immutable attribute on older files. Or it is not properly reading the lock and file metadata to get updated info on immutability expiration?

[david.domask]

Hi @joebranca,

Thank you for your post here and sorry to hear that the issue is still on-going.

The case was recently escalated to our Advanced Support Team, and I can see that you recently uploaded fresh logs for their review. The behavior is not expected, so let's allow the Advanced Support Team a bit of time to review the newest logs and we will get to the bottom of this issue.

From my time in Support, I've seen something similar to this when the .lock files on the Linux Hardened Repository have somehow gotten unexpected owner/group, but let's allow the Advanced Support Team time to check this a bit more deeply. I also suspect as you do that the Immutable Service (or the lock files it works with) are having permission difficulties, and I'll comment the case as such.

[joebranca]

Thanks for the update. This has been quite the head scratcher!

[Virtuollie]

Any news on this one?
I just opened case #07299050 with the same issue...

[Mildur]

Hi Oliver

I can't see a solution yet in the other case.
Please keep working with your support engineer.

Best,
Fabian

[pieter.beel]

Hi,

We have the same issue after an update from 12 to 12.1
The flags can be reset manually as root but that is only a temp workaround.
Case # 07295148

Kr

[HannesK]

Hello,
just by chance... did anyone of you follow a Red Had blog post for installation and has 2775 permissions on the repository paths? SGID is not supported anymore for security reasons and the Red Hat blog post was updated to "chmod 700" some weeks ago.

Best regards,
Hannes

[zbe]

Is there anything new about this? I'm having the same problem on hardened linux repo on zfs. No SGID bit set.

[zbe]

Anyway, for the time being I just created (with help of AI) a script to compare both xattr and .lock file with current time and systemd service and timer to run it hourly. If it's of any help:

/usr/local/bin/veeam-unlock.sh (chown it to root and chmod 700):

Code: Select all

#!/bin/bash
set -euo pipefail

INSTANCE_ID="${1:-}"
if [[ -z "$INSTANCE_ID" ]]; then
    echo "Usage: $0 <instance-id>"
    exit 1
fi

# Reconstruct path from systemd instance (e.g., mnt-hdd-veeambackups → /mnt/hdd/veeambackups)
ROOT_REPO="/${INSTANCE_ID//-//}"

NOW_EPOCH=$(date +%s)
TMP_XML="$(mktemp /tmp/veeam-lockdata.XXXXXX.xml)"

find "$ROOT_REPO" -type f -name '.veeam.*.lock' ! -path '*/.zfs/*' | while read -r LOCK_FILE; do
    REPO_DIR=$(dirname "$LOCK_FILE")
    echo "Processing repo: $REPO_DIR"

    rm -f "$TMP_XML"
    /opt/veeam/transport/veeamimmureposvc --convert yes -i "$LOCK_FILE" -o "$TMP_XML" || {
        echo "Failed to convert $LOCK_FILE"
        continue
    }

    declare -A lock_times
    while read -r line; do
        name=$(echo "$line" | grep -oP 'Name="\K[^"]+')
        until=$(echo "$line" | grep -oP 'ImmutableTillUtc="\K[^"]+')
        [[ -n "$name" && -n "$until" ]] && lock_times["$name"]="$until"
    done < <(grep -oP '<File [^>]+' "$TMP_XML")

    for filepath in "$REPO_DIR"/*; do
        [[ -f "$filepath" ]] || continue
        filename=$(basename "$filepath")

        if [[ -z "${lock_times[$filename]:-}" ]]; then
            echo "  [SKIP] $filename not listed in .lock"
            continue
        fi

        lock_until="${lock_times[$filename]}"
        lock_epoch=$(date -d "${lock_until//T/ }" +%s 2>/dev/null)

        xattr_until=$(getfattr -n user.immutable.until --only-values "$filepath" 2>/dev/null)
        if [[ -n "$xattr_until" ]]; then
            xattr_epoch=$(date -d "$xattr_until" +%s 2>/dev/null)
        else
            xattr_epoch=0
        fi

#        echo "  [INFO] $filename lock=$lock_epoch xattr=$xattr_epoch now=$NOW_EPOCH"

        if (( NOW_EPOCH > lock_epoch && NOW_EPOCH > xattr_epoch )); then
            if lsattr "$filepath" | grep -q '\-i\-'; then
                echo "  [UNLOCK] Removing +i from $filepath"
                chattr -i "$filepath" || echo "  [ERROR] Failed to remove +i"
#            else
#                echo "  [SKIP] $filepath not marked immutable"
            fi
#        else
#            echo "  [LOCKED] $filepath still under retention"
        fi
    done

    unset lock_times
done

rm -f "$TMP_XML"

veeam-unlock@.service:

Code: Select all

[Unit]
Description=Veeam Unlock Script for %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/veeam-unlock.sh %i
Nice=10

veeam-unlock@.timer:

Code: Select all

[Unit]
Description=Run Veeam Unlock Hourly for %i

[Timer]
OnCalendar=hourly
Persistent=true
Unit=veeam-unlock@%i.service

[Install]
WantedBy=timers.target

You enable it with repo path, e.g.

Code: Select all

systemctl enable --now veeam-unlock@mnt-veeambackups.timer

if you have repo under /mnt/veeambackups.

[HannesK]

Hello,
we are not aware of issues without SGID. Please open a support case to get it investigated if you have an issue.

Best regards
Hannes

[zbe]

Hey Hannes,
I already opened it but it was closed due to "low support staff availability". It was #07669424.
I'd provide logs if I knew where anything relevant was saved (if it even is in regards to xattr).

I will get errors/warnings like this:

Code: Select all

Failed to pre-process the job Error: boost::filesystem::remove: Operation not permitted: "/mnt/hdd/veeambackups/hpe-vm-mgmt/ADMIN-VSRV-MANAGEMENT.fa238d98-541c-4233-b1f3D2025-05-10T220530_DA83.vbk" Agent failed to process method {FileSystem.FileRemove}.
Failed to delete 'ADMIN-VSRV-MANAGEMENT.fa238d98-541c-4233-b1f3D2025-05-10T220530_DA83.vbk' per retention policy
/mnt/hdd/veeambackups/hpe-vm-mgmt/ADMIN-VSRV-MANAGEMENT.fa238d98-541c-4233-b1f3D2025-05-10T220530_DA83.vbk
Task failed. Error: boost::filesystem::remove: Operation not permitted: "/mnt/hdd/veeambackups/hpe-vm-mgmt/ADMIN-VSRV-MANAGEMENT.fa238d98-541c-4233-b1f3D2025-05-10T220530_DA83.vbk" Agent failed to process method {FileSystem.FileRemove}.
Processing finished with errors at 24. 05. 2025 22:15:35

but backup is then removed from veeam database and everything works fine, except that the file still sits on the disk until I manually remove it. Will see in a week if everything works as supposed with this script.

[HannesK]

Hello,
yes, getting support without license is a lottery. There is no guarantee that unpaid users get support.

I saw that you mentioned 3 days retention with two synthetic fulls per week in the support case. That cannot work with 7 days immutability minimum enforced. What was the result after you changed the backup job retention to 7 days?

I saw you mentioning ZFS on TrueNAS and Proxmox. How is that meant? Do you run VMs on Proxmox and then do passthrough of a ZFS file system or what do you do there? Same question for Truenas... do you run a VM on TrueNAS or do you do iSCSI and put ZFS on it?

Best regards
Hannes

[zbe]

Hey Hannes,
sorry for delayed response.

Yes, I understand that 3 days retention with 2 fulls per week and 7 days of immutability "can't work", but I would expect that after those 7 days pass, the +i flag gets removed and Veeam cleans those files. Anyway, even after setting retention to 7 days, the problem was/is that immutability flag just doesn't get removed. No idea why. And those backup files, although removed from Veeam (e.g. when trying to restore guest files those backups won't be available), they still persist on disk.

I run VMs on Hyper-V, one of them with Veeam. Proxmox and TrueNAS are just used as two linux repositories added as Linux hardened repos.(On Proxmox it's via LXC, on TrueNAS veeam services are installed directly on.)

[HannesK]

Hello,
with 7 days retention and one weekly synthetic full (the default setting), the backups can be removed only after 13 days because there are dependencies on the full before that (see https://www.veeam.com/kb1932 animation)

My suggestion is the following: try the default settings on a supported system (plain VM for testing with XFS recommended)
- 7 days retention
- 7 days immutability
- one weekly synthetic full.

For the containers & TrueNAS: it feels like LXC should work from what you wrote. I remember LXC requires privileged containers, because otherwise chattr +i / -i doesn't work. But it feels like you have not seen any issues around this. Running Hardened Repository directly on TrueNAS seems like you did some hacks because I remember that fails because TrueNAS uses ZSH instead of bash and installation fails with a 64bit error.

Best regards
Hannes

[zbe]

Yes, I understand it takes additional time to remove backups — it's actually the reason I had 2 fulls per week with 3 days retention, so that after ~10 days, when one full and ~3 other backups are out of immutability period, they get removed. But since +i never gets removed, doesn't help. Had same problem with other backups with 7 day immutability and at least 14 days retention. So whatever sets chattr +i, doesn't remove it.

In regards to both repos: I'm using (privileged) LXC on Proxmox. On TrueNAS I'm running it directly, yes, and for that you need to run `install-dev-scripts` to make filesystem writable. Shell for specific (e.g. veeam) user can be set individually, so no problem setting bash.