A problem occurred during setting the immutable flag: repository time shift detected.

[rodrigormoraes]

Dear all,

I have the following error message for the backup of a Windows workstation with storage via the immutable Ubuntu Server Linux repository.

I have already followed KB4482 by restarting the servers, I have already removed the immutability of the folder and deleted it from the repository, and I have run the job again (full), however, the same warning remains every time the job is executed.

Any other tips to follow?

11/5/2024 12:05:39 PM :: A problem occurred during setting the immutable flag: repository time shift detected, immutability flag cannot be set. Please refer to KB4482 for more details

[PetrM]

Hello,

It is not a known issue and I don't think that we can find out the root cause by guessing, without debug logs analysis or a remote session. Please contact our support team and provide a support case ID as requested when you make a post about a technical issue. All posts about technical issues without case ID will be eventually deleted by the forum moderator.

You can upload debug logs for our support as per the instructions in this KB.

Thanks!

[efd121]

I had one of my hardened repo's that was down over the weekend and after it was repaired I'm getting the time drift errors. When I attempted to follow KB4484 I keep getting denied when running the sudo chattr -i /etc/veeam/immureposvc/retainLock command. The error is Sorry, user <VHRUSER> is not allowed to execute '/usr/bin/chattr -i /etc/veeam/immureposvc/retainlock' as root on <SERVER>
<VHRUSER> is the user I created when configuring the VHR. I ran the command multiple times so I'm sure its not a password issue. Is there another option besides re-deploying the OS and preserving the data drives?
Dave

[d.artzen]

Hi Dave,
you normally have two users on a VHR (for example veeamadmin and veeamuser), one (veeamadmin) with sudo rights and a second one (veeamuser) that is used by veeam, which does not have that right (the names could be different in your environment, as you configure them yourself). Are you sure you are using the correct user? You should be using the one with the sudo rights.

[efd121]

I dont recall adding a second user to the system during the installation. This repo was built on the older Ubuntu hardened ISO.

I looked at my documentation and I'm using the username from the Profile setup screen. I assume that is the root user that I would use to run sudo?

The /home directory only has the lost+found and directory with the name of my user.

[d.artzen]

Ah, OK I wasn't aware that you used the older ISO from Veeam. Unfortunately I have no experience with that, we build our hardened repo with Ubuntu ourselves since the ISO did not exist back then. I used one of the guides from the community and there you created two users, one with sudo rights during the installation and a second one you created that Veeam would use and then was manually removed from the sudoers group, since that user should not have sudo rights for security reasons. I think the current ISO does it the same way.

[HannesK]

Hello,
for the old Ubuntu ISO, the easiest way is probably going into single user mode and deleting the file https://www.veeam.com/blog/ubuntu-linux ... -user.html

Best regards
Hannes

[efd121]

It appears that the option to boot into single-user mode was removed during the hardening from the ISO.

I suspect my best option is to reload the OS, preserve the data drives. I'm also considering putting the extents into maintenance mode and evacuating the backups. This would not be a quick option, they show about 150TB in use due to synthetic fulls, this would fill the remaining extents to a critical level.

[HannesK]

Hello,
hmm, then there is another alternative: use live boot from the new Hardened Repository ISO 2.0

after you logged in and changed the default vhradmin / vhradmin password, you should see something like this with "lsblk". /dev/sda8 should be the "root" file system.


get root with

Code: Select all

sudo su

mount the root file system to /mnt with

Code: Select all

mount /dev/sda8 /mnt/

remove the immutability flag

Code: Select all

chattr -i /mnt/etc/veeam/immureposvc/retainlock

remove the retainlock file

Code: Select all

rm /mnt/etc/veeam/immureposvc/retainlock

Best regards
Hannes

[efd121]

Thanks, I will give this a try on one of my test repos. I kept a few of my older/smaller repos to test with.

I ended up reloading the OS on my production repo and was able to get it back online

Dave

[MariusN]

Hello,
hmm, then there is another alternative: use live boot from the new Hardened Repository ISO 2.0

after you logged in and changed the default vhradmin / vhradmin password, you should see something like this with "lsblk". /dev/sda8 should be the "root" file system.


get root with

Code: Select all

sudo su

mount the root file system to /mnt with

Code: Select all

mount /dev/sda8 /mnt/

remove the immutability flag

Code: Select all

chattr -i /mnt/etc/veeam/immureposvc/retainlock

remove the retainlock file

Code: Select all

rm /mnt/etc/veeam/immureposvc/retainlock

Best regards
Hannes

Am i missing something or is this a complete tutorial how to remove immutable flags on backups? I am just wondering why this information is released officially in this forum - isn't this is a perfect guide for attackers?

[HannesK]

Hello,
yes, it's public information and documented multiple times already because this is how operating systems work. No need to hide anything and it should hopefully raise awareness

There is no way to protect against root and/or from anyone with physical access to the system and we cover this also in the FAQ. Section "Q: What if an attacker has access to root credentials and an ability to logon locally or remotely to the Hardened Repository?" That also applies to "live boot" because anyone can get themselves root access by mounting disks into a controlled OS environment. Therefore, disabling remote access and limiting physical access is the key.

For appliances: ask the appliance vendor what happens if you accidentally set 99 years immutability... will they force you to live with this and buy new hardware or will they help you to delete the data (whether "do it yourself" as Veeam does or via "support / professional service" like they usually do).

For cloud object storage: same story, cancel the account / subscription / whatever the term of the cloud provider is and they will purge all your "immutable" data after a short period of time and will certainly not wait for 99 years.

Best regards
Hannes

[d.artzen]

I agree with Hannes. If someone, that is not authorized, has physical access to your servers (and the Hardened Repo should have SSH disabled) you have already lost. Nothing can protect you against that, aside from physical access control. This is the main reason why the VHR should be a physical machine with internal disks or DAS and not a VM or used with external storage, it reduces the ways an attacker can interact with it. Also the reason why the Veeam-User on the VHR should never have sudo rights.

Also like Hannes already wrote, this is not some secret Veeam mechanism, but simply the usage of OS features. Even if Veeam did not document it, it would not be hard to google the way immutability is handled in the Linux distribution used und how it can be removed. A user with "root" rights always can do everything with the system, and on a live boot media you are always "root".

History has shown us that "security through obscurity" does not work, so I personally like it, that such things are discussed, so that one knows where the limits are.