Deleting from a full Linux Immutable Repo

[JamesMcG]

I have an immutable linux repo that's full. I'm going to tune retention etc to fix it up, however how do I clean up the now expired immutable backups and increments? The jobs are just failing rather than rotating, and attempt at manual deletion from the Veeam console is giving "Failed to delete backup Error: No space left on device POSIX: Failed to create or open file [/mnt/md0/backups/ Failed to invoke rpc command".

Naturally SSH is disabled on the repo, so I'd rather not travel 1hr into the office to the console just for a simple clean up.

[PetrM]

Hi James,

Unfortunately, we cannot effectively troubleshoot technical issues over the forum posts so please open a support case, upload debug logs as per this instruction to let our support engineers to investigate this issue. Also, please share a support case ID with us, posts about technical issues without case ID will be eventually deleted by forum moderator.

Thanks!

[JamesMcG]

Ok thanks, was hoping maybe another user knew how to handle this. Case opened Support ID: #01953814

[mweissen13]

Isn't the whole point of an "Immutable Repository" that it's not possible to do this without local root access?

[PetrM]

Hi Michael,

It's a very good observation! For sure, the main idea is to prevent deletion and modification of immutable backup files without having root access. However, it does not look like that the provided error message is related to Immutability itself but most probably appears due to the lack of space. Also, if I'm getting right James's message, the immutability period has already been expired for some backups. Let's wait for a workaround from the support team.

Thanks!

[micoolpaul]

I take it you don’t have any form of IPMI/out of band console access?

[JamesMcG]

No, there's no out of band on this setup as that would just be another vector for attack. As for the point being not possible to delete, sure but these are 'expired' files.

[micoolpaul]

Thought that’d be your answer (and I’m glad it is from a security perspective) but was just a straw clutch to avoid the travel!

[LickABrick]

Correct me if I'm wrong, but if the repo gets its time from a NTP server and DNS from a DNS server you control you can setup your own NTP server and temporarily change the DNS record for the NTP server configured on your repo.
Then on the NTP server change the current time to sometime in the future where the immutability has expired. I believe you would be able to delete the restore points.

But if this works you should also consider changing the DNS and NTP server on the repo

[micoolpaul]

@LickABrick yep that would work if that’s the case, but that’s also why it’s recommended to not use NTP and rely on a hardware clock instead!

[hexadecimal]

I’m in the same boat as the OP. I was going to delete the expired files manually and then rescan the repo. The challenge will be if I have synthetic backup enabled. Might need a whole new active full.

[micoolpaul]

@hexadecimal, provided it’s not your active backup chain you will be fine to continue as normal once your expired backups are purged.

[JamesMcG]

Figured I'd pop back in to explain my resolution. One thing I didn't mention was that this was a new setup, with 3 sites copying to this repo. I set my retention too long to fit it all in it seems! So I reduce the retention on all jobs, then waited for the existing sets to expire. I re-enable my jobs slowly and let them clean themselves up. I'm not sure why cleanup wasn't possible till the entire set was expired, however all is running well now!

[hexadecimal]

@hexadecimal, provided it’s not your active backup chain you will be fine to continue as normal once your expired backups are purged.

Thank you and this was indeed the case. In the properties of the backup set, while the backup was ongoing the items I'd deleted were greyed and showed a red "x". At the end of the backup job, they were gone and the remaining were fine. The set that had expired was from a chain almost 5 weeks ago. I'll ultimately be expanding the repo however I'm not terribly concerned as it's just one part of our rather elaborate backup architecture.

As had been noted, going through using root was not ideal however it was needed in this case. Thanks Micoopaul!