Some thoughts about immutable backups

[hushpuppies]

Hi folks,
after reading Gostev's digest today, I want to ask the community about their thoughts about how immutable immutable backups are and what they are really helping in case of emergency.
The usual definition of immutable as I understand is, that it's meant to be a backup which cannot be changed or deleted after it's done. So basically it means a WORM backup.
Ok, which kind of protection this offers? (besides the technical side of it, how to get it really undeletable). If it's done correctly you get a backup which reflects the status quo on a certain date in which everything is conserved as it is. This is plain wisdom but as I think is also the core of the problem, because it simulates a kind of safety which isn't there.
At least not when it should counter a real attack and not some "simple" infection via mail and stupid user clicking everything flashing.
In this "simple" case the cause and the problem solving are usually close by - you've got the infection, it scrambles your data, you buy a ticket to Hawaii and do the restore from remote while nursing your shock
So, but if somebody evil really wants to do some damage or do some real black mailing?
Such an evil genius would probably first gain access to some privileged account (e.g. the famous admin with jesus12345 as password), clone that account to something inconspicuous like John Smith.
Then he would use this account to place some time bomb which gets active in let's say 3 months (or later).
After the time bomb goes off your data gets scrambled, you buy a ticket to Hawaii and start your recovery processes only to realize that you can't do enough hula hoop to find a working backup without that infection - besides the fact that you need some IT forensics to find the source of the infection at all.
Sure, you might say "But I have backups dating back 6 months or even one year" - but how useful are these backups?
And in business today - depending on your industrial sector and how depending you are on actual live data like e-mail, messaging and so on, even a three day old backup would do a lot of harm.
Sure even a old backup is better than none, but the practical usefulness rapidly degrades with time and in an attack like the one above, I think there is no real defence against it.
What do you think?
Am I too pessimistic or do I miss something?

[grantemsley]

True, if your attacker is patient and waits for all/most of your backups to include their malware, you've got a problem that immutable backups doesn't fully solve. But you're still better off than without them.

Without immutable backups, the attacker encrypts/deletes your files and your backups. Now you have nothing.

With the immutable backups, they can't delete your backups. Those backups might include whatever malware and extra accounts they added. You might end up having to rebuild the entire environment from scratch at that point. But you can at least do file level restores of important stuff, virus scanning the files as you restore them. It might take a substantial effort to make sure everything is clean and you don't reopen a door the attacker created for themselves, but at least you have the option of doing that.

[mcz]

This is the perfect thread to add my thoughts. Anton Gostev's input contained a very important point I've never thought about: The encryption password itself. Now the point is that (if I'm not wrong) every backup for every restore point (=vbk, vib, vbr) is protected by a seperate encryption key which gets generated randomly. The password that you specify in the UI is the one to decrypt the 'real' encryption key to read the data. Now I assume that the same logic is applied on the object storage logic.

Now when you change the password for the encryption, the encryption keys for the existing backups won't change - or better said, they can't when the backups are immutable.

The point is: I never thought about an attacker that would gain access to the backup server for changing the encryption password. Because when the password get's changed, you won't notice it first, everything runs well. But if the attacker deletes the local backups and databases say after 6 months, your immutable backups on your object storage are still valid - but you won't have the password for decryption. That way it would be a kind of ransomware attack where the encryption was initiated by yourself. How bizzar!

Now I guess the only way to prevent something like this is (as Anton mentioned) to monitor your environment and test your immutable backups. I guess there's no way to prevent this because a root (or SYSTEM) user would be able to change the key in the database or in memory when the application runs and then you'd have the same result. My question is: Are my assumptions correct and if so, is there a way to at least restore the 'old' backups/restore points which got encrypted by the old key that you know. If so, is there an already existing (automated) way or would it be very painful and time consuming for the support team to get some valid data from restore points before the attacker has changed the key?

Thanks for letting me know!