Hardened repository

[vbussiro]

Hi folks,

Just tested this really good feature, and saw a minor flaw.
Let's say I configured an hardened repo, cerated a job , all run fine. Now a "clumsy" colleague goes to the repository config in veeam and disable immutability. I understand (and verified) that existing backup files won't be affected, only new files that will be created.
But... no warning/notification for me or in the next job run (like when a vm is deleted from a job)? I must check manually if backup files still have an immutable flag ?

Or have I missed something ?

[Mildur]

Yes, auditing the backup server and the jobs Setting is important, if you want to make sure, that nothing is wrong when you need the backups.

Veeam One has a Report for that to help you find out configuration changes.

https://helpcenter.veeam.com/docs/one/r ... ml?ver=110
https://helpcenter.veeam.com/docs/one/r ... ml?ver=110

[vbussiro]

Ok, make sense. Thank for your fast reply.
Maybe a notification in case immutability disablement could be beneficial ? As you do when a vm is "no longer processed by this job. Make sure this is intentional"

[Mildur]

That would be a nice feature.
But then again, the hacker or your insider can disable mail notification and you never receive a warning. I see a problem here. Without an active backup monitoring system or manually check the backup job state, this configuration change would not be easily detectable. But it would be a good start with the warning in the backup job.

[PTide]

Hi,

@vbussiro,

Are you talking about a mandatory, one-time warning that cannot be disabled?

Thanks!

[vbussiro]

@Mildur : I know that's not the right way, but truth is I probably won't look at audit log unless I suspect a problem. But seeing an unexpected warning or not receiving backup reports will ring a bell.
@PTide : In each concerned job, why not ? and maybe informational on next runs ? If I create a job storing backup on an immutable repository, there's a high chance I won't move it to an non immutable one. Again it's the same approach for your "vm no longer processed".

[vbussiro]

Hi Folks,

Old post, i know, but still on current issue i think

Just had an ramsomware attack on a customer with no immutable repository (we warned), but that's not the problem here. What bugs me : we found the attacker was probably here 4 months ago. Largely enough to silently disable immutability feature.

Yes, we should audit this. But there's no harm to add this warning as this reflect a configuration change as hazardous as vm removed from backup config, a case where there's a warning.

In fact, I have two more suggestions :
- have a way to set once immutability on the hardened repo and only be able to remove it from the hardened server itself, not from veeam backup server (i don't mean the immutabilty flag but the fact that any new backup file added on the repo would have the same immutability period that was initially configured)
- have a warning or even an error if there too much of time offset between veeam backup server and hardened repo (in case an attacker change time settings/server to cheat immutability, or reveal a time configuration error)

[PetrM]

Hello,

1. I don't think we should restrict disabling the Immutability feature from the Veeam server because it is a single orchestration point for the entire backup infrastructure and it must have control over all the provided features. However, I agree that it makes sense to track the fact of disabling such a feature, probably, it might be a request for Veeam ONE.

2. The immutable flag is set on the repository and the time change on Veeam server must not affect the flag expiration time on the repository.

Thanks!

[vbussiro]

Thanks,

1. Veeam One can audit this, yes, if installed... I still lthink it would be a nice idea to add this warning (like "immutabilty configuration changed. make sure this was intentional"). I also understand that V&B is the central point, but, in he same way it can't go against backup files immutability ("cannot delete since immutability flag is set"), why cant' we have immutability duration set in "hard" in the repo ?

2. I agree, but time offset should trigger an alert, since there could be consequences (maybe spoofed ntp, config error... on the repo ?)

I promise I won't push further, just wanted to share my point of view...

[PetrM]

Hello,

Don't hesitate to push any ideas and share any thoughts: in a dispute, truth is born

1. Ok, we can consider such a warning. I still think that we must control the backup infrastructure settings on the backup server side to ensure a single orchestration point, at least to avoid potential issues when two points of control are out of sync. Moreover, it looks like a big design change that requires a lot of resources for implementation and testing. Also, I think if the purpose is to prevent an intruder attack and keep the immutability setting intact, maybe we could ask to enter credentials for the repository to change the setting (single-use creds are not stored in Veeam database).

2. I don't argue that it is a good idea in general but I'm not sure that it is within the scope of problems that must be addressed by a backup application. Looks like something for monitoring software.

Thanks!

[HannesK]

Hello,
1. in V12, Hardened Repository is a dedicated repository type. One cannot remove the checkbox, because the checkbox does not exist anymore. The V12 user guide will describe how updating works.

2. Spoofing NTP is almost impossible with default settings of chrony and ntpd. And there are a few settings, that prevent attacks during reboot / service restart. A blog post on that topic is upcoming around V12 (because the blog post also describes how to set up NTS service with Ubuntu 22.04, but Ubuntu 22.04 is only supported in V12...). Detecting time changes on the Hardened Repository itself is still something we are looking into. For time difference between backup server & Hardened Repo I ask myself the following questions:

a. do backup server and Hardened Repository have different NTP servers in your case? If it's the same, then both times would be wrong (but the same) if the attacker owns the NTP server.
b. somebody who is administrator on the backup server can also change the time on the backup server. so the alarm would not work in that case. So a check for time difference would mainly be for "good feeling" right? Just to be clear: "good feeling" is also a valid reason to implement such things

Best regards,
Hannes

[vbussiro]

1. Perfect, that's solve it the best way.

2.a. It may be not the best way, but I put hardened repo on a separate network with only network access to the backup server, so not even access to an ntp. So.. it's a monthly "manual" check for updates, plugging keyboard, screen and second network cable with access to updates and ntp. Time consuming and not applicable for many... but well I don't think an attacker could both change time in backup server and repo server.
b. to be honest, it's also to prevent a time configuration error...it's amazing how this kind of error is sometimes hard to diagnose.

Thanks for all your answers.

[RubinCompServ]

Hello,
1. in V12, Hardened Repository is a dedicated repository type. One cannot remove the checkbox, because the checkbox does not exist anymore. The V12 user guide will describe how updating works.

And here's me that just finished building/testing my PowerShell scripts to build an Immutable repo