V11 Linux Immutability

[mkretzer]

Hello,

we are just creating a new ~480 TB XFS repo which we want to use with V11 XFS immutability.

Do we need to do something special to be able to use the same XFS with immutability for the then existing backups in V11?

Markus

[Gostev]

Hello!

Not at this time, you're good as is. There's no need for any special XFS settings, or anything like this.

After you upgrade to v11, you will only need to run a shell command to change the owner for the existing backup files. There's will be a message with instructions in the user interface if you enable immutable backups option on the existing repository. The rest is optional (additional hardening like disabling SSH server and remote consoles like iLO completely).

Thanks!

[mkretzer]

Very nice! Looking forward to using this and finding all the bugs in XFS

[Gostev]

Immutability is quite perpendicular to XFS, as this feature works on any file system really. It's just that XFS is the match made in heaven for it due to spaceless synthetic full backups. But as far as XFS specifically, it is holding up against our customer base quite nicely for over a year now (since first v10 beta) - so far no one was able to run it into the wall.

[mkretzer]

Challange accepted.

One thing is strange with XFS: We formated our 480 TB and before one backup was written to the XFS there was already a usage of 3-4 TB...

[Gostev]

I guess it makes sense considering XFS must use 4KB block size to support block cloning. This has to require a lot of disk space for metadata. In ReFS, this is controllable - and as you know, we recommend 64KB block size, which means 16 times less blocks!

[crackocain]

Hello

Is there any immutability disabled warning in V11?

For example hacker passed all security and disable the immutability options. 7 day to 0 day. And wait 7 days, after 7 days encrypt all backup files. How can we survive this scenario?

Could we have "immutability disabled" alerts? Some admins can not want enable immutability so warning is not important for them. But when immutability enabled one time in repository i think disable alert is important.

[mikeely]

Am I missing something or is immutability just "chattr +i backupfile" here that can be reversed by anyone with root access?

[crackocain]

Sorry for misunderstanding. Attacker change immutability in Veeam console repository settings. Not root access.
Attacker change setting and wait 7 days. So in 7 days all immutable flags removed by Veeam service in Linux XFS. After that attacker initiate encryption.

[Gostev]

@mikeely no matter how immutability is achieved, there's no protection from a root user. However, before a hacker could leverage the power of root account, they would need to obtain root credentials, and be able to logon to the server with them. And this is exactly what the new Hardened Linux Repository feature makes impossible for them to achieve.

First, unlike with regular repositories, root credentials for hardened repositories are not stored on the backup server (or anywhere else) - so even if the backup server is compromised, a hacker will not be able to fetch root credentials from it.

Second, even if a hacker somehow manages to obtain root credentials by other means, they will still be useless because hardened repository can work with SSH server disabled. So an attacker would have to be physically present at the server console to leverage those credentials, which means you can wave hackers goodbye.

As such, this solution indeed provides a 100% bulletproof protection against cyber-threats. But not from malicious insiders, since an upset employee with big enough hammer (or with a canister of gasoline) will easily bypass even most sophisticated access control and WORM storage technologies regardless, destroying all your data in no time. The only way to protect against those folks is to store a copy of your backups off-site with a 3rd party.

[Gostev]

Sorry for misunderstanding. Attacker change immutability in Veeam console repository settings. Not root access.
Attacker change setting and wait 7 days. So in 7 days all immutable flags removed by Veeam service in Linux XFS. After that attacker initiate encryption.

The UI should not allow you to set immutability to less than 7 days. At least I remember setting this limit myself when proofreading the corresponding UI control, but it was too long ago - so I will have to go back and look (also to make sure no one changed it later).

But in any case, the new immutability setting does not apply to existing backups, but only to newly created ones. So, an attacker will only be able to do anything with the most recent backups.

Otherwise, it's not any different that for example changing the job settings to quietly exclude your most critical machines from backup instead. So this is nothing new - you do need to audit access to your backup server, and monitor its configuration changes, for which Veeam provides a number of tools already.

Alerts on the other hand will not provide any decent protection against this kind of attack, because at this point a hacker has already took over the backup server, and can simply disable alerts temporarily, before doing all the dirty stuff.

[pirx]

As such, this solution indeed provides a 100% bulletproof protection against cyber-threats. But not from malicious insiders, since an upset employee with big enough hammer (or with a canister of gasoline) will easily bypass even most sophisticated access control and WORM storage technologies regardless, destroying all your data in no time. The only way to protect against those folks is to store a copy of your backups off-site with a 3rd party.

I'm a bit skeptical about this 100%. How will this Linux be patched? What about bugs in Veeam or other OS components, even without ssh enabled / port closed, there will be other ports open (somehow the data has to be stored). So this sounds better as a default and not hardened repository, but 100%... I think this is a bit too optimistic. Especially in times where even security appliances get hacked due to flaws in the product.

[Gostev]

You are right of course, nothing is ever 100% guaranteed, so I should never be using this number in principle when talking about IT-related things

But consider a beauty of the design. The only component "listening" on the network on the repository is the Veeam Transport Service. This one has much code after 15 years, so there's high chance some vulnerabilities can be found in it. However, this component runs as a limited user, so hackers won't be able to do much with those potential vulnerabilities anyway.

Then, there's tiny local service that runs as root and actually manages immutability flags. This one, on the other hand, provides literally just a couple of API calls with only a few lines of code behind each. While the presence of vulnerabilities cannot be excluded even in one line of code, due to such a small attack surface these will be discovered and addressed quite fast once v11 ships and this functionality gets sufficient scrutiny.

So, I guess it is more correct for me to say that IF vulnerabilities appear in Linux which will allow hackers to defeat such repository, which is a minimal hardened Linux install that is not listening to an outside world on any port except with the above-mentioned component running under a limited user account, then this will mean Linux ecosystem has much bigger things to worry about.

[mkretzer]

How will the hardened repo work with copy jobs? I know that GFS is mandatory but will it also protect the non GFS (latest) restore points? In theory this is not possible as these points are only working together with the "master" vbk which gets merged all the time.

[Gostev]

Backup Copy GFS retention was changed in v11 to work in the same way as it does in v10 primary jobs. This functionality above was actually one of the main reasons although there are some other features and issues this change addresses. Restore-point based retention and merges were problematic in so many ways! While time-based retention enables a few new features in v11, such as GFS backups immutability for their entire retention duration, continued GFS retention processing even after the corresponding backup or backup copy job is deleted, etc.

[mkretzer]

Very nice! Thank you for that info!
I just hope we can take the same backup chain that we have and it won't be the immediate copy mode situation all over again (we just converted everything to immediate)...

[Gostev]

Yes, you will be able to continue the existing chain.

[pirx]

You are right of course, nothing is ever 100% guaranteed, so I should never be using this number in principle when talking about IT-related things

But consider a beauty of the design. The only component "listening" on the network on the repository is the Veeam Transport Service. This one has much code after 15 years, so there's high chance some vulnerabilities can be found in it. However, this component runs as a limited user, so hackers won't be able to do much with those potential vulnerabilities anyway.

Then, there's tiny local service that runs as root and actually manages immutability flags. This one, on the other hand, provides literally just a couple of API calls with only a few lines of code behind each. While the presence of vulnerabilities cannot be excluded even in one line of code, due to such a small attack surface these will be discovered and addressed quite fast once v11 ships and this functionality gets sufficient scrutiny.

So, I guess it is more correct for me to say that IF vulnerabilities appear in Linux which will allow hackers to defeat such repository, which is a minimal hardened Linux install that is not listening to an outside world on any port except with the above-mentioned component running under a limited user account, then this will mean Linux ecosystem has much bigger things to worry about.

It definitely sounds interesting! I'm just thinking how this can be compared to a S3 bucket at AWS + immutability, where even AWS claims that they can not delete the data. Personally I still would rate AWS etc S3 immutability higher as it is at a different location/organization, so it is more difficult to compromise immutable backups there. This all depends on requirements and of course budget.

[Gostev]

Yes, there's no doubt cloud object storage with S3 Object Lock support provides much higher level of protection, simply because it also provides protection against malicious insiders. In fact, it is fully analogous to shipping tapes to Iron Mountain: your data is offline and in a 3rd party "vault". No on-prem WORM solution will ever be able to provide an adequate insider protection due to physical access to media.

I am skeptical though to AWS claims that they can not delete the data. Imagine someone uploading hundreds of TB worth of backups locked for 10 years, and then blocking the credit card so Amazon can't charge it. Will Amazon really hold all this data for 10 years? It's very unlikely they don't have a process to deprovision inactive tenants including all of their resources.

[crackocain]

Hello

I want to ask about immutability time sync. What is the best practise? We consider block all traffic except backup.

[Gostev]

My first reaction - if you disable time sync on a hardened repository server completely, then you won't have to worry about this attack vector at all. And I don't expect the clock to be running away by more than a some seconds per year... so you can just adjust it manually while installing new Veeam versions periodically.

Otherwise, if you do want mathematic time precision for whatever reason, then you could get a GPS time dongle.

[crackocain]

Thank you Anton, manually control is enough for this scenerio. Customer check failures every month in data center so add one more job to do

[mkretzer]

@Gostev: How much drift would be acceptable for Veeam Components to work correctly?

[Gostev]

Not sure about all Veeam components, but local time of a hardened repository server does not matter for anything in principle.

[tedsteenvoorden]

Hello, I have a question regarding the new immutability function of Veeam v11. What happens if the immutable function is disabled from the Veeam Console? Is the immutability removed immediately or after the immutable period expires?

[NightBird]

After the immutable period expires, if not so it's not immutable and an insider could delete immutable backup by disabling immutability in Veeam Console

[tedsteenvoorden]

Thank you Nightbird, for the information!

[crackocain]

Hello

I configuring Immutability different setup. I want to ask something.

We have a server and two storage. This storages are direct attached to server via FC
So i add two different extent in Scale Out Backup repository but one server. I don't want to use LVM to combine two volumes.
Is Immutability works?

[Mildur]

1) Attach the storages over FC to the linux hardware server
2) Format the FC LUNs with xfs
3) mount both LUNs under a different Mount Point
4) add the linux hardware server as a linux hardened repo to the VBR
5) Create the Backup repos to the both mount points
6) Activate the immutable Retention

After that, you can create your SOBR with both extends.

Adding the Linux Server as a hardened Repo:

When adding the Linux server, use temporary credentials. To do that, click Add and select Single-use credentials for hardened repository at the SSH Connection step of the New Linux Server wizard

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[TonioRoffo]

But in any case, the new immutability setting does not apply to existing backups, but only to newly created ones. So, an attacker will only be able to do anything with the most recent backups.

I'd like to make a humble suggestion for the immutable setting on hardened repositories.

What if you remove the setting in Veeam console all together and make it so that you can only set the parameter on the interactive linux console on the repository?