Veeam v11 - Single Use Passwords for Linux Repo

[crackocain]

Hello

I watched Anyhony and Michael session and i liked immutability feature. But immutability is not complete solution. Because attacker could login to SSH and destroy everything.

I think one time (single use) passwords is great solution for this subject. I want to ask is PAM packages used for this feature and could you eloborate explain for the solution?

[HannesK]

Hello,

Because attacker could login to SSH and destroy everything.

how? SSH Server can be deactivated after installation of the immutable repository.

I want to ask is PAM packages used for this feature and could you eloborate explain for the solution?.

hmm, not sure what you mean. If you mean pluggable authentication modules, then they are on the Linux side. Not on the Windows (Veeam) side. We simply don't store the password anywhere. You enter the password once and there is no way to extract it from the database, because we do not store it anywhere.

Veeam components use certificate based authentication against each other (different to V10 and earlier where SSH was used)

In general: the attack vector against the software solution is the same like against every other WORM software solution: if you gain root / administrator access, then you can delete the data. It's in the nature of every WORM software. For a WORM hardware solution: physical access breaks it.

Best regards,
Hannes

[crackocain]

Is Immutability feature support both Reverse & Forward Incremental? Especially Reverse?

1 full backup & 30 days retention

[Gostev]

Hannes, I believe Mehmet is simply asking for details... I'm not sure at what level they covered it, since they had 25 minutes to talk about 11 features

But no, there's nothing fancy like PAM (I don't know what is it btw). The concept is very simple: the repository machine's credentials with root privileges are not stored anywhere - neither on a backup server nor anywhere else in the environment. So, even if the whole infrastructure is overtaken, the hacker can never obtain those credentials from any component.

So, disabling SSH Server on the backup repository machine is completely optional. However, for paranoid-level protection you will want to do this, as doing this will protect from other attack vectors: zero day security vulnerabilities in the SSH Server itself, hackers obtaining the root password via keylogging or social engineering, etc. With SSH Server disabled, only local console access remains available - which means the attacker must be physically present at the repository server. So, you can wave ransomware and hackers goodbye even if they do obtain root credentials.

Now, physical access to hardware and media of course allows to bypass any protection at all... but this is the case with any WORM solution.

[Gostev]

Is Immutability feature support both Reverse & Forward Incremental? Especially Reverse?

Well, since backup files are made truly immutable, obviously no backup mode that changes them in place can be supported even theory. Thus, you must use a backup mode with periodic fulls. This makes XFS the perfect candidate for hardened repository, since synthetic full backups do not take any disk space with it.

[crackocain]

Hello Anton.

Thanks for the update! My colleague mentioned me about PAM so we think Veeam maybe use it. https://wiki.archlinux.org/index.php/One_Time_PassWord

My colleague asks if Veeam set "i" flag for immutability also Veeam could remove the "i" flag. Is back end certificate based authentication could be used by attacker? The scenario is Veeam Backup Server credentials is compromised.

[HannesK]

you can make it a one time password. Up to you. For Veeam, it's not relevant. We need to login once for the installation of persistent components. Whatever you do afterwards with the SSH or root password is irrelevant for the software.

Veeam could remove the "i" flag

full documentation will follow at GA. The veeam repository service runs as normal Linux user. A normal Linux user cannot remove the immutable flag.

Only root can remove the immutable flag.

There is a separate local service that runs under root, but has no network interface. This service checks the current time and removes the immutable flag from files where immutability period has expired. There is no way to connect to that service from the network for an attacker.

The scenario is Veeam Backup Server credentials is compromised.

that's exactly the scenario we cover. That attack is no problem

[crackocain]

Thanks Hannes.

Good solution! I think forward incremental with weekly syntetic fulls and 7 days immutability is protect daily data. right?

[Gostev]

Yes, that will work. But even if you specify 1 day of immutability, the entire latest chain will be protected regardless, because of dependencies on earlier backups in chain for the recovery. These dependencies are analyzed before the immutability flag is removed from a backup file.

The approach of setting +i has actually changed a bit from the one known to Hannes from the training a few months ago. A normal Linux user can NOT set +i by default, but this is easily solvable with the setuid bit, which is what we were planning to do originally. However, in the end for a number of reasons we decided to isolate all the immutability flag operations to that separate local service that runs under root. The only API function this service exposes [locally to our data mover] allows to set +i. Or if the flag is already set on a backup file, then it allows to extend its expiration time value - but the new value provided has to be beyond the currently specified value, or the operation will be refused.

In any case, we're actually going through the official WORM certification to address any security concern or possible speculations in regards to our approach.

[Mildur]

Well, since backup files are made truly immutable, obviously no backup mode that changes them in place can be supported even theory. Thus, you must use a backup mode with periodic fulls. This makes XFS the perfect candidate for hardened repository, since synthetic full backups do not take any disk space with it.

Hi Anton

I see a problem for Nutanix AHV Backups. There is no possibility today for synthetic Fulls for AHV Backups.
Only active fulls. With the new secure Linux Repo, we have to do now Active Fulls of a 15-20TB Nutanix Cluster to use the new immutable feature.

Is synthetic Full planned for the next Nutanix release? I want to use the immutable option, but buying new space for active fulls, is a bit to much right now for us

[Gostev]

Hi Fabian, as far as I know, it is considered. It is best to direct the AHV functionality questions to the corresponding sub-forum, where the corresponding PMs are much better equipped to comment. Thanks!

[crackocain]

Thanks Anton for detailed explanation.

When will v11 release? We can't wait CDP and Immutability

[Mildur]

Hi Fabian, as far as I know, it is considered. It is best to direct the AHV functionality questions to the corresponding sub-forum, where the corresponding PMs are much better equipped to comment. Thanks!

Thanks, I will ask there

[Gostev]

Mehmet, v11 should be around the same time as v10 was this year, as our major release cycle has been approximately annual.

But it's been feature-complete and in beta since August, so you can contact your Veeam sales rep to see if they still have any open spots. We're about to do the last beta code drop in the next few days actually, so right now is the perfect time to test all these features you mentioned and provide feedback.

[Gostev]

In any case, we're actually going through the official WORM certification to address any security concern or possible speculations in regards to our approach.

Just to update on this, V11 has successfully passed a 3rd party assessment of compliance with the U.S. financial industry regulations for WORM (Write Once Read Many) storage. A compliant hardened repository configuration ensures protection of backup data against manipulation and meets the requirements for non-rewritable, non-erasable storage as specified by SEC 17a-4(f), FINRA 4511(c) and CFTC 1.31(c)-(d).

The assessment was done by Cohasset Associates, which is actually the very company that validated Amazon's own S3 Object Lock implementation! As well as WORM capabilities of Google Cloud Storage, IBM Cloud Object Storage, Microsoft Azure Blob Storage; and those of on-prem object storage like Cloudian and Minio to name a few.

This puts our hardened repository in a very good company indeed

[unsichtbarre]

Hi Gostev,

Can you provide the actual results or a link on Cohasset with the results for client reference?

THX,
-JB

[tsightler]

@unsichtbarre I believe this is what you are looking for.

[unsichtbarre]

Exactly, thanks tsightler!