Immutability with on-prem S3 storage

[sfirmes]

The Veeam Ready site will include the list of vendors who pass immutability/object locking mid to late April. Until then use the link that @dalbertson provided for you.

[steve_costigan]

Full guide on setting up Zadara for immutability with v10 and using Zadara as an end to end SOBR capability with Veeam VBR v10 is here https://support.zadarastorage.com/hc/en ... Repository.
This allows you to setup a multi-tenant Object Storage capability if you want to provide isolation between tenants, proxies / services any questions please let me know.

[dcit]

Summary:

Will be minio (after implementing versioning) "the right way" for building cheap solution for "emulated air-gapped backup" - custom built server with bunch of disks and RAID card hosted in offsite datacenter?

Detailed:

Btw I have almost added (to that summary above): with TPM & using Windows BitLocker - or another method for encrypting whole disk such as RAID card level encryption - from our perspective encrypting data on our server of our company is mandatory.
But probably encryption of backup data on Veeam Backup side can be enough, and protecting that server would be only "nice bonus".

For now we have as offsite backup custom build server with bunch of disks in RAID10 running Windows Server 2016 (and using built-in dedup) in datacenter in another city (data encrypted by BitLocker protected by TPM). For some time I have been thinking about air-gapped backup and more recently about "emulating" air-gap in cloud - thanks for Gostev's amazing Veeam Digest posts. And last one got me thinking that we can try immutable backups in S3 object storage, but I am afraid of costs of this Amazon cloud storage, so it occured to me that may be we can run some S3 compatible object storage ourselves. Then by googling get to this forum, and for now it seems for me that after minio gets versioning support it could be the right solution. It seems it supports even Windows, so we can still use BitLocker, and of course that machine can not be then administratively accessible from our primary site, I mean such as entering aministrator credentials to Veeam Backup (which we have now as that machine is serving as Veeam Backup Proxy). Then it should be rather resistant against ransomware attacks.

Am I thinking right?

Btw another way can be may be to "promote" that offsite server to Veeam Backup and "demote" local server (with bunch of disks in RAID6) to Veeam Backup Proxy, and change administration password of that offsite server. What I am afraid of is that for this to be done it would be necessary to enter admin crendentials for our internal servers to something running offsite. I am aware that even now malicious element in that datacenter can froze memory of our server, then read them using right tools and find BitLocker keys and then access our data, and then read everything from backups including password databases of our systems, but entering those crendentials directly to Veeam Backup server can make it even easier. But may be imaging someone frozing memory of our server hosted in datacenter is a bit too much paranoid, or it is possible that it would be easier for attacker to put gun to my head and just ask politely

Btw all traffic to that current offsite server (and it would be the same for eventual another one) is either encrypted by IPsec (directly client to server without tunnel using built-in Windows firewall) or denied, so attack over network would be probably very dificult, baring eventual bugs in Windows TCP/IP stack may be even impossible, of course exluding hacking computers with set up IPsec encryption for communication with that server, such as our local Veeam Backup server.

[dcit]

Next question is when Veeam Backup is locking new backup in S3 (Amazon or compatible) object storage for e.g. 3 days, is it locking already existing objects corresponding to initial full backup too (except parts of initial backup which were overwritten before 3 days)? It seems logical as other way attacker can delete large part of backup and protected will be only incremental changes from last 3 days, but just to be sure, I am asking now.

[sfirmes]

Will be minio (after implementing versioning) "the right way" for building cheap solution for "emulated air-gapped backup"

MinIO is working on developing versioning and object lock that will work with our Immutability feature. When they launch that capability we will note it in the list of compatible products object-storage-f52/unoffizial-compatibi ... 56956.html

Thanks

Steve

[Gostev]

@dcit yes, of course we will extend the lock on "older" objects for the appropriate time when they are reused in new restore points.

[fordhprefect]

Hi @sfirmes,
You mentioned in your post dated Feb 13 2020 that you would post any Ceph test results when they were conducted. Are you aware of anyone working on this? Or is there an opportunity for me to get involved with the testing?

[HannesK]

Hi Joff,
please see the link Steve posted... Ceph is already on the list as "compatible including immutability"

Best regards,
Hannes

[mengl]

MinIO doesn't support versioning yet but it is being actively worked on https://github.com/minio/minio/tree/xl-v2

The relevant issue to track https://github.com/minio/minio/issues/2 ... -577406100

Seems that this PR is merged since a few days. Couldn't try it yet.

[sfirmes]

MinIO is getting close to releasing the version of their software with supports versioning and will also support our immutability feature. When they pass the Veeam Ready Object with Immutability testing, I will update this thread with the links to the Veeam Ready site and our compatibility list.

Thanks

Steve

[kmertens]

They just posted a new version of the software with versioning support. Going to test this soon!

[sfirmes]

@kmertens you are correct. MinIO just released RELEASE.2020-07-12T19-14-17Z which supports object locking and versioning. This release also supports Veeam's immutability feature introduced in VBR v10. You should see some "how-to" materials which will guide you through the setup process.

One thing to note is that you will need to configure erasure coding for MinIO. A guide exists for this already and is very helpful https://docs.min.io/docs/minio-erasure- ... uide.html

I will update this thread when the new guides are available.

[poulpreben]

While the latest MinIO release supports versioning, it is still lacking support for immutability. I just tested the build and it refuses to create a bucket with object lock enabled:

Code: Select all

A header you provided implies functionality that is not implemented

This was tested using the AWS boto3 SDK with the following client settings (which works for other S3 compatible storages):

Code: Select all

client.create_bucket(
    Bucket=bucket_name,
    ObjectLockEnabledForBucket=True
)

[sfirmes]

@poulpreben Not sure why you are having issues. The latest of MinIO does support our immutability. They have passed the Veeam Ready Object with Immutability testing and our website should be updated this week to reflect that. Did you implement the erasure coding that I noted earlier? It requires at least 4 disks to be used by MinIO.

[poulpreben]

Hi Stephen. You are right, I just started a single-node instance for testing the functionality, and totally missed that EC is a requirement for object locking/versioning. I will head back to the labs and test again.

[sfirmes]

Glad to help. We will be publishing some guides soon to help make the setup and configuration easier. I’ll post a link when the guides are finished.

Steve

[omsaay]

Hello I am trying to ask something directly to Gostev, hope this is the right way....

I have read your last two posts on Veeam Community Forums Digest and you're talking about MinIO. I have a client that needs to store historical backups for 5 years. They are trying to find the best (and cheaper ) solution, does MinIO can work for them with let say some SuperMicro Servers (SuperStorage 6049SP-DE2CR90)? What do you think?

Regards,

[Gostev]

Hello, please check with MinIO on this, as it's their product. Thanks!

[omsaay]

Yes Gostev you are right, I think that I made a mistake with my question, do you think that MinIO could work fine for this project?

[Gostev]

I don't see why not. But we can't make recommendation as it comes to production usage, because we don't have such experience with MinIO.

[dimaslan]

I was able to - after a dozen hours of struggling - create a bucket with Immutability on an Ubuntu vm I created in my home Veeam lab to test the feature. Sadly, I have spent too many hours trying to just figure out how to get the certificate Veeam is looking for but I cannot. It's unfortunate that minio documentation takes so many things for granted and the prerequisites are missing.
Do I need to install an Apache server? Do I just create the certificate or are there more steps to incorporate it after creation?
I wish it would be a bit clearer but apparently with Linux nothing ever is.

[ctalbot]

@dimitris, Jorge put together a comprehensive blog post on this topic a few months back. https://jorgedelacruz.uk/2020/07/22/vee ... #confminio

You need to make sure minio is enabled with erasure encoding. No need for an Apache server as the mino server responds to https requests. You also don't need to use a Let's encrypt certificate, you can create your own: https://docs.min.io/docs/how-to-secure- ... h-tls.html

[dimaslan]

@ctalbot, thanks for the quick reply. I am using erasure encoding. I will check Jorge's blog, hopefully steps will be more detailed.

Thank you.

[dimaslan]

Yeah the guide is great but it's assuming certbot which I cannot use. I guess it's more reading then.

[sfirmes]

@dimaslan I used openssl to create a self-signed cert using these steps https://docs.min.io/docs/how-to-secure- ... ertificate.

This is the openssl.conf file that I used. Other than the ip address of my MinIO server, I used the defaults.

Code: Select all

# cat openssl.conf
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = US
ST = VA
L = Somewhere
O = MyOrg
OU = MyOU
CN = MyServerName

[v3_req]
subjectAltName = @alt_names

[alt_names]
IP.1 = <ip address of MinIO server>

Hope this helps.

Steve

[dimaslan]

I will try that, thank you Stephen.

[dimaslan]

Steve,

Can you please help me with
1. After creating the ssl certificate, where you you place it
2. What is the command you're using to start minio with erasure code to include the certificate?

Thank you.

[poulpreben]

For a complete example, I suggest that you create the following folders:

Code: Select all

mkdir -p ~/.minio/certs/CAs
mkdir -p /minio/data01 /minio/data02 /minio/data03 /minio/data04

Run the following command:

Code: Select all

cd ~/.minio/certs
openssl req -x509 -newkey rsa:4096 -keyout private.key -out public.crt -days 365 -nodes
cp ~/.minio/certs/public.crt ~/.minio/certs/CAs

You can enter all the parameters ad-hoc instead of creating the configuration file as suggested by @sfirmes. I typed in as follows:

Code: Select all

Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Bla bla
Locality Name (eg, city) []:Forums
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Veeam Community
Organizational Unit Name (eg, section) []:Forums
Common Name (e.g. server FQDN or YOUR name) []:myminio.storage.local
Email Address []:hostmaster@storage.local

This will give you two files: private.key which is your private key, and public.crt which is your public key.

You should now be able to start MinIO using:

Code: Select all

minio server --address=":443" /minio/data{01...04}

[poulpreben]

You can add Ceph version >= 14.2.6 to that list.

The bug described in the current listed under Ceph as a supported system has now been merged, and backported to the three supported major versions; Nautilus, Octopus and Pacific (bug tracker).

As such, the fix is included in the following versions:

• 14.2.22
• 15.2.14
• 16.2.5

[Gostev]

Thanks @poulpreben, I updated the sticky with new version numbers and removed a reference to the immutability circumvention bug.
Looks like there's still another issue open around the date format though right @Andreas Neufert?