Discussions related to using object storage as a backup target.
sfirmes
Veeam Software
Posts: 225
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Immutability with on-prem S3 storage

Post by sfirmes » 1 person likes this post

The Veeam Ready site will include the list of vendors who pass immutability/object locking mid to late April. Until then use the link that @dalbertson provided for you.
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
steve_costigan
Lurker
Posts: 2
Liked: 7 times
Joined: Apr 01, 2020 5:46 pm
Full Name: Steve Costigan
Location: United Kingdom
Contact:

Re: Immutability with on-prem S3 storage

Post by steve_costigan » 2 people like this post

Full guide on setting up Zadara for immutability with v10 and using Zadara as an end to end SOBR capability with Veeam VBR v10 is here https://support.zadarastorage.com/hc/en ... Repository.
This allows you to setup a multi-tenant Object Storage capability if you want to provide isolation between tenants, proxies / services any questions please let me know.
dcit
Enthusiast
Posts: 33
Liked: 4 times
Joined: Mar 13, 2015 1:06 am
Contact:

Re: Immutability with on-prem S3 storage

Post by dcit »

Summary:

Will be minio (after implementing versioning) "the right way" for building cheap solution for "emulated air-gapped backup" - custom built server with bunch of disks and RAID card hosted in offsite datacenter?

Detailed:

Btw I have almost added (to that summary above): with TPM & using Windows BitLocker - or another method for encrypting whole disk such as RAID card level encryption - from our perspective encrypting data on our server of our company is mandatory.
But probably encryption of backup data on Veeam Backup side can be enough, and protecting that server would be only "nice bonus".

For now we have as offsite backup custom build server with bunch of disks in RAID10 running Windows Server 2016 (and using built-in dedup) in datacenter in another city (data encrypted by BitLocker protected by TPM). For some time I have been thinking about air-gapped backup and more recently about "emulating" air-gap in cloud - thanks for Gostev's amazing Veeam Digest posts. And last one got me thinking that we can try immutable backups in S3 object storage, but I am afraid of costs of this Amazon cloud storage, so it occured to me that may be we can run some S3 compatible object storage ourselves. Then by googling get to this forum, and for now it seems for me that after minio gets versioning support it could be the right solution. It seems it supports even Windows, so we can still use BitLocker, and of course that machine can not be then administratively accessible from our primary site, I mean such as entering aministrator credentials to Veeam Backup (which we have now as that machine is serving as Veeam Backup Proxy). Then it should be rather resistant against ransomware attacks.

Am I thinking right?

Btw another way can be may be to "promote" that offsite server to Veeam Backup and "demote" local server (with bunch of disks in RAID6) to Veeam Backup Proxy, and change administration password of that offsite server. What I am afraid of is that for this to be done it would be necessary to enter admin crendentials for our internal servers to something running offsite. I am aware that even now malicious element in that datacenter can froze memory of our server, then read them using right tools and find BitLocker keys and then access our data, and then read everything from backups including password databases of our systems, but entering those crendentials directly to Veeam Backup server can make it even easier. But may be imaging someone frozing memory of our server hosted in datacenter is a bit too much paranoid, or it is possible that it would be easier for attacker to put gun to my head and just ask politely ;-)

Btw all traffic to that current offsite server (and it would be the same for eventual another one) is either encrypted by IPsec (directly client to server without tunnel using built-in Windows firewall) or denied, so attack over network would be probably very dificult, baring eventual bugs in Windows TCP/IP stack may be even impossible, of course exluding hacking computers with set up IPsec encryption for communication with that server, such as our local Veeam Backup server.
dcit
Enthusiast
Posts: 33
Liked: 4 times
Joined: Mar 13, 2015 1:06 am
Contact:

Re: Immutability with on-prem S3 storage

Post by dcit »

Next question is when Veeam Backup is locking new backup in S3 (Amazon or compatible) object storage for e.g. 3 days, is it locking already existing objects corresponding to initial full backup too (except parts of initial backup which were overwritten before 3 days)? It seems logical as other way attacker can delete large part of backup and protected will be only incremental changes from last 3 days, but just to be sure, I am asking now.
sfirmes
Veeam Software
Posts: 225
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Immutability with on-prem S3 storage

Post by sfirmes »

dcit wrote: Apr 21, 2020 6:21 pm Will be minio (after implementing versioning) "the right way" for building cheap solution for "emulated air-gapped backup"
MinIO is working on developing versioning and object lock that will work with our Immutability feature. When they launch that capability we will note it in the list of compatible products object-storage-f52/unoffizial-compatibi ... 56956.html

Thanks

Steve
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutability with on-prem S3 storage

Post by Gostev » 1 person likes this post

@dcit yes, of course we will extend the lock on "older" objects for the appropriate time when they are reused in new restore points.
fordhprefect
Lurker
Posts: 1
Liked: never
Joined: Jun 15, 2020 12:57 am
Full Name: Joff Pearce
Contact:

Re: Immutability with on-prem S3 storage

Post by fordhprefect »

Hi @sfirmes,
You mentioned in your post dated Feb 13 2020 that you would post any Ceph test results when they were conducted. Are you aware of anyone working on this? Or is there an opportunity for me to get involved with the testing?
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Immutability with on-prem S3 storage

Post by HannesK »

Hi Joff,
please see the link Steve posted... Ceph is already on the list as "compatible including immutability"

Best regards,
Hannes
mengl
Service Provider
Posts: 12
Liked: 10 times
Joined: Oct 19, 2018 7:02 am
Full Name: Michael Engl
Location: Germany
Contact:

Re: Immutability with Minio possible?

Post by mengl »

y4m4 wrote: Feb 10, 2020 8:08 pm MinIO doesn't support versioning yet but it is being actively worked on https://github.com/minio/minio/tree/xl-v2

The relevant issue to track https://github.com/minio/minio/issues/2 ... -577406100
Seems that this PR is merged since a few days. Couldn't try it yet.
sfirmes
Veeam Software
Posts: 225
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Immutability with on-prem S3 storage

Post by sfirmes » 4 people like this post

MinIO is getting close to releasing the version of their software with supports versioning and will also support our immutability feature. When they pass the Veeam Ready Object with Immutability testing, I will update this thread with the links to the Veeam Ready site and our compatibility list.

Thanks

Steve
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
kmertens
Service Provider
Posts: 6
Liked: never
Joined: Dec 26, 2016 1:55 pm
Location: Belgium
Contact:

Re: Immutability with on-prem S3 storage

Post by kmertens »

They just posted a new version of the software with versioning support. Going to test this soon!
sfirmes
Veeam Software
Posts: 225
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Immutability with on-prem S3 storage

Post by sfirmes » 1 person likes this post

@kmertens you are correct. MinIO just released RELEASE.2020-07-12T19-14-17Z which supports object locking and versioning. This release also supports Veeam's immutability feature introduced in VBR v10. You should see some "how-to" materials which will guide you through the setup process.

One thing to note is that you will need to configure erasure coding for MinIO. A guide exists for this already and is very helpful https://docs.min.io/docs/minio-erasure- ... uide.html

I will update this thread when the new guides are available.
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
poulpreben
Certified Trainer
Posts: 1024
Liked: 448 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: Immutability with on-prem S3 storage

Post by poulpreben »

While the latest MinIO release supports versioning, it is still lacking support for immutability. I just tested the build and it refuses to create a bucket with object lock enabled:

Code: Select all

A header you provided implies functionality that is not implemented
This was tested using the AWS boto3 SDK with the following client settings (which works for other S3 compatible storages):

Code: Select all

client.create_bucket(
    Bucket=bucket_name,
    ObjectLockEnabledForBucket=True
)
sfirmes
Veeam Software
Posts: 225
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Immutability with on-prem S3 storage

Post by sfirmes »

@poulpreben Not sure why you are having issues. The latest of MinIO does support our immutability. They have passed the Veeam Ready Object with Immutability testing and our website should be updated this week to reflect that. Did you implement the erasure coding that I noted earlier? It requires at least 4 disks to be used by MinIO.
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
poulpreben
Certified Trainer
Posts: 1024
Liked: 448 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: Immutability with on-prem S3 storage

Post by poulpreben »

Hi Stephen. You are right, I just started a single-node instance for testing the functionality, and totally missed that EC is a requirement for object locking/versioning. I will head back to the labs and test again.
sfirmes
Veeam Software
Posts: 225
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Immutability with on-prem S3 storage

Post by sfirmes » 1 person likes this post

Glad to help. We will be publishing some guides soon to help make the setup and configuration easier. I’ll post a link when the guides are finished.

Steve
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
omsaay
Service Provider
Posts: 2
Liked: never
Joined: Mar 17, 2020 4:25 pm
Full Name: Omar Sanchez
Contact:

Re: Immutability with on-prem S3 storage

Post by omsaay »

Hello I am trying to ask something directly to Gostev, hope this is the right way....

I have read your last two posts on Veeam Community Forums Digest and you're talking about MinIO. I have a client that needs to store historical backups for 5 years. They are trying to find the best (and cheaper ) solution, does MinIO can work for them with let say some SuperMicro Servers (SuperStorage 6049SP-DE2CR90)? What do you think?

Regards,
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutability with on-prem S3 storage

Post by Gostev »

Hello, please check with MinIO on this, as it's their product. Thanks!
omsaay
Service Provider
Posts: 2
Liked: never
Joined: Mar 17, 2020 4:25 pm
Full Name: Omar Sanchez
Contact:

Re: Immutability with on-prem S3 storage

Post by omsaay »

Yes Gostev you are right, I think that I made a mistake with my question, do you think that MinIO could work fine for this project?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutability with on-prem S3 storage

Post by Gostev »

I don't see why not. But we can't make recommendation as it comes to production usage, because we don't have such experience with MinIO.
dimaslan
Service Provider
Posts: 99
Liked: 9 times
Joined: Jul 01, 2017 8:02 pm
Full Name: Dimitris Aslanidis
Contact:

Re: Immutability with on-prem S3 storage

Post by dimaslan »

I was able to - after a dozen hours of struggling - create a bucket with Immutability on an Ubuntu vm I created in my home Veeam lab to test the feature. Sadly, I have spent too many hours trying to just figure out how to get the certificate Veeam is looking for but I cannot. It's unfortunate that minio documentation takes so many things for granted and the prerequisites are missing.
Do I need to install an Apache server? Do I just create the certificate or are there more steps to incorporate it after creation?
I wish it would be a bit clearer but apparently with Linux nothing ever is.
ctalbot
Enthusiast
Posts: 60
Liked: 19 times
Joined: Oct 19, 2016 2:14 pm
Full Name: Carlos Talbot
Location: Chicago, IL
Contact:

Re: Immutability with on-prem S3 storage

Post by ctalbot » 1 person likes this post

@dimitris, Jorge put together a comprehensive blog post on this topic a few months back. https://jorgedelacruz.uk/2020/07/22/vee ... #confminio

You need to make sure minio is enabled with erasure encoding. No need for an Apache server as the mino server responds to https requests. You also don't need to use a Let's encrypt certificate, you can create your own: https://docs.min.io/docs/how-to-secure- ... h-tls.html
dimaslan
Service Provider
Posts: 99
Liked: 9 times
Joined: Jul 01, 2017 8:02 pm
Full Name: Dimitris Aslanidis
Contact:

Re: Immutability with on-prem S3 storage

Post by dimaslan »

@ctalbot, thanks for the quick reply. I am using erasure encoding. I will check Jorge's blog, hopefully steps will be more detailed.

Thank you.
dimaslan
Service Provider
Posts: 99
Liked: 9 times
Joined: Jul 01, 2017 8:02 pm
Full Name: Dimitris Aslanidis
Contact:

Re: Immutability with on-prem S3 storage

Post by dimaslan »

Yeah the guide is great but it's assuming certbot which I cannot use. I guess it's more reading then.
sfirmes
Veeam Software
Posts: 225
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Immutability with on-prem S3 storage

Post by sfirmes »

@dimaslan I used openssl to create a self-signed cert using these steps https://docs.min.io/docs/how-to-secure- ... ertificate.

This is the openssl.conf file that I used. Other than the ip address of my MinIO server, I used the defaults.

Code: Select all

# cat openssl.conf
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
C = US
ST = VA
L = Somewhere
O = MyOrg
OU = MyOU
CN = MyServerName

[v3_req]
subjectAltName = @alt_names

[alt_names]
IP.1 = <ip address of MinIO server>
Hope this helps.

Steve
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
dimaslan
Service Provider
Posts: 99
Liked: 9 times
Joined: Jul 01, 2017 8:02 pm
Full Name: Dimitris Aslanidis
Contact:

Re: Immutability with on-prem S3 storage

Post by dimaslan »

I will try that, thank you Stephen.
dimaslan
Service Provider
Posts: 99
Liked: 9 times
Joined: Jul 01, 2017 8:02 pm
Full Name: Dimitris Aslanidis
Contact:

Re: Immutability with on-prem S3 storage

Post by dimaslan »

Steve,

Can you please help me with
1. After creating the ssl certificate, where you you place it
2. What is the command you're using to start minio with erasure code to include the certificate?

Thank you.
poulpreben
Certified Trainer
Posts: 1024
Liked: 448 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: Immutability with on-prem S3 storage

Post by poulpreben »

For a complete example, I suggest that you create the following folders:

Code: Select all

mkdir -p ~/.minio/certs/CAs
mkdir -p /minio/data01 /minio/data02 /minio/data03 /minio/data04
Run the following command:

Code: Select all

cd ~/.minio/certs
openssl req -x509 -newkey rsa:4096 -keyout private.key -out public.crt -days 365 -nodes
cp ~/.minio/certs/public.crt ~/.minio/certs/CAs
You can enter all the parameters ad-hoc instead of creating the configuration file as suggested by @sfirmes. I typed in as follows:

Code: Select all

Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Bla bla
Locality Name (eg, city) []:Forums
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Veeam Community
Organizational Unit Name (eg, section) []:Forums
Common Name (e.g. server FQDN or YOUR name) []:myminio.storage.local
Email Address []:hostmaster@storage.local
This will give you two files: private.key which is your private key, and public.crt which is your public key.

You should now be able to start MinIO using:

Code: Select all

minio server --address=":443" /minio/data{01...04}
poulpreben
Certified Trainer
Posts: 1024
Liked: 448 times
Joined: Jul 23, 2012 8:16 am
Full Name: Preben Berg
Contact:

Re: Immutability with Minio possible?

Post by poulpreben » 1 person likes this post

poulpreben wrote: Feb 12, 2020 12:47 pm You can add Ceph version >= 14.2.6 to that list.
The bug described in the current listed under Ceph as a supported system has now been merged, and backported to the three supported major versions; Nautilus, Octopus and Pacific (bug tracker).

As such, the fix is included in the following versions:
  • 14.2.22
  • 15.2.14
  • 16.2.5
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Immutability with on-prem S3 storage

Post by Gostev »

Thanks @poulpreben, I updated the sticky with new version numbers and removed a reference to the immutability circumvention bug.
Looks like there's still another issue open around the date format though right @Andreas Neufert?
Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests