Comprehensive data protection for all workloads
Gustav
Enthusiast
Posts: 45
Liked: 51 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Build an immutable backup repository - article series

Post by Gustav » 32 people like this post

As a Windows admin, having only smelled to Linux, I decided to set up an immutable repository, thinking, how hard can that be?

It is, because everywhere (including official guides by Veeam) some knowledge is taken for granted - you just do this and that, often black talk, when you don't even know what distribution to use.

So, having spent weeks on finding the bits and pieces, correcting and repeating the install over and over, I thought it could be a good idea to write it up for others, that are reluctant to fire up Linux servers:

Part 1 - Prepare the install of Linux
Part 2 - Install Linux on the server
Part 3 - Prepare the Linux server for Veeam
Part 4 - Create the immutable Veeam backup repository
Part 5 - Prepare for backup of the Linux server itself
Part 6 - Backup of the Linux server itself
Part 7 - Bare Metal Recovery of the Linux server
Part 8 - Tighten security on the Linux server (MFA/2FA)
Part 9 - Maintenance and deactivation/reactivation of MFA/2FA

Each and every step and command is documented for you to follow, supported by about 180 screenshots.

Please note, this is not a Linux crash-course; it is a guide for Veeam admins having immutable backup on the to-do list, but too little time for endless experiments.

Comments and suggestions are, of course, welcome.
robg
Expert
Posts: 171
Liked: 16 times
Joined: Aug 15, 2014 11:21 am
Full Name: Rob
Contact:

Re: Build an immutable backup repository - article series

Post by robg » 4 people like this post

Someone needs to make an immutable backup repository product/appliance so that busy (or uninterested) people can all benefit from the tech.
Keyser
Enthusiast
Posts: 50
Liked: 9 times
Joined: Feb 13, 2014 10:11 am
Contact:

Re: Build an immutable backup repository - article series

Post by Keyser » 2 people like this post

This is REALLY good stuff mate :-) Excellent work and thank you soooooo much.
mweissen13
Enthusiast
Posts: 93
Liked: 54 times
Joined: Dec 28, 2017 3:22 pm
Full Name: Michael Weissenbacher
Contact:

Re: Build an immutable backup repository - article series

Post by mweissen13 » 3 people like this post

Hey Gustav!
While I have no problem setting up a Linux repository (having >15 years of Linux experience under my belt) I can certainly imagine (and remember) that the first steps on Linux can be really hard. Your guide seems to be really perfectly done and should be easy for any Windows-only admin to follow. In the future I will certainly give that link to my co-workers when they are assigned with the task of setting up a Linux repo. Thanks for putting that much effort into it and sharing it with the world.
[OIT]Francis
Service Provider
Posts: 5
Liked: 2 times
Joined: Apr 27, 2018 7:15 am
Full Name: Francis Theys
Contact:

Re: Build an immutable backup repository - article series

Post by [OIT]Francis » 1 person likes this post

This is an excellent writeup ! Would you care to share this as a PDF as well ?
NickKulkarni
Enthusiast
Posts: 28
Liked: 7 times
Joined: Feb 08, 2021 6:11 pm
Full Name: Nicholas Kulkarni
Contact:

Re: Build an immutable backup repository - article series

Post by NickKulkarni » 1 person likes this post

Big Thanks to Gustav, will be looking into this ASAP.
Entropy
Novice
Posts: 7
Liked: 4 times
Joined: Nov 03, 2020 1:29 pm
Full Name: Ryan
Contact:

Re: Build an immutable backup repository - article series

Post by Entropy » 2 people like this post

Wow, great resource and impeccable timing - our hardware for this exact use case arrives this week :)
rweis
Veeam Software
Posts: 460
Liked: 71 times
Joined: Jun 13, 2011 7:46 pm
Full Name: Randy Weis
Location: Raleigh, NC, USA
Contact:

Re: Build an immutable backup repository - article series

Post by rweis »

And...it is behind a paywall.
Randy Weis
Enterprise SE, NA Strategic Accounts
rciscon
Influencer
Posts: 20
Liked: 2 times
Joined: Dec 14, 2010 8:48 pm
Full Name: Raymond Ciscon
Contact:

Re: Build an immutable backup repository - article series

Post by rciscon » 1 person likes this post

Gustav,

This is a marvelous post! I'm really quite interested in digging into this.

One question I do have before I expend the necessary man-hours required for this, but my initial skimming of the first couple of articles/steps seems to indicate that this configuration requires a PHYSICAL SERVER to implement this.

VMs are essentially free, while physical servers have a capital cost, if older hardware is unavailable.

I have seen no indication that this is indeed the case. Can you confirm that implementing this solution will require a physical server?

Thanks in advance!

Ray
joelwj
Lurker
Posts: 2
Liked: never
Joined: Jan 16, 2017 9:15 pm
Full Name: Joel Johnson
Contact:

Re: Build an immutable backup repository - article series

Post by joelwj »

Experts Exchange...
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Build an immutable backup repository - article series

Post by veremin » 2 people like this post

Can you confirm that implementing this solution will require a physical server?
Sure, as you cannot create actual immutable storage inside Virtual Machine - anyone having access to datastore can simply remove virtual disks, leaving no backups (even though they are immutable inside VM). Thanks!
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Build an immutable backup repository - article series

Post by Mildur » 1 person likes this post

Sure, as you cannot create actual immutable storage inside Virtual Machine - anyone having access to datastore can simply remove virtual disks, leaving no backups (even though they are immutable inside VM). Thanks!
Or with access to the vcenter, you can get really easily root access by rebooting the vm into linux Recovery Mode and reset the password. After booting up the vm, you can logon as root and delete the backups even from an iscsi mounted storage :)
Product Management Analyst @ Veeam Software
vbussiro
Enthusiast
Posts: 74
Liked: 5 times
Joined: Feb 18, 2009 10:05 pm
Contact:

Re: Build an immutable backup repository - article series

Post by vbussiro » 1 person likes this post

While I can easily understand the need to backup system drives of the hardened install, don't this Veeam Agent installation goes against the need to have the hardened as "simple" as possible ? Any additional software (even Veeam Backup agent) could lead to possible exploit ?
Gustav
Enthusiast
Posts: 45
Liked: 51 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav » 1 person likes this post

rweis wrote: Feb 07, 2022 2:26 pm And...it is behind a paywall.
No, it is not. Use the links provided.
Gustav
Enthusiast
Posts: 45
Liked: 51 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav »

rciscon wrote: Feb 07, 2022 2:50 pm Can you confirm that implementing this solution will require a physical server?
Yes, as already noted by others. It is expressed in the paragraph:
Applying immutability to your backup files hosted on a physical server introduces a virtual air gap in your backup chain, protecting the backup files from anything else than direct physical access. This way, the backup files will be protected from any attack caused by advanced malware or possible hackers.
And in Part 8:
Epilogue
Keep in mind, though, that everything falls to the ground if the bad guy (black hat) can get physical access to the Linux server.
However, to check out and be acquainted with most of the process, it is perfectly OK to use a VM. That's what I did, indeed for capturing many of the screenshots.
Gustav
Enthusiast
Posts: 45
Liked: 51 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav »

vbussiro wrote: Feb 08, 2022 10:57 am While I can easily understand the need to backup system drives of the hardened install, don't this Veeam Agent installation goes against the need to have the hardened as "simple" as possible ? Any additional software (even Veeam Backup agent) could lead to possible exploit ?
Sure. And one method - if you believe the system is secure - is to leave it as is and cease updating; seal it, so to say.
However, updates to any software (Ubuntu, Veeam Agent, Google MFA) may contain steps to tighten security, so it really is up to your judgement.
The use of the LTS version of Ubuntu and the very strict testing of Veeam software represents to me a good balance.
Gustav
Enthusiast
Posts: 45
Liked: 51 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav »

[OIT]Francis wrote: Feb 07, 2022 8:54 am This is an excellent writeup ! Would you care to share this as a PDF as well ?
I'll look into it. If success, I'll post a link.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Build an immutable backup repository - article series

Post by Gostev » 1 person likes this post

rweis wrote: Feb 07, 2022 2:26 pmAnd...it is behind a paywall.
Not for me. I was actually very surprised not to encounter one seeing links to Experts Exchange :D
Gustav
Enthusiast
Posts: 45
Liked: 51 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav » 4 people like this post

[OIT]Francis wrote: Feb 07, 2022 8:54 am Would you care to share this as a PDF as well?
Yes, these are online now at GitHub: https://github.com/GustavBrock/Veeam.Li ... e/main/pdf.
Gustav
Enthusiast
Posts: 45
Liked: 51 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: Build an immutable backup repository - article series

Post by Gustav » 5 people like this post

The complete series can now also be found on GitHub:

https://github.com/GustavBrock/Veeam.Linux
einhirn
Enthusiast
Posts: 53
Liked: 18 times
Joined: Feb 02, 2015 1:51 pm
Contact:

Re: Build an immutable backup repository - article series

Post by einhirn » 3 people like this post

Some of the aspects of that tutorial can also be accomplished using a wizard-like tool called "VeeamHubRepo":

https://github.com/tdewin/veeamhubrepo

It doesn't take care of 2FA and setting up the system backup, but other than that it seems to work fine to create a Repo on Ubuntu20 - it even allows you to format the repo disk and update the system via text-mode-UI, so you don't need to fiddle around with the command line if you don't want to.
Butha
Enthusiast
Posts: 39
Liked: 20 times
Joined: Oct 03, 2012 10:59 am
Full Name: Butha van der Merwe
Contact:

Re: Build an immutable backup repository - article series

Post by Butha » 1 person likes this post

Some background for my own use case, we are using a NetApp SAN with FAS disks via a dedicated physical server - presented as iscsi LUNS for this. Especially with reflink, there are many empty blocks left behind over time, and depending on your configuration, background cleanup/reclaiming space/trim might not be enabled (Usually NOT by default) Especially if you are using thin provisioned luns.

There are some additional configuration needed - slightly different depending on storage vendors, and also some configuration on the Ubuntu side. You could schedule or do manual "trim" to cleanup blocks, or you could mount the luns with different set of parameters to enable " background trimming" to happen - but some sites mention a performance penalty when doing this - I have not noticed any.

You might ask why worry about this? Reason is simple - over time deleted blocks will be left behind and you will have a discrepancy where veeam/ubuntu might report % free space on volumes, but in reality on the physical storage vendor you are 100% used - and you might wonder why LUNS start to go offline. There is no indication of this from the Veeam side or any logs to indicate this - as it's just reporting what ubuntu shows, but with thin provisioned luns and no space reclaim/trim enabled you will fill up the volumes on the physical storage. If it's configured correctly from the start it's one less " little unknown" thing that could cause issues.

I'll post an example from my /etc/fstab file with the flags we use to mount luns on our system, but please note this is not to be used as a general rule - speak to you storage vendor to find out how to enable "space reclaim/trim" etc - different names for it. The "discard" flag specifically enabled automatic background space reclaim. Also to reiterate - you have to make sure it's enabled on your storage vendor side, before mounting the luns. If you enable it afterwards, you might require a restart on the ubuntu server. Happy to post some commands we used to see if it's enabled, as well as manual trim commands for anybody interested.

Note : I have crossed out some UUID's and mount paths - which are unique to your enviroment.

UUID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx /mnt/xxxxxxx20 xfs discard,nosuid,nodev,nofail,x-gvfs-show 0 0
einhirn
Enthusiast
Posts: 53
Liked: 18 times
Joined: Feb 02, 2015 1:51 pm
Contact:

Re: Build an immutable backup repository - article series

Post by einhirn » 5 people like this post

Maybe I'm pointing out the obvious: When you are using SAN as storage for the hardened Linux repository, you need to isolate the SAN same as the linux repository server - if you care about the "hardened, immutable" aspect. Otherwise the SAN would be a weak link in the setup - why attack a hardened Linux repository if you can "just" delete LUNs on a SAN-Storage? Same goes for RAID controllers or subsystems with ethernet management ports etc.

Monitoring should be done from the hardened server, probably sending a mail or some other kind of outgoing alert if something is wrong...
vmJoe
VeeaMVP
Posts: 426
Liked: 103 times
Joined: Aug 02, 2011 1:06 pm
Full Name: Joe Gremillion
Location: Dallas, TX USA
Contact:

Re: Build an immutable backup repository - article series

Post by vmJoe » 3 people like this post

@einhirn you are correct about a SAN array needing to be hardened as well. A SAN array definitely makes a nice target for attackers as it controls server disks outside the server. It is easier to harden a DAS/JBOD/storage server as the disks are contained in the same server that hosts the OS.

Remember- immutability doesn’t equal hardened. They need to go hand in hand.

If you have a repository that is using a SAN array then it is highly recommended that the array be hardened.

There are many ways to hardened SAN arrays. This includes making sure that the management access point is on a private subnet, that all unnecessary services are disabled, unused port close, and restrict user access accounts. If you are using iSCSI make sure the storage traffic goes over a private non-routed subnet and that all initiators Authenticate to the LUNs via CHAP.
Joe Gremillion
NA Core Solutions Architect - Central region
_james
Enthusiast
Posts: 30
Liked: 6 times
Joined: Nov 15, 2018 3:51 pm
Contact:

Re: Build an immutable backup repository - article series

Post by _james »

Speaking about SAN/RAID controller hardening - is there much you can do to harden a RAID controller?
We are considering building an immutable repository, however im not sure how to configure RAID, via RAID controller or by making a software RAID within Linux.

Can someone advise should I choose one over the other, or maybe there is another alternative?
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Build an immutable backup repository - article series

Post by Mildur » 1 person likes this post

Hi James

Software raids are not recommended. You should use an enterprise-grade hardware-based RAID controller for your backup storage when ever possible.

In my opinion, there isn‘t much hardening todo here. Just make sure, no one can login remotely with root permissions or log in physically to the repository server. If he has physical access, he can destroy anything (or just take the disks with him). With remote access, he can delete the immutable flag or destroy the filesystem/partitions on the disks.

Thanks
Fabian
Product Management Analyst @ Veeam Software
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Build an immutable backup repository - article series

Post by albertwt » 1 person likes this post

wow, this is great, thank you for sharing the experience and the GitHub repo guys !
--
/* Veeam software enthusiast user & supporter ! */
Backup.Operator
Enthusiast
Posts: 64
Liked: 1 time
Joined: Oct 31, 2022 11:39 pm
Full Name: Backup Administrator
Contact:

Re: Build an immutable backup repository - article series

Post by Backup.Operator »

I have configured the Veeam Immutable backup using Linux as shown by @Gustav by using Ubuntu VM + iSCSI XFS LUN.

However, I wonder what would happens if somehow the Ubuntu Linux VM got deleted by the attacker or accidentally by human error.

How can I restore the VM from the Immutable repo?
:arrow: :mrgreen:
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Build an immutable backup repository - article series

Post by Regnor » 1 person likes this post

In that case you just connect the iSCSI to a new Ubuntu or Linux based system and add this as a repository in Veeam.
The bigger threat is, that the bad guys access your iSCSI storage and wipe your backup from there.
So make sure that management is separated or even physically disconnected.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Build an immutable backup repository - article series

Post by Gostev » 2 people like this post

Regnor wrote: Feb 14, 2023 2:29 pmThe bigger threat is, that the bad guys access your iSCSI storage and wipe your backup from there.
Or if the bad guys access your hypervisor where said VM runs... then not only you can nuke the VM, which is a smaller issue, but also just format the mounted iSCSI volume along with all immutable backups.

I would never recommend a hardened repository design that is a VM with mounted iSCSI storage. This just renders the whole idea of the hardened repository useless. Because you're going from the only possible hardened repository interaction being its physical console, to a huge attack surface of both the hypervisor and the storage array management interfaces and API.
Post Reply

Who is online

Users browsing this forum: xSOU1 and 189 guests