Hardened Repo with Encryption

[vertices]

Greetings,

We are in the process of designing our migration away from Windows Repos to Ubuntu Hardened Repos. I've found some guides, such as the Starwind guide, but one thing that no one talks about is full disk encryption.

We have a requirement to encrypt all backups at rest. We do not desire to use Veeam built-in job encryption. We have been accomplishing this with 64K ReFs and BitLocker. It's been working great. But moving away from Windows, well now we need to figure out the disk encryption aspect.

What is everyone else doing to encrypt the XFS volume? Are there any guides? I've been having trouble finding a good one. I'm not new to linux, but I know enough to to manage them and keep them going, but definitely need to brush up and learn a bit more.

Edit~~ I should add that fast clone is a definite requirement, along with immutability. So whatever disk encryption solution is used, it must still have those features.

Thanks!

[HannesK]

Hello,
not directly XFS, but the layer below... if I use disk encryption on Linux, then I use dm-crypt since more than 15 years. https://en.wikipedia.org/wiki/Dm-crypt

Best regards,
Hannes

[vertices]

Well so far not having any fun or luck. Trying to blend together the following articles and want to smash my face into the keyboard.

https://oak-tree.tech/blog/lvm-luks
https://www.starwindsoftware.com/blog/v ... ory-part-1

The opening of the encrypted volume group and volume, to put an XFS file system on it is making me cry. It's hard to understand.

[vertices]

If anyone knows what they are doing with this stuff. I have a separate disk, /dev/sdb that is 72.5TB. I would love a list of commands to create a proper encrypted XFS volume with 4k blocks and block cloning enabled. Below is my latest that I've been trying to work with. Starting fresh:

sudo pvcreate /dev/sdb
sudo vgcreate veeamrepo2-vg1 /dev/sdb
sudo lvcreate -L +72.50T --name veeamrepo2-lv1 veeamrepo2-vg1
sudo cryptsetup luksFormat /dev/veeamrepo2-vg1/veeamrepo2-lv1

then the command below is where it gets hard for me to understand. It wants me to open /dev/veeamrepo2-vg1/veeamrepo2-lv1 but I have to name it something else?!? so I name it veeamrepo2-lv1-opened and it opens after I enter the luks phrase.

sudo cryptsetup open /dev/veeamrepo2-vg1/veeamrepo2-lv1 veeamrepo2-lv1-opened

I find it odd though that in /dev/mapper, as soon as I create the volume group and volume it shows there as "veeamrepo2--vg1-veeamrepo2--lv1". But then the guide makes it sounds like I am supposed to open "veeamrepo2--vg1-veeamrepo2--lv1" up as a new name, in this case "veeamrepo2-lv1-opened". I then was able to:

sudo mkfs.xfs -b size=4096 -m reflink=1,crc=1 /dev/mapper/veeamrepo2-lv1-opened

And that worked. It seems to create the filesystem just fine. But it seems odd that now I have in /dev/mapper:

drwxr-xr-x 2 root root 120 Jul 28 20:03 .
drwxr-xr-x 21 root root 4400 Jul 28 20:03 ..
crw------- 1 root root 10, 236 Jul 28 19:36 control
lrwxrwxrwx 1 root root 7 Jul 28 19:36 ubuntu--vg-ubuntu--lv -> ../dm-1
lrwxrwxrwx 1 root root 7 Jul 28 20:06 veeamrepo2-lv1-opened -> ../dm-2
lrwxrwxrwx 1 root root 7 Jul 28 20:03 veeamrepo2--vg1-veeamrepo2--lv1 -> ../dm-0

That just looks wrong to me. the veeamrepo2-lv1-opened is just an opened device of veeamrepo2--vg1-veeamrepo2--lv1?


Anywho, if anyone can help me get this encryption going I would very much appreciate it. I don't want to give up and go back to Windows but if I can't get this volume encryption down I'll have no choice. I hate disk mgmt in Linux lol. Thanks.

[vertices]

If it helps, this is where I'm at:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 70.3M 1 loop /snap/lxd/21029
loop1 7:1 0 55.4M 1 loop /snap/core18/1944
loop2 7:2 0 69.9M 1 loop /snap/lxd/19188
loop3 7:3 0 32.3M 1 loop /snap/snapd/12704
loop4 7:4 0 31.1M 1 loop /snap/snapd/10707
loop5 7:5 0 55.5M 1 loop /snap/core18/2074
sda 8:0 0 256G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 254.5G 0 part
└─ubuntu--vg-ubuntu--lv 253:1 0 225G 0 lvm /
sdb 8:16 0 72.5T 0 disk
└─veeamrepo2--vg1-veeamrepo2--lv1 253:0 0 72.5T 0 lvm
└─veeamrepo2-lv1-opened 253:2 0 72.5T 0 crypt
sdc 8:32 1 14.9G 0 disk
└─sdc1 8:33 1 14.9G 0 part
sr0 11:0 1 1024M 0 rom

[vertices]

I think I got it? it seems to work and do what I want. From the above I then continued:

sudo cryptsetup luksUUID /dev/veeamrepo2-vg1/veeamrepo2-lv1
which gave me the UUID of the encrypted volume

I edited /etc/crypttab to request an unlock at boot requiring me to enter a key. I added the following line:
veeamrepo2-lv1-opened UUID=9cdbabaa-4b7c-41f1-95a7-4b9c8b03a3e5 none luks,discard

I made a directory to mount the unencrypted vol:
sudo mkdir /mnt/veeamrepo2-lv1-opened

I then edited /etc/fstab to mount it after decrypting at boot by adding the following line:
/dev/mapper/veeamrepo2-lv1-opened /mnt/veeamrepo2-lv1-opened xfs nosuid,nodev,nofail,x-gvfs-show 0 0

And after a reboot it asks me to enter the passphrase, I do, it continues to boot and mounts normally. So I guess that's good?

[vertices]

Ok, after playing around and testing a couple of different ways, I have found this to be the simplest method (that I know of) if you are like me with a server, with a raid controller and a bunch of disks. For this situation, I just want to dedicate the entire large virtual disk to Veeam. We won’t be setting up LVM or anything on it. We will build this big and max it out so we don't have to mess with it later. Additionally, I won’t be saving any keys anywhere on the server. Since this is in a datacenter environment, it never reboots on its own. We will enter the key from the console at each boot via drac (yes protected with a very strong pass and up to date firmware but I just can't live without a drac, we will be uploading to Wasabi with SOBR anyways).

1. Set up 2 virtual disks on your raid controller spanning your raid6 vol or whatever raid you are using. First one small for OS, second one huge for your repo. Install Ubuntu on your OS disk, doing nothing with your data disk at install.
2. Once installed, make sure we have what we need ‘sudo apt install cryptsetup’
3. Now do a ‘lsblk’ command to see your drives. If you only have 2, you should see sda disk with partitions (sda1, sda2, etc.) under it, and then a huge empty sdb with no partitions under it.
4. Lets put a new partition on sdb. ‘sudo gdisk /dev/sdb’. Hit ‘n’ for new partition and just accept the default and let it use the whole disk.
5. If you ‘lsblk’ again now, you should see sdb1 as a partition underneath the disk sdb
6. Lets encrypt the partition we just made: ‘sudo cryptsetup -y -v luksFormat /dev/sdb1’ It will ask you for a passphrase. Enter it and don’t lose it.
7. Now lets get the UUID of the encrypted partition: ‘sudo cryptsetup luksUUID /dev/sdb1’. Output will be similar to ‘8a10e7f5-db7d-44e0-b673-56442006061b’ copy this ID
8. Now lets temporarily mount this encrypted partition as ‘veeamdata’ (or whatever you prefer to use for a name): ‘sudo cryptsetup luksOpen /dev/sdb1 veeamdata’. Enter your passphrase.
9. Time to put XFS on it: ‘sudo mkfs.xfs -b size=4096 -m reflink=1,crc=1 /dev/mapper/veeamdata’
10. Check it: ‘ls -l /dev/mapper/veeamdata’
11. Check mapping status: ‘sudo cryptsetup -v status veeamdata’
12. Now lets make a spot to mount it. ‘sudo mkdir /mnt/veeamdata’
13. Lets edit /etc/crypttab so it decrypts it at boot after asking us for the phrase. Do ‘sudo vi /etc/crypttab’ and add the following line ‘veeamdata UUID=8a10e7f5-db7d-44e0-b673-56442006061b none luks,discard’ making sure to use the same UUID that discovered in step 7.
14. Next we need to edit /etc/fstab to mount it at boot. ‘sudo vi /etc/fstab’ and add this line ‘/dev/mapper/veeamdata /mnt/veeamdata xfs nosuid,nodev,nofail,x-gvfs-show 0 0’
15. Check your mounts now with “df -h”, you should not see it.
16. Now ‘sudo mount -a’ to mount everything, and if you ‘df -h’ again you should see it.
17. Reboot
18. Now looking at the console via your well protected iDRAC or similar, at boot it should ask you for a passphrase for it. Enter it and it will continue booting and mount it as /mnt/veeamdata. Verify with a df -h. If all went well you should see it.
19. Now we need to add the Veeam repo user ‘sudo useradd -m -d /home/veeamrepouser/ -s /bin/bash -G sudo veeamrepouser’
20. Give it a password with ‘passwd veeamrepouser’
21. Change ownership of /mnt/veeamdata with ‘sudo chown -R veeamrepouser: veeamrepouser /mnt/veeamdata’
22. Change permissions to it with ‘sudo chmod 700 /mnt/veeamdata’
23. Now drink a pint and feel good about yourself!
24. You still need to remove sudo rights, and add MFA if you want. All of that is in the articles I’m listing below. But at this point you should be good to go ahead and get it added as a linux server in Veeam B&R.

I at first made it far too complicated following the first article and using LVM. I then found the second and that helped. I ended up blending all of them together to come up with the above simplified steps. If anyone has a better way, or any corrections I’d love to hear them! This was the one piece of the puzzle that was escaping me with these hardened repos but now we are good to go. Hope this helps someone else and saves them the headache I went through today!

Thanks to these pages:
https://oak-tree.tech/blog/lvm-luks
https://dev-notes.eu/2020/12/LUKS-Encry ... ntu-20.04/
https://www.starwindsoftware.com/blog/v ... ory-part-1
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[HannesK]

yep, looks good. LVM is optional and in many situations unnecessary.

[rjiupat]

Your way looks good, and this doesn't address LUKS, but just to throw some more information out there the guide I found most helpful was this:
https://nolabnoparty.com/en/veeam-v11-h ... lity-pt-1/

[vertices]

Yeah I saw that one as well. I just had trouble getting the encryption down.

Also, I don't see how to edit a post, perhaps that's disabled. But above in step 4 it should have been 'sudo gdisk /dev/sdb' as using fdisk will only give 2tb as it only supports MBR and not GPT.

[HannesK]

yes, edit gets disabled after some time. I fixed step 4 for you