How-to needed: Linux backup restore for LUKS partition

[sl0n]

Hi All,

Is there a good detailed how-to for $subj? Thanks!

[HannesK]

Hello,
I'm not aware of any special "how-to" blog post or whatever you are looking for... the reason might be, that there is nothing special "how-to". At least I just treat machines with dm-crypt like any other machine: entire machine backup. set it and forget it.

The user guide has a few requirements for dm-crypt https://helpcenter.veeam.com/docs/agent ... tml?ver=40

You might want to try the forum search to find about problems.

Did you see any issues?

Best regards,
Hannes

[birdwaffle]

I would like to restore an Ubuntu host to bare metal without changing the partition layout or encryption. During the installation of Ubuntu, I checked the checkbox that enables LUKS encryption (Encrypt the LVM group with LUKS). The host has one disk. The OS version is Ubuntu 22.04.1 LTS.

This question was asked here, but it doesn't have an answer yet. I'm trying to write a how-to but I have run into a snag myself.

I would like to restore the host as it was before, but after the restore, I have a different partition layout and LUKS encryption is gone. The backup was made using the "Entire computer" option, but I don't have the option of restoring the "entire computer". I have to choose "Restore volume" and then I get the screen where disks and volumes are mapped.

Initially, it looks like this:

Code: Select all

              CURRENT SYSTEM           │              IN BACKUP
                                       │
     Device       Restore      Size    │  Device       Size    Usage
                                       │
     sda                       127.0G  │  mapper/dm... 123.9G
                                       │  sda          127.0G
                                       │   sda1        1.04G   /boot/efi...
                                       │   sda2        2.00G   /boot (ext4)
                                       │  ubuntu-vg    123.9G
                                       │   ubuntu-lv   61.96G  / (ext4)

This is where it is not clear what I need to do. The release notes state

If you restore data from an encrypted backup and create a LUKS volume using the product graphical
user interface, the product creates a LUKS2 volume. The LUKS2 volume can be decrypted only on Linux
kernel version 4.19 or later.

But I only seem to have two options, which are both "bad".
Option 1:
I can start by creating a LUKS volume on sda. The problem here is that LUKS will span over the entire disk, not allowing me to create the un-encrypted boot partitions (sda1, sda2). I believe the system would not boot.

Option 2:
I can map sda to sda. This will lead to this:

Code: Select all

              CURRENT SYSTEM           │              IN BACKUP
                                       │
     Device       Restore      Size    │  Device       Size    Usage
                                       │
     sda                       127.0G  │  mapper/dm... 123.9G
      sda1        sda1 (/bo... 1.04G   │  sda          127.0G
      sda2        sda2 (/boot) 2.00G   │   sda1        1.04G   /boot/efi...
      free                     123.9G  │   sda2        2.00G   /boot (ext4)
                                       │  ubuntu-vg    123.9G
                                       │   ubuntu-lv   61.96G  / (ext4)

From there, I can proceed to map the "mapper/dm_crypt-0" entry into the free space on sda. The result is:

Code: Select all

              CURRENT SYSTEM           │              IN BACKUP
                                       │
     Device       Restore      Size    │  Device       Size    Usage
                                       │
     sda                       127.0G  │  mapper/dm... 123.9G
      sda1        sda1 (/bo... 1.04G   │  sda          127.0G
      sda2        sda2 (/boot) 2.00G   │   sda1        1.04G   /boot/efi...
      sda3 (lvm)               123.9G  │   sda2        2.00G   /boot (ext4)
      free                     19.00M  │  ubuntu-vg    123.9G
     ubuntu-vg                 123.9G  │   ubuntu-lv   61.96G  / (ext4)
      ubuntu-lv   ubuntu--v... 61.96G  │
      free                     61.96G  │

This will lead to a system that is bootable and works but LUKS is gone. Ubuntu will boot, but it will not ask for a password anymore. It will be fully decrypted. When I start to map sda <--> sda, there is no way anymore to create a LUKS volume anywhere on sda. It doesn't allow me to map sda1, sda2 and create a new LUKS volume.

[HannesK]

Hello,
yes, that's expected behavior. From the other thread, I could not identify the problem... During backup, the software does not know anything about the encryption (explained here). The software also does not know the password which you type in during boot.

Without the password, the software cannot create any encrypted device. The encrypted partitions need to be created manually outside Veeam. LUKS cli tools should be on the recovery media (I can hopefully check that next week)

Agree, it might be a good idea to document how to do that

Best regards,
Hannes

[birdwaffle]

Thank you for your reply! I suspected that during the backup, Veeam is not aware of LUKS if the LUKS container/partition is opened. This seems completely fine to me as it allows for compression and fast clone to work, and the backup can be encrypted through Veeam anyway.

I have had some success during my testing. Unfortunately, I do not have the experience that allows me to re-create the LUKS/lvm partition from command line. What I have been able to do (with success) is:

1. On bare metal, I install Ubuntu. This will re-create all the partitions, LUKS and LVM.
2. In Veeam recovery media UI, the LUKS partition is recognized ("Found 1 crypto devices"), and it can be opened. I map sda1, sda2 and the LVM (either ubuntu-vg or ubuntu-lv, both will work). This will lead to a system with the same partition layout as before the bare metal restore but unfortunately, it will not boot because there is a new LUKS partition with a new uuid but Ubuntu is looking for the old LUKS partition with a different uuid during boot.
3. To fix the uuids, I boot into Ubuntu desktop live media. I open the LUKS partition and mount the file systems (sda1, sda2 and lvm) and chroot into the system. Now I can
   • update the LUKS partiton uuid (sda3) in /etc/crypttab
   • In case I chose to map only ubuntu-lv instead of the entire LVM, I suspect that the recovery media creates a new lvm. In that case, I need to update the lvm uuid in /etc/fstab
   • upate GRUB config so it can find the LUKS partition, with update-initramfs

Now the system boots and works as usual with LUKS. It makes sense that the recovery media would have LUKS cli tools because it can open the LUKS container. I have been able to open LUKS in command line and find the lvm:

Code: Select all

veeamuser@veeam-recovery-iso:~$ sudo cryptsetup luksOpen /dev/sda3 dm_crypt-0
Enter passphrase for /dev/sda3:
veeamuser@veeam-recovery-iso:~$ sudo vgscan
  Found volume group "ubuntu-vg" using metadata type lvm2
veeamuser@veeam-recovery-iso:~$

I believe somebody with knowledge of creating the LUKS partition in command line could find a simpler, better solution than the one I have found so far. I believe all of it could be done using only the recovery media. There should be no need to re-install ubuntu.

Another idea, that just came to me: It might be simpler to simply change the partition/lvm uuids rather than update the config files with the new uuids.

[birdwaffle]

I have put another hour of testing into this, and I have come up with something that is, in my opinion much simpler, much better. Re-installation of Ubuntu is no longer necessary, and everything can be done with the Veeam recovery media. Admittedly, this is still not what I would consider "simple", but it is mostly copy & paste now.

I also believe that eventually the Veeam recovery media UI could do all of this for us so that it could offer the option to "Restore computer" and it would just work, without the need for manual steps to be taken.

I hope this helps!

1. restore the EFI, boot partitions
   1. boot into recovery media
   2. choose "Restore volumes"
   3. map the device that is the operating system disk (here: sda <--> sda)
      It should look like this

      Code: Select all

                    CURRENT SYSTEM           │              IN BACKUP
                                             │
           Device       Restore      Size    │  Device       Size    Usage
                                             │
           sda                       127.0G  │  mapper/dm... 123.9G
            sda1        sda1 (/bo... 1.04G   │  sda          127.0G
            sda2        sda2 (/boot) 2.00G   │   sda1        1.04G   /boot/efi...
            free                     123.9G  │   sda2        2.00G   /boot (ext4)
                                             │  ubuntu-vg    123.9G
                                             │   ubuntu-lv   61.96G  / (ext4)

   4. proceed with the restore
   5. close UI, exit to shell, but do not shutdown or reboot
2. re-create the LUKS container
   1. create a new partition on the OS disk

      Code: Select all

      sudo gdisk /dev/sda

      Type "n" to create a new partition and use all default values.
      Type "w" to write the changes to disk.
      Optionally, you can now confirm that the partition has been created (here: sda3)

      Code: Select all

      veeamuser@veeam-recovery-iso:~$ lsblk /dev/sda
      NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
      sda      8:0    0  127G  0 disk
      ├─sda1   8:1    0  1.1G  0 part
      ├─sda2   8:2    0    2G  0 part
      └─sda3   8:3    0  124G  0 part
      veeamuser@veeam-recovery-iso:~$

   2. encrypt the new partition
      You can choose a new password here.
      A note on encryption: Some people may recommend at this point to either securely wipe the disk using ATA/NVMe commands, or to fill the partition with random data before creating the container, or to fill the container with zeros after its creation. However, I did not do any of this during my testing. It is explained here: https://wiki.archlinux.org/title/Dm-cry ... reparation

      Code: Select all

      sudo cryptsetup luksFormat /dev/sda3

      Note that, as of now (recovery media 5.0.0, Ubuntu server 22.04.1), when using default values, this will re-create the LUKS container, in my opinion, exactly as it was created by the Ubuntu installer. For instance, cipher will be "aes-xts-plain64" before and after the restore. Optionally, you can check for that with:

      Code: Select all

      sudo cryptsetup luksDump /dev/sda3

3. restore LVM volume group and logical volume
   1. enter UI with

      Code: Select all

      sudo veeam

   2. once again, choose "Restore volumes"
      This time, the UI will report "Found 1 crypto devices. Do you want to decrypt them all?". This is the new LUKS container.
   3. choose "Yes" to open the container and enter the password chosen above.
   4. map the two mapper devices (here: "luks-bf..." <--> "mapper/dm_crypt-0").
      This should automatically map the LVM volume group "ubuntu-vg" and the logical volume "ubuntu-lv". The good thing here is that, even though Veeam, during the backup creation, does not have knowledge of the LUKS container, it does seem to have knowledge of the mapper device "dm_crypt-0" and that information is contained in the backup.
      It should now look like this:

      Code: Select all

                    CURRENT SYSTEM           │              IN BACKUP
                                             │
           Device       Restore      Size    │  Device       Size    Usage
                                             │
           sda                       127.0G  │  mapper/dm... 123.9G
            sda1                     1.04G   │  sda          127.0G
            sda2                     2.00G   │   sda1        1.04G   /boot/efi...
            sda3                     123.9G  │   sda2        2.00G   /boot (ext4)
             luks-bf...              123.9G  │  ubuntu-vg    123.9G
           ubuntu-vg                 123.9G  │   ubuntu-lv   61.96G  / (ext4)
            ubuntu-lv   ubuntu--v... 61.96G  │
            free                     61.96G  │

   5. once again, proceed with the restore
   6. close UI, exit to shell, but do not shutdown or reboot
      All data should now be restored, but we still cannot boot into Ubuntu because the UUID of the LUKS partition has changed and it is now different from what Ubuntu expects during boot.
4. change LUKS partition UUID
   1. get the original LUKS partition UUID
      Since I just opened the LUKS container in the UI and restored the LVM, both should still be opened and available.
      mount the LVM:

      Code: Select all

      sudo mount /dev/mapper/ubuntu--vg-ubuntu--lv /mnt

      get UUID

      Code: Select all

      veeamuser@veeam-recovery-iso:~$ cat /mnt/etc/crypttab
      dm_crypt-0 UUID=569f8381-74cb-4930-ad22-a113863c2458 none luks
      veeamuser@veeam-recovery-iso:~$

      My UUID is: 569f8381-74cb-4930-ad22-a113863c2458
      unmount LVM

      Code: Select all

      sudo umount /dev/mapper/ubuntu--vg-ubuntu--lv

      close LVM

      Code: Select all

      sudo vgchange -a n

   2. set LUKS partition UUID

      Code: Select all

      sudo cryptsetup luksUUID --uuid 569f8381-74cb-4930-ad22-a113863c2458 /dev/sda3

      Optionally, you can now confirm that the UUID has been set

      Code: Select all

      sudo cryptsetup luksUUID /dev/sda3

      close the LUKS container. The name might be different each time.

      Code: Select all

      sudo cryptsetup luksClose /dev/mapper/luks-bf3f0eac-adce-4357-adaf-45a403a4ffba\\x0a

5. shutdown recovery media system

   Code: Select all

   sudo shutdown -h now

6. remove recovery media
7. boot from disk

[birdwaffle]

I would like to add, for the backup job:

I just treat machines with dm-crypt like any other machine: entire machine backup. set it and forget it.

This is what I did during testing, in order to keep it simple. Ultimately, this is not what I need, because data on other disks, that are not the operating system disk, is backed up by another job. I don't know if it would even be possible to do it in one job, as the other disks contain ZFS, which cannot be backed up with snapshots. The job would need to run using two different backup modes: Snapshots for the OS volumes and file level backup for ZFS pool. Even if it was possible, it is not what I want. The question becomes:

What needs to be backed up, when the intention is to just back up the OS disk?

My testing shows that, in the backup job settings, instead of "Entire computer", I can select "Volume level backup". In "Objects \ Objects to back up \ Add \ Device \ Path to block device", I need to add (see screenshot below)

• the OS disk device
  In my case, it is "/dev/sda"
• and the mapper device for the LUKS container
  This should always be "/dev/mapper/dm_crypt-0", if the container was created by the Ubuntu installer.

This will backup everything that is needed to restore the entire operating system disk. It is not necessary to explicitly include the LVM (/dev/mapper/ubuntu--vg-ubuntu--lv).

[HannesK]

Hello,
first of all: thanks for describing the solution. That saved me some time to go through it Yes, the second way is what I had in mind and yes, with UUIDs, it's all a bit complicated.

I don't know if it would even be possible to do it in one job, as the other disks contain ZFS

correct, that needs two jobs.

What needs to be backed up, when the intention is to just back up the OS disk?

as far as I see, you already gave the answer Volume level backup is the most efficient way, yes. Using /dev/mapper/ubuntu--vg-ubuntu--lv as LVM source also works (and sounds more logical to me to take). Theoretically, also file-based backup would work for backup. But it's slower and no bare-metal restore available (which you like to use as far as I see).

Agree, there are things we could simplify on Veeam side. I added +1 for the "entire computer" restore feature request. On the other hand, there are other features that are requested much more than improvements around dmcrypt / luks. How many machines with dm-crypt / luks to you manage?

Best regards,
Hannes

[birdwaffle]

Agree, there are things we could simplify on Veeam side. I added +1 for the "entire computer" restore feature request. On the other hand, there are other features that are requested much more than improvements around dmcrypt / luks. How many machines with dm-crypt / luks to you manage?

I agree that it should not be a high priority because we now have it written down what needs to be done during restore. This is completely fine with me. I have just one server with Ubuntu and LUKS. Other than that, we don't really use bare metal restore. Everything is inside of VMs and is restored as VM. In the unlikely event that an entire virtualization host goes down, I would restore the VMs to a different host and then sometime later, re-install the physical server and migrate the VMs back eventually.

I will include a few more things I have come across during my testing as I moved the testing to the physical server. (in the hopes that, if somebody later googles this, they will find the answer here).

The re-created LUKS container has a different sector size than the original conatiner
Above, I wrote that the LUKS container that I create with the Veeam recovery media, has the same properties as the original container that is created by the Ubuntu installer. This was not entirely true. I overlooked the sector size of the container. My understanding of this is that LUKS writes changes to the disk in chunks of the size that is the container's 'sector size'. This is similar to the cluster size in Microsoft file systems or block size in Linux file systems. Setting a sector size for the container that does not match the physical disk layout, can lead to decreased performance.
In my case I had before the restore:

Code: Select all

Data segments:
  0: crypt
        offset: 16777216 [bytes]
        length: (whole device)
        cipher: aes-xts-plain64
        sector: 4096 [bytes]

and after:

Code: Select all

Data segments:
  0: crypt
        offset: 16777216 [bytes]
        length: (whole device)
        cipher: aes-xts-plain64
        sector: 512 [bytes]

You can check for the sector size with:

Code: Select all

sudo cryptsetup luksDump /dev/sda3

In my case the disk is "512e" that is, a disk that reports as Sector size (logical/physical): 512 bytes / 4096 bytes.
There are several ways to fix this:

• create the LUKS container with a newer version of cryptsetup than the version that comes with the recovery media
  For instance, you could boot into Ubuntu desktop live and create the container in Ubuntu live.
• for a solution, that is entirely contained to using only the recovery media:
  You can force the correct sector size during the container creation. But if the disk internally works with 4K sectors but reports a sector size of 512 bytes (512e disks), cryptsetup will return an error.

  Code: Select all

  sudo cryptsetup luksFormat --sector-size=4096 /dev/sda3
  Device size is not aligned to requested sector size.

  To fix this, the end of the partition needs to be aligned, not just the beginning of the partition.
  • if you already created the partition that will be the LUKS container, delete it. Remember, the third partition is the LUKS partition (see above). Replace sda with the device name for your operating system disk.

    Code: Select all

    sudo sgdisk --delete=3 /dev/sda

  • Now, create a new partition that has the start and end aligned.

    Code: Select all

    DISK='/dev/sda'
    start_position=$(sudo sgdisk --first-aligned-in-largest /dev/sda $DISK)
    end_position=$(sudo sgdisk --end-of-largest /dev/sda $DISK)
    aligned_end_position=$((end_position - ($end_position + 1) % 2048))
    sudo sgdisk --set-alignment=2048 --new=3:$start_position:$aligned_end_position /dev/sda

  • optionally, you can now confirm that the partition was created

    Code: Select all

    sudo gdisk -l $DISK

  • now cryptsertup will allow the creation of a 4096 byte sector size container

    Code: Select all

    sudo cryptsetup luksFormat --sector-size=4096 /dev/sda3

    confirm that the sector size is 4096

    Code: Select all

    veeamuser@veeam-recovery-iso:~$ sudo cryptsetup luksDump /dev/sda3 | grep sector
            sector: 4096 [bytes]
    veeamuser@veeam-recovery-iso:~$

  • now proceed with the restore as usual (see above)