-
- Influencer
- Posts: 12
- Liked: never
- Joined: Jan 23, 2017 3:40 pm
- Contact:
How-to needed: Linux backup restore for LUKS partition
Is there a good detailed how-to for $subj? Thanks!
-
- Product Manager
- Posts: 14585
- Liked: 2959 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: How-to needed: Linux backup restore for LUKS partition
I'm not aware of any special "how-to" blog post or whatever you are looking for... the reason might be, that there is nothing special "how-to". At least I just treat machines with dm-crypt like any other machine: entire machine backup. set it and forget it.
The user guide has a few requirements for dm-crypt https://helpcenter.veeam.com/docs/agent ... tml?ver=40
You might want to try the forum search to find about problems.
Did you see any issues?
Best regards,
Hannes
-
- Influencer
- Posts: 10
- Liked: 3 times
- Joined: Jun 19, 2019 9:55 am
- Contact:
[MERGED] Bare metal restore of Ubuntu host (LUKS)
This question was asked here, but it doesn't have an answer yet. I'm trying to write a how-to but I have run into a snag myself.
I would like to restore the host as it was before, but after the restore, I have a different partition layout and LUKS encryption is gone. The backup was made using the "Entire computer" option, but I don't have the option of restoring the "entire computer". I have to choose "Restore volume" and then I get the screen where disks and volumes are mapped.
Initially, it looks like this:
Code: Select all
CURRENT SYSTEM │ IN BACKUP
│
Device Restore Size │ Device Size Usage
│
sda 127.0G │ mapper/dm... 123.9G
│ sda 127.0G
│ sda1 1.04G /boot/efi...
│ sda2 2.00G /boot (ext4)
│ ubuntu-vg 123.9G
│ ubuntu-lv 61.96G / (ext4)
But I only seem to have two options, which are both "bad".If you restore data from an encrypted backup and create a LUKS volume using the product graphical
user interface, the product creates a LUKS2 volume. The LUKS2 volume can be decrypted only on Linux
kernel version 4.19 or later.
Option 1:
I can start by creating a LUKS volume on sda. The problem here is that LUKS will span over the entire disk, not allowing me to create the un-encrypted boot partitions (sda1, sda2). I believe the system would not boot.
Option 2:
I can map sda to sda. This will lead to this:
Code: Select all
CURRENT SYSTEM │ IN BACKUP
│
Device Restore Size │ Device Size Usage
│
sda 127.0G │ mapper/dm... 123.9G
sda1 sda1 (/bo... 1.04G │ sda 127.0G
sda2 sda2 (/boot) 2.00G │ sda1 1.04G /boot/efi...
free 123.9G │ sda2 2.00G /boot (ext4)
│ ubuntu-vg 123.9G
│ ubuntu-lv 61.96G / (ext4)
Code: Select all
CURRENT SYSTEM │ IN BACKUP
│
Device Restore Size │ Device Size Usage
│
sda 127.0G │ mapper/dm... 123.9G
sda1 sda1 (/bo... 1.04G │ sda 127.0G
sda2 sda2 (/boot) 2.00G │ sda1 1.04G /boot/efi...
sda3 (lvm) 123.9G │ sda2 2.00G /boot (ext4)
free 19.00M │ ubuntu-vg 123.9G
ubuntu-vg 123.9G │ ubuntu-lv 61.96G / (ext4)
ubuntu-lv ubuntu--v... 61.96G │
free 61.96G │
-
- Product Manager
- Posts: 14585
- Liked: 2959 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Bare metal restore of Ubuntu host (LUKS)
yes, that's expected behavior. From the other thread, I could not identify the problem... During backup, the software does not know anything about the encryption (explained here). The software also does not know the password which you type in during boot.
Without the password, the software cannot create any encrypted device. The encrypted partitions need to be created manually outside Veeam. LUKS cli tools should be on the recovery media (I can hopefully check that next week)
Agree, it might be a good idea to document how to do that
![Smile :-)](https://forums.veeam.com/images/smilies/icon_smile.gif)
Best regards,
Hannes
-
- Influencer
- Posts: 10
- Liked: 3 times
- Joined: Jun 19, 2019 9:55 am
- Contact:
Re: Bare metal restore of Ubuntu host (LUKS)
I have had some success during my testing. Unfortunately, I do not have the experience that allows me to re-create the LUKS/lvm partition from command line. What I have been able to do (with success) is:
- On bare metal, I install Ubuntu. This will re-create all the partitions, LUKS and LVM.
- In Veeam recovery media UI, the LUKS partition is recognized ("Found 1 crypto devices"), and it can be opened. I map sda1, sda2 and the LVM (either ubuntu-vg or ubuntu-lv, both will work). This will lead to a system with the same partition layout as before the bare metal restore but unfortunately, it will not boot because there is a new LUKS partition with a new uuid but Ubuntu is looking for the old LUKS partition with a different uuid during boot.
- To fix the uuids, I boot into Ubuntu desktop live media. I open the LUKS partition and mount the file systems (sda1, sda2 and lvm) and chroot into the system. Now I can
- update the LUKS partiton uuid (sda3) in /etc/crypttab
- In case I chose to map only ubuntu-lv instead of the entire LVM, I suspect that the recovery media creates a new lvm. In that case, I need to update the lvm uuid in /etc/fstab
- upate GRUB config so it can find the LUKS partition, with update-initramfs
Code: Select all
veeamuser@veeam-recovery-iso:~$ sudo cryptsetup luksOpen /dev/sda3 dm_crypt-0
Enter passphrase for /dev/sda3:
veeamuser@veeam-recovery-iso:~$ sudo vgscan
Found volume group "ubuntu-vg" using metadata type lvm2
veeamuser@veeam-recovery-iso:~$
Another idea, that just came to me: It might be simpler to simply change the partition/lvm uuids rather than update the config files with the new uuids.
-
- Influencer
- Posts: 10
- Liked: 3 times
- Joined: Jun 19, 2019 9:55 am
- Contact:
Re: Bare metal restore of Ubuntu host (LUKS)
I also believe that eventually the Veeam recovery media UI could do all of this for us so that it could offer the option to "Restore computer" and it would just work, without the need for manual steps to be taken.
I hope this helps!
- restore the EFI, boot partitions
- boot into recovery media
- choose "Restore volumes"
- map the device that is the operating system disk (here: sda <--> sda)
It should look like thisCode: Select all
CURRENT SYSTEM │ IN BACKUP │ Device Restore Size │ Device Size Usage │ sda 127.0G │ mapper/dm... 123.9G sda1 sda1 (/bo... 1.04G │ sda 127.0G sda2 sda2 (/boot) 2.00G │ sda1 1.04G /boot/efi... free 123.9G │ sda2 2.00G /boot (ext4) │ ubuntu-vg 123.9G │ ubuntu-lv 61.96G / (ext4)
- proceed with the restore
- close UI, exit to shell, but do not shutdown or reboot
- re-create the LUKS container
- create a new partition on the OS disk
Type "n" to create a new partition and use all default values.
Code: Select all
sudo gdisk /dev/sda
Type "w" to write the changes to disk.
Optionally, you can now confirm that the partition has been created (here: sda3)Code: Select all
veeamuser@veeam-recovery-iso:~$ lsblk /dev/sda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 127G 0 disk ├─sda1 8:1 0 1.1G 0 part ├─sda2 8:2 0 2G 0 part └─sda3 8:3 0 124G 0 part veeamuser@veeam-recovery-iso:~$
- encrypt the new partition
You can choose a new password here.
A note on encryption: Some people may recommend at this point to either securely wipe the disk using ATA/NVMe commands, or to fill the partition with random data before creating the container, or to fill the container with zeros after its creation. However, I did not do any of this during my testing. It is explained here: https://wiki.archlinux.org/title/Dm-cry ... reparationNote that, as of now (recovery media 5.0.0, Ubuntu server 22.04.1), when using default values, this will re-create the LUKS container, in my opinion, exactly as it was created by the Ubuntu installer. For instance, cipher will be "aes-xts-plain64" before and after the restore. Optionally, you can check for that with:Code: Select all
sudo cryptsetup luksFormat /dev/sda3
Code: Select all
sudo cryptsetup luksDump /dev/sda3
- create a new partition on the OS disk
- restore LVM volume group and logical volume
- enter UI with
Code: Select all
sudo veeam
- once again, choose "Restore volumes"
This time, the UI will report "Found 1 crypto devices. Do you want to decrypt them all?". This is the new LUKS container. - choose "Yes" to open the container and enter the password chosen above.
- map the two mapper devices (here: "luks-bf..." <--> "mapper/dm_crypt-0").
This should automatically map the LVM volume group "ubuntu-vg" and the logical volume "ubuntu-lv". The good thing here is that, even though Veeam, during the backup creation, does not have knowledge of the LUKS container, it does seem to have knowledge of the mapper device "dm_crypt-0" and that information is contained in the backup.
It should now look like this:Code: Select all
CURRENT SYSTEM │ IN BACKUP │ Device Restore Size │ Device Size Usage │ sda 127.0G │ mapper/dm... 123.9G sda1 1.04G │ sda 127.0G sda2 2.00G │ sda1 1.04G /boot/efi... sda3 123.9G │ sda2 2.00G /boot (ext4) luks-bf... 123.9G │ ubuntu-vg 123.9G ubuntu-vg 123.9G │ ubuntu-lv 61.96G / (ext4) ubuntu-lv ubuntu--v... 61.96G │ free 61.96G │
- once again, proceed with the restore
- close UI, exit to shell, but do not shutdown or reboot
All data should now be restored, but we still cannot boot into Ubuntu because the UUID of the LUKS partition has changed and it is now different from what Ubuntu expects during boot.
- enter UI with
- change LUKS partition UUID
- get the original LUKS partition UUID
Since I just opened the LUKS container in the UI and restored the LVM, both should still be opened and available.
mount the LVM:get UUIDCode: Select all
sudo mount /dev/mapper/ubuntu--vg-ubuntu--lv /mnt
My UUID is: 569f8381-74cb-4930-ad22-a113863c2458Code: Select all
veeamuser@veeam-recovery-iso:~$ cat /mnt/etc/crypttab dm_crypt-0 UUID=569f8381-74cb-4930-ad22-a113863c2458 none luks veeamuser@veeam-recovery-iso:~$
unmount LVMclose LVMCode: Select all
sudo umount /dev/mapper/ubuntu--vg-ubuntu--lv
Code: Select all
sudo vgchange -a n
- set LUKS partition UUID
Optionally, you can now confirm that the UUID has been set
Code: Select all
sudo cryptsetup luksUUID --uuid 569f8381-74cb-4930-ad22-a113863c2458 /dev/sda3
close the LUKS container. The name might be different each time.Code: Select all
sudo cryptsetup luksUUID /dev/sda3
Code: Select all
sudo cryptsetup luksClose /dev/mapper/luks-bf3f0eac-adce-4357-adaf-45a403a4ffba\\x0a
- get the original LUKS partition UUID
- shutdown recovery media system
Code: Select all
sudo shutdown -h now
- remove recovery media
- boot from disk
-
- Influencer
- Posts: 10
- Liked: 3 times
- Joined: Jun 19, 2019 9:55 am
- Contact:
Re: Bare metal restore of Ubuntu host (LUKS)
This is what I did during testing, in order to keep it simple. Ultimately, this is not what I need, because data on other disks, that are not the operating system disk, is backed up by another job. I don't know if it would even be possible to do it in one job, as the other disks contain ZFS, which cannot be backed up with snapshots. The job would need to run using two different backup modes: Snapshots for the OS volumes and file level backup for ZFS pool. Even if it was possible, it is not what I want. The question becomes:HannesK wrote:I just treat machines with dm-crypt like any other machine: entire machine backup. set it and forget it.
What needs to be backed up, when the intention is to just back up the OS disk?
My testing shows that, in the backup job settings, instead of "Entire computer", I can select "Volume level backup". In "Objects \ Objects to back up \ Add \ Device \ Path to block device", I need to add (see screenshot below)
- the OS disk device
In my case, it is "/dev/sda" - and the mapper device for the LUKS container
This should always be "/dev/mapper/dm_crypt-0", if the container was created by the Ubuntu installer.
![Image](https://i.imgur.com/LbY5j3w.png)
-
- Product Manager
- Posts: 14585
- Liked: 2959 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: How-to needed: Linux backup restore for LUKS partition
first of all: thanks for describing the solution. That saved me some time to go through it
![Smile :-)](https://forums.veeam.com/images/smilies/icon_smile.gif)
correct, that needs two jobs.I don't know if it would even be possible to do it in one job, as the other disks contain ZFS
as far as I see, you already gave the answerWhat needs to be backed up, when the intention is to just back up the OS disk?
![Smile :-)](https://forums.veeam.com/images/smilies/icon_smile.gif)
Agree, there are things we could simplify on Veeam side. I added +1 for the "entire computer" restore feature request. On the other hand, there are other features that are requested much more than improvements around dmcrypt / luks. How many machines with dm-crypt / luks to you manage?
Best regards,
Hannes
-
- Influencer
- Posts: 10
- Liked: 3 times
- Joined: Jun 19, 2019 9:55 am
- Contact:
Re: How-to needed: Linux backup restore for LUKS partition
I agree that it should not be a high priority because we now have it written down what needs to be done during restore. This is completely fine with me. I have just one server with Ubuntu and LUKS. Other than that, we don't really use bare metal restore. Everything is inside of VMs and is restored as VM. In the unlikely event that an entire virtualization host goes down, I would restore the VMs to a different host and then sometime later, re-install the physical server and migrate the VMs back eventually.birdwaffle wrote: ↑Jan 06, 2023 11:27 am Agree, there are things we could simplify on Veeam side. I added +1 for the "entire computer" restore feature request. On the other hand, there are other features that are requested much more than improvements around dmcrypt / luks. How many machines with dm-crypt / luks to you manage?
I will include a few more things I have come across during my testing as I moved the testing to the physical server. (in the hopes that, if somebody later googles this, they will find the answer here).
The re-created LUKS container has a different sector size than the original conatiner
Above, I wrote that the LUKS container that I create with the Veeam recovery media, has the same properties as the original container that is created by the Ubuntu installer. This was not entirely true. I overlooked the sector size of the container. My understanding of this is that LUKS writes changes to the disk in chunks of the size that is the container's 'sector size'. This is similar to the cluster size in Microsoft file systems or block size in Linux file systems. Setting a sector size for the container that does not match the physical disk layout, can lead to decreased performance.
In my case I had before the restore:
Code: Select all
Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 4096 [bytes]
Code: Select all
Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 512 [bytes]
Code: Select all
sudo cryptsetup luksDump /dev/sda3
There are several ways to fix this:
- create the LUKS container with a newer version of cryptsetup than the version that comes with the recovery media
For instance, you could boot into Ubuntu desktop live and create the container in Ubuntu live. - for a solution, that is entirely contained to using only the recovery media:
You can force the correct sector size during the container creation. But if the disk internally works with 4K sectors but reports a sector size of 512 bytes (512e disks), cryptsetup will return an error.To fix this, the end of the partition needs to be aligned, not just the beginning of the partition.Code: Select all
sudo cryptsetup luksFormat --sector-size=4096 /dev/sda3 Device size is not aligned to requested sector size.
- if you already created the partition that will be the LUKS container, delete it. Remember, the third partition is the LUKS partition (see above). Replace sda with the device name for your operating system disk.
Code: Select all
sudo sgdisk --delete=3 /dev/sda
- Now, create a new partition that has the start and end aligned.
Code: Select all
DISK='/dev/sda' start_position=$(sudo sgdisk --first-aligned-in-largest /dev/sda $DISK) end_position=$(sudo sgdisk --end-of-largest /dev/sda $DISK) aligned_end_position=$((end_position - ($end_position + 1) % 2048)) sudo sgdisk --set-alignment=2048 --new=3:$start_position:$aligned_end_position /dev/sda
- optionally, you can now confirm that the partition was created
Code: Select all
sudo gdisk -l $DISK
- now cryptsertup will allow the creation of a 4096 byte sector size container
confirm that the sector size is 4096
Code: Select all
sudo cryptsetup luksFormat --sector-size=4096 /dev/sda3
Code: Select all
veeamuser@veeam-recovery-iso:~$ sudo cryptsetup luksDump /dev/sda3 | grep sector sector: 4096 [bytes] veeamuser@veeam-recovery-iso:~$
- now proceed with the restore as usual (see above)
- if you already created the partition that will be the LUKS container, delete it. Remember, the third partition is the LUKS partition (see above). Replace sda with the device name for your operating system disk.
Who is online
Users browsing this forum: No registered users and 2 guests