-
- Influencer
- Posts: 12
- Liked: 10 times
- Joined: Jul 08, 2015 10:19 am
- Contact:
LUKS support
I've been looking around for a backup-solution of our field-workers Linux Laptops.
I am glad that Veeam now offers this BETA.
Just downloaded and tried out my first time this morning, so this is my first emotional oppinion
Sorry, it's long, I know...
First the short:
Installation on a Mint Linux 18 64bit (which is Ubuntu 16.04 based): no problems.
Then the backup:
I wanted to create "Entire system", but it failed on choosing "local" as a destination:
It's okay to say "/mnt/backup" is part of the entire system, so it cannot be chosen as a destination.
Veeam told me to exclude that directory... but there is no option to exclude it anywhere.
Otherwise choosing /dev/sdd1 should be an option, but even this failed because veeam needed a not a device I think.
The solution should be: Let me choose a directory, but please exclude that automatically from backups when "entire system - local".
So I just created a volume based backup job and chose any volume and partitions except the partition containing the backups.
I have a dedicated disk for local backups in our laptops, so this no problem for me, but I think the next versions should be able to backup even within the same partition and have to bring in "Exclusion" options for the Target-Directories.
The first run took 86 minutes (1TB read from SSD and 385 GB wrote to SATA-disk).
Most annoying was: During that run the system really often becam unresponsive for seconds.
The load average grew up to 7,26 on my Core i7, so the CPU's really were heavy on duty.
Veeam told me "source" had been the bottleneck. Nice to know my SSD is a bottleneck
But it felt my CPU was the bottleneck - it's the first time I really thought my Core i7 4770 is a bottleneck
Tried another run 15 minutes later. That run was "quick" in 6 minutes, read 340MB and wrote 138MB.
Wondering what 138 MB of data changed in that short time.
Reboted the machine and tried another run 2 hours later.. That run needed 38 minutes, read again 1TB and wrote 202MB.
In my oppinion an incremental backup run should not run about 40 minutes (just half the time of a full backup) even after a reboot...
Apropos incrmental backup:
Feature request:
As known from our Veeam enterprise: Will there be an reverse incrmental option someday?
I'm in love with that concept of "reverse incremental" the newest Backup-File alwas is complete and I may just discard the old diffs as my storage goes out of space without needing to do a new Full-Backup. That should become a standard in my humble oppinion
Restore:
Just tried to restore some data (online without the iso) and mounted the Restore point:
The mounted mount points worked so far... but... not every backed up device could be mounted.
May be the thing is:
I have got a luks-encrypted device and backed it up (/dev/sdc1).
So I think there is mission luks-support (at least for restore? Hope the backup worked, he used a lot of time to fully backup the device first time).
Veeam should try to detect luks devices, ask for a password and try to mount it for restore...?
Or is there any manual way to mount that restore-point using luks?
I think it will be the same on Restore with CD:
Does the CD contain everything to support luks encrypted volumes within the backup?
Next thing is:
The backups target-directory itself is stored on a luks-encrypted volume.
Within my running system it's no problem to mount that backup-target-directory and make it accessible to Veeam, but is the iso capable of mounting that encrypted target-filesystem?
To avoid using encrypted filesystems as a target-directory there should be a password option for backup jobs.
Especially for field-workers and their laptops often it is not possible to backup to clouds, so the backup should get local but secure for the case of loosing the hardware.
TLDR:
My first impression is:
Great beta as expected for Veeam product, but like a beta should be: Working but in need of round some corners!
Thank you for that good job until here, guys!
I am glad that Veeam now offers this BETA.
Just downloaded and tried out my first time this morning, so this is my first emotional oppinion
Sorry, it's long, I know...
First the short:
Installation on a Mint Linux 18 64bit (which is Ubuntu 16.04 based): no problems.
Then the backup:
I wanted to create "Entire system", but it failed on choosing "local" as a destination:
It's okay to say "/mnt/backup" is part of the entire system, so it cannot be chosen as a destination.
Veeam told me to exclude that directory... but there is no option to exclude it anywhere.
Otherwise choosing /dev/sdd1 should be an option, but even this failed because veeam needed a not a device I think.
The solution should be: Let me choose a directory, but please exclude that automatically from backups when "entire system - local".
So I just created a volume based backup job and chose any volume and partitions except the partition containing the backups.
I have a dedicated disk for local backups in our laptops, so this no problem for me, but I think the next versions should be able to backup even within the same partition and have to bring in "Exclusion" options for the Target-Directories.
The first run took 86 minutes (1TB read from SSD and 385 GB wrote to SATA-disk).
Most annoying was: During that run the system really often becam unresponsive for seconds.
The load average grew up to 7,26 on my Core i7, so the CPU's really were heavy on duty.
Veeam told me "source" had been the bottleneck. Nice to know my SSD is a bottleneck
But it felt my CPU was the bottleneck - it's the first time I really thought my Core i7 4770 is a bottleneck
Tried another run 15 minutes later. That run was "quick" in 6 minutes, read 340MB and wrote 138MB.
Wondering what 138 MB of data changed in that short time.
Reboted the machine and tried another run 2 hours later.. That run needed 38 minutes, read again 1TB and wrote 202MB.
In my oppinion an incremental backup run should not run about 40 minutes (just half the time of a full backup) even after a reboot...
Apropos incrmental backup:
Feature request:
As known from our Veeam enterprise: Will there be an reverse incrmental option someday?
I'm in love with that concept of "reverse incremental" the newest Backup-File alwas is complete and I may just discard the old diffs as my storage goes out of space without needing to do a new Full-Backup. That should become a standard in my humble oppinion
Restore:
Just tried to restore some data (online without the iso) and mounted the Restore point:
The mounted mount points worked so far... but... not every backed up device could be mounted.
May be the thing is:
I have got a luks-encrypted device and backed it up (/dev/sdc1).
So I think there is mission luks-support (at least for restore? Hope the backup worked, he used a lot of time to fully backup the device first time).
Veeam should try to detect luks devices, ask for a password and try to mount it for restore...?
Or is there any manual way to mount that restore-point using luks?
I think it will be the same on Restore with CD:
Does the CD contain everything to support luks encrypted volumes within the backup?
Next thing is:
The backups target-directory itself is stored on a luks-encrypted volume.
Within my running system it's no problem to mount that backup-target-directory and make it accessible to Veeam, but is the iso capable of mounting that encrypted target-filesystem?
To avoid using encrypted filesystems as a target-directory there should be a password option for backup jobs.
Especially for field-workers and their laptops often it is not possible to backup to clouds, so the backup should get local but secure for the case of loosing the hardware.
TLDR:
My first impression is:
Great beta as expected for Veeam product, but like a beta should be: Working but in need of round some corners!
Thank you for that good job until here, guys!
-
- Product Manager
- Posts: 5797
- Liked: 1215 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: LUKS support
About exclusions, this is currently known and should be improved within an upcoming release .
About what changed, well that can be a lot of things, logs, temp files,... Machines do change quick (quicker then you think) and 138MB isn't that much imo. I guess the incremental was around 60-70MB in file size?
In regards to reverse incremental, this probably won't come soon as it requires much more I/O on the backup target and even within VBR we advise forward incremental (with synthetic full).
LUKS support is something we might have to look into as encryption is getting more and more used, so thanks for the FR.
About what changed, well that can be a lot of things, logs, temp files,... Machines do change quick (quicker then you think) and 138MB isn't that much imo. I guess the incremental was around 60-70MB in file size?
In regards to reverse incremental, this probably won't come soon as it requires much more I/O on the backup target and even within VBR we advise forward incremental (with synthetic full).
LUKS support is something we might have to look into as encryption is getting more and more used, so thanks for the FR.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: LUKS support
Hi,
Regarding LUKS support - I agree with Niels, we will look into it later.
Thanks for your feedback!
You should exclude the device that contains the target directory. If you choose the "Entire machine" VAL attempts to backup all local drives, so you've decided to use "Volume level" mode, which was a valid approach.It's okay to say "/mnt/backup" is part of the entire system, so it cannot be chosen as a destination.
Veeam told me to exclude that directory... but there is no option to exclude it anywhere.
I'm not sure that I follow you - did you try to choose /dev/sdd1 as a destination?Otherwise choosing /dev/sdd1 should be an option, but even this failed because veeam needed a not a device I think.
Imagine if you had CIFS share mounted to a local directory (say /backup) , so if you chose "Entire" - "Local" - "your CIFS" everything would be OK. Now imagine that, for some reason, the CIFS share have been accidentally unmounted. With the approach you've offered VAL would automatically exclude the device where the /backup directory resides at (root device for example) thus leaving you without a backup.The solution should be: Let me choose a directory, but please exclude that automatically from backups when "entire system - local"
...which gives us an average speed of ~100 MB/s - that's a normal speed of a SATA drive, looks like the bottleneck stats are wrong, thanks for heads up!The first run took 86 minutes (1TB read from SSD and 385 GB wrote to SATA-disk).
That hapens due to reboot - the driver is loaded out off the kernel thus CBT is get reset. Will look into that in future releases.Reboted the machine and tried another run 2 hours later.. That run needed 38 minutes, read again 1TB and wrote 202MB. In my oppinion an incremental backup run should not run about 40 minutes (just half the time of a full backup) even after a reboot...
Depending on the amount of requests, however not in the first version for sure.Will there be an reverse incrmental option someday?
Regarding LUKS support - I agree with Niels, we will look into it later.
Thanks for your feedback!
-
- Influencer
- Posts: 12
- Liked: 10 times
- Joined: Jul 08, 2015 10:19 am
- Contact:
Re: LUKS support
Did not think about the approach of missing network mounts... now it's making sense
The CBT is reset.. no as you say it... sounds somewhat logical...
In summary: Yes, the LUKS approach in my point of view would be one of the prior things, the rest is nice to have and performance improvements for future versions.
The CBT is reset.. no as you say it... sounds somewhat logical...
In summary: Yes, the LUKS approach in my point of view would be one of the prior things, the rest is nice to have and performance improvements for future versions.
That's what the BETA is forThanks for your feedback!
-
- Novice
- Posts: 6
- Liked: never
- Joined: Feb 24, 2017 6:27 am
- Full Name: Frédéric PRAT
- Contact:
Re: LUKS support
Hello,
We have the Veeam Backup & Replication solution and we need to perform backup/restore plan of Oracle Linux 6.8 system with LVM over LUKS volume (Entire system backup).
In order to perform this task we use the last Veeam Linux agent.
It seems that the LUKS isn't currently available in the last Veeam Linux agent.
Is there any plan to integrate it in a future release of the Veeam Linux agent.
Thank you.
Best regards
Frédéric PRAT
We have the Veeam Backup & Replication solution and we need to perform backup/restore plan of Oracle Linux 6.8 system with LVM over LUKS volume (Entire system backup).
In order to perform this task we use the last Veeam Linux agent.
It seems that the LUKS isn't currently available in the last Veeam Linux agent.
Is there any plan to integrate it in a future release of the Veeam Linux agent.
Thank you.
Best regards
Frédéric PRAT
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: LUKS support
Hi,
Backup of LUKS-encrypted volumes should work just fine, the only limitation is that the volumes form such backup will be restored unencrypted. Did you get any error messages during the backup session?
Thank you
Backup of LUKS-encrypted volumes should work just fine, the only limitation is that the volumes form such backup will be restored unencrypted. Did you get any error messages during the backup session?
Thank you
-
- Novice
- Posts: 6
- Liked: never
- Joined: Feb 24, 2017 6:27 am
- Full Name: Frédéric PRAT
- Contact:
Re: LUKS support
Hello,
Thank you for your fast feedback.
Currently, We want to backup/restore an entire linux server with the following disks architecture:
The backup process haven't returned any errror and I haven't see any error in the log file
During the restoration process (though the veeam bootable linux image), the disk sdc1 is restored empty and we have lose the encrypted disk with the Volume group and the logical volume available under it.
After restoration, the restored disks architecture looks like:
With the veeam Linux Agent and the Veeam Backup and Replication application, how we must proceed for backuping and restoring LVM over LUKS object?
Thank you for your response.
Best regards
Frédéric PRAT
Thank you for your fast feedback.
Currently, We want to backup/restore an entire linux server with the following disks architecture:
Code: Select all
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 99,5G 0 part
├─root_vg-lv_root (dm-0) 251:0 0 29,3G 0 lvm /
├─root_vg-lv_swap (dm-1) 251:1 0 15,6G 0 lvm [SWAP]
└─root_vg-lv_home (dm-7) 251:7 0 29,3G 0 lvm /home
sdb 8:16 0 50G 0 disk
└─sdb1 8:17 0 50G 0 part
├─oracle_vg-lv_ora1120 (dm-3) 251:3 0 29,3G 0 lvm /oracle/oradb/11.2.0
└─oracle_vg-lv_diag (dm-4) 251:4 0 2G 0 lvm /oracle/diag
sdc 8:32 0 600G 0 disk
└─sdc1 8:33 0 600G 0 part
└─luks-56a83e13-e115-4eb0-bbd7-21caba21736a (dm-2) 251:2 0 600G 0 crypt
├─data_vg-lv_appli_POGSINDV (dm-5) 251:5 0 293G 0 lvm /appli/oracle/POGSINDV
└─data_vg-lv_sauvebase (dm-6) 251:6 0 195,3G 0 lvm /sauvebase
During the restoration process (though the veeam bootable linux image), the disk sdc1 is restored empty and we have lose the encrypted disk with the Volume group and the logical volume available under it.
After restoration, the restored disks architecture looks like:
Code: Select all
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 99,5G 0 part
├─root_vg-lv_root (dm-0) 251:0 0 29,3G 0 lvm /
├─root_vg-lv_swap (dm-1) 251:1 0 15,6G 0 lvm [SWAP]
└─root_vg-lv_home (dm-7) 251:7 0 29,3G 0 lvm /home
sdb 8:16 0 50G 0 disk
└─sdb1 8:17 0 50G 0 part
├─oracle_vg-lv_ora1120 (dm-3) 251:3 0 29,3G 0 lvm /oracle/oradb/11.2.0
└─oracle_vg-lv_diag (dm-4) 251:4 0 2G 0 lvm /oracle/diag
sdc 8:32 0 600G 0 disk
└─sdc1 8:33 0 600G 0 part
Thank you for your response.
Best regards
Frédéric PRAT
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: LUKS support
Hi Frederic,
Could you please try to perform File-level restore from the backup in order to check if the files that reside on the encrypted disks are available?
Thanks
Could you please try to perform File-level restore from the backup in order to check if the files that reside on the encrypted disks are available?
Thanks
-
- Novice
- Posts: 6
- Liked: never
- Joined: Feb 24, 2017 6:27 am
- Full Name: Frédéric PRAT
- Contact:
Re: LUKS support
Hello,
So, I have tested the File-Level restore but before using the File-level restore some préerequisite must applied.
As, during the Entire System restore process, it doesn't restore any object under the encrypted partition, the volume group,the logical volumes and the associated data aren't be restored.
Before, running the File-level restore process, we must create through the Linux Rescue disk:
- The missing volume group
- The missing logical volumes and format they
After this, we can perform the File-Level restore process in the last created logical volume.
So, in summarry (correct me if It is not right), the restore process of an Entire Linux system with LVM over luks volumes will looks like as below:
1. Boot on Veeam recovery media disk
2. Perform the Restore volumes process
3. Reboot
4. Boot on the Linux rescue disk
5. Create missing volume group, logical volumes and format they
6. Reboot
7. Boot on Veeam recovery media disk
8. Perform the Restore files process in order to restore the data in the logical volumes previously created
Disadvantages
Lost of the LUKS encrypted volumes (Data backuped from an encrypted volume are restored in an unencrypted volume)
A simple restore process is transformed into a difficult restore process.
Did you have any others process to restore the backup of a system with lvm over luks volumes.
Thank you for your feedback
Best regards
Frédéric PRAT
So, I have tested the File-Level restore but before using the File-level restore some préerequisite must applied.
As, during the Entire System restore process, it doesn't restore any object under the encrypted partition, the volume group,the logical volumes and the associated data aren't be restored.
Before, running the File-level restore process, we must create through the Linux Rescue disk:
- The missing volume group
- The missing logical volumes and format they
After this, we can perform the File-Level restore process in the last created logical volume.
So, in summarry (correct me if It is not right), the restore process of an Entire Linux system with LVM over luks volumes will looks like as below:
1. Boot on Veeam recovery media disk
2. Perform the Restore volumes process
3. Reboot
4. Boot on the Linux rescue disk
5. Create missing volume group, logical volumes and format they
6. Reboot
7. Boot on Veeam recovery media disk
8. Perform the Restore files process in order to restore the data in the logical volumes previously created
Disadvantages
Lost of the LUKS encrypted volumes (Data backuped from an encrypted volume are restored in an unencrypted volume)
A simple restore process is transformed into a difficult restore process.
Did you have any others process to restore the backup of a system with lvm over luks volumes.
Thank you for your feedback
Best regards
Frédéric PRAT
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: LUKS support
I think that some sort of misunderstanding occured and I'd like to clerify the situation.
Can you see the files that are located on the encrypted device if you press 'R' on the main screen and choose the backup? Also, do you see the logical volumes in the right panel (like on this picture) that are located on an encrypted device?
Thank you.
Can you see the files that are located on the encrypted device if you press 'R' on the main screen and choose the backup? Also, do you see the logical volumes in the right panel (like on this picture) that are located on an encrypted device?
Thank you.
-
- Novice
- Posts: 6
- Liked: never
- Joined: Feb 24, 2017 6:27 am
- Full Name: Frédéric PRAT
- Contact:
Re: LUKS support
Hello,
Here is a screen capture of the Veeam Recovery Media view:
It seems that the volume group are correctly backuped but the encrypted volume doesn't appear (sdc1). So it seems that during the restore process, the Veeam recovery media doesn't know where restore the volume group (cf screen capture).
The luks volume is under sdc1:
It seems that the trouble is more during the backup process.
Here is the Agent Source Log and Job.log of the backup.
Best regards
Frédéric PRAT
Here is a screen capture of the Veeam Recovery Media view:
It seems that the volume group are correctly backuped but the encrypted volume doesn't appear (sdc1). So it seems that during the restore process, the Veeam recovery media doesn't know where restore the volume group (cf screen capture).
The luks volume is under sdc1:
Code: Select all
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 99,5G 0 part
├─root_vg-lv_root (dm-0) 251:0 0 29,3G 0 lvm /
├─root_vg-lv_swap (dm-1) 251:1 0 15,6G 0 lvm [SWAP]
└─root_vg-lv_home (dm-7) 251:7 0 29,3G 0 lvm /home
sdb 8:16 0 50G 0 disk
└─sdb1 8:17 0 50G 0 part
├─oracle_vg-lv_ora1120 (dm-3) 251:3 0 29,3G 0 lvm /oracle/oradb/11.2.0
└─oracle_vg-lv_diag (dm-4) 251:4 0 2G 0 lvm /oracle/diag
sdc 8:32 0 600G 0 disk
└─sdc1 8:33 0 600G 0 part
└─luks-56a83e13-e115-4eb0-bbd7-21caba21736a (dm-2) 251:2 0 600G 0 crypt
├─data_vg-lv_appli_POGSINDV (dm-5) 251:5 0 293G 0 lvm /appli/oracle/POGSINDV
└─data_vg-lv_sauvebase (dm-6) 251:6 0 195,3G 0 lvm /sauvebase
Here is the Agent Source Log and Job.log of the backup.
Best regards
Frédéric PRAT
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: LUKS support
Got it, thanks! And the last question - what happens if you specifically choose sdc1 and/or the underlying volumes using the "Volume level backup" mode?
Thanks
Thanks
-
- Novice
- Posts: 6
- Liked: never
- Joined: Feb 24, 2017 6:27 am
- Full Name: Frédéric PRAT
- Contact:
Re: LUKS support
I'm not able to select the sdc1 volume in the "Volume level backup" mode (it doesn't appear in the interface).
In order to test, I have selected the sdc volume and the dependant volume group and logical volume and perform the backup.
The backup job is configured as baloow:
In the job.log, I see the following message:
Here is the Agent.source.log and Job.log of the last backup.
When I process to a "Restore volumes" through the Veeam Recovery Media, it doesn't restore the volume group when I try to restore the sdc volume:
And when I try to restore only teh volume group, the application doesn't propose any option:
As the backup process doesn't want to backup the sdc1 volumes (the encrypted volume), the restore process doesn't know where restore the volume group.
Best regards
Frédéric PRAT
In order to test, I have selected the sdc volume and the dependant volume group and logical volume and perform the backup.
The backup job is configured as baloow:
Code: Select all
# veeamconfig job info --name TestVolumeBackup
Backup job
ID: {b68f3b6e-7703-4ae5-9a10-accc4a33e725}
Name: TestVolumeBackup
Repository ID: {88788f9e-d8f5-4eb4-bc4f-9b3f5403bcec}
Repository name: [SrvSauvTlsOoc01]Default Backup Repository
Creation time: 2017-03-07 14:27:24
Options:
Compression: Lz4
Max Points: 14
Objects for backup:
Include Disk: data_vg
Include Disk: sdc
Code: Select all
[07.03.2017 15:24:51] <139837375076096> lpbcore| Partition [/dev/sdc1] has usage type [crypto] and will be skipped.
When I process to a "Restore volumes" through the Veeam Recovery Media, it doesn't restore the volume group when I try to restore the sdc volume:
And when I try to restore only teh volume group, the application doesn't propose any option:
As the backup process doesn't want to backup the sdc1 volumes (the encrypted volume), the restore process doesn't know where restore the volume group.
Best regards
Frédéric PRAT
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: LUKS support
Ok, I see. For now the only way to restore the LVs is to manually create sdc1 partition via Recovery Media, create VG, and manually restore LVs into that VG.
Regarding LUKS preservation - we will look what can be done with that problem so you will be able to keep encryption in place after restore.
Thanks
Regarding LUKS preservation - we will look what can be done with that problem so you will be able to keep encryption in place after restore.
Thanks
-
- Novice
- Posts: 6
- Liked: never
- Joined: Feb 24, 2017 6:27 am
- Full Name: Frédéric PRAT
- Contact:
Re: LUKS support
Ok, I will perform the restoration process like this.
Thank you for your availability et professionalism.
Best regards
Frédéric PRAT
Thank you for your availability et professionalism.
Best regards
Frédéric PRAT
Who is online
Users browsing this forum: No registered users and 13 guests