Bare metal restore with software RAID is very difficult

[aj_potc]

I'm testing out bare metal restoration on a CentOS system that uses software RAID 10 (md-raid). Unfortunately, this process is extremely difficult, and I'm looking for suggestions to make it easier.

The problems

Veeam Agent for Linux doesn't seem to be able to capture and re-create complex partitioning or RAID configurations. Instead, you need to prepare the target system manually, which can be quite difficult if it's in a datacenter and your remote access is limited to KVM or some other out-of-band management system.

In addition, the post-restore "fixup" process, in which you check that various system configs are pointing to the correct RAID devices, is difficult to do from a recovery environment.

Finally, the bare metal recovery procedure is so time consuming and inconvenient that it's almost easier to rebuild the system from scratch and restore backed up files manually.

General restore procedure

To illustrate the difficulty, here is a simplified list of steps I'm following:

1. Using a CentOS or other LiveCD, boot into a recovery environment.
2. Set up the partitions and md RAID devices identically to the source system. I find that the utility sgdisk is ideal for backing up and restoring these configs, assuming the recovery system has network access (but virtually all of them don't).
3. Boot the Veeam recovery ISO. Ensure that md RAID is running. Recover the contents of each md device.
4. Finally, before booting, check /etc/fstab, /etc/mdadm.conf, and the Grub configuration to ensure all of these are pointing to the correct devices. Regenerating initramfs is usually required as well. Depending on which utilities you need, you may need to boot into a different LiveCD to accomplish these steps.

A much more detailed and excellent list of steps is available in this post:
veeam-agents-for-linux-mac-aix-solaris- ... 67734.html

My questions

1. I suppose that the problem of recreating partitions and RAID devices is shared by other backup software. I know how to back up disk configurations and the superblock (which contains info about the RAID config), but how do I restore this to my target system from inside a recovery environment without network access? Is there another tool I can use to handle this?
2. Should I be looking at creating custom recovery media to do what I need?
3. The sgdisk utility isn't included on the Veeam recovery media. Is there a way for me to add it?
4. Does anyone know of any better restoration procedures? Perhaps Veeam can do more than I realize?

Thanks a lot for any thoughts on this frustrating but extremely important topic!

[HannesK]

Hello,
off-topic: I recommend to use hardware raid. That eliminates all the hassles with software-raid.

1. No idea how to access the recovery without network access. The next version of the recovery media (a few more days to wait) allows SSH access per default. It's a Debian live image and you can add tools.
2. I'm not sure about your situation with the network. But you can install the Veeam agent also on a live media, yes
3. in the version 5 it is included. Instead of trying it manually, I would wait for general availability of version 5.
4. Use the next version that becomes available in a few days

Best regards,
Hannes

[aj_potc]

Hi HannesK,

Thanks for your reply. It sounds like I might have started my disaster recovery testing a week too early!

I have seen that there are some nice things in store for B&R v11. But is there any documentation or change log available specifically about version 5 of the Linux agent?

M procedure should be a little easier if the recovery media is more fully featured than the current BusyBox version.

In the past, @PTide has mentioned that there were plans to make the Linux recovery more automated. This would be a big help, at least from the perspective of the post-restore "cleanup" process, in which various system files are checked to ensure that they are pointing to the correct storage UUIDs. As things stand now, you can't be sure you'll have a bootable system after a restore in which software RAID is involved. Do you know if any effort has been made on this?

Thanks again for the reply. I'm always happy to see progress on the Linux front.

[HannesK]

welcome! Yes, a What's New document will be available at GA.

Bare metal restore automation is not part of V5.

[aj_potc]

I've just attempted to test the new ISO, but unfortunately it will not boot on my system.

After some ACPI-related errors, it drops to an initramfs prompt.

I tried booting with both kernel versions offered in the Grub menu on the ISO (4 and 5). Neither one works.

Here's a screenshot:



I was able to boot the ISO successfully on a Hyper-V VM, but not on my bare metal test system.

Does anyone have an idea what might be going on here?
Thanks!

[PTide]

Kindly tell us what's the model of the motherboard? Is it a laptop by any chance?

Is it a patched ISO or just a generic one? Did the previous ISO boot ok on the same machine? Which linux distro starts successfully on that hardware?

Thanks!

[aj_potc]

Thanks for your reply.

The motherboard is an Asus P8H67-M PRO. Latest BIOS revision. It's a desktop system, not a laptop.

The ISO is not patched -- it's the one downloaded from your site. The version 4 ISO booted without problems, and so does everything else I've tried on the system:

CentOS 8 (kernel 4.18)
CentOS 7 (kernel 3.10)
Debian (kernel 4.9.190)

I also booted a later Debian with a version 5 kernel (don't have the exact version though). In fact, I couldn't find anything else that doesn't boot.

I'm really at a loss here!

[aj_potc]

Update:

I've tried booting the same system with a different LiveCD, and then installing Veeam to run the recovery from that (as I outlined above). I added the following to /etc/veeam/veeam.ini:

Code: Select all

[recoveryui]
enableOnLiveSystem = 1

Then I restarted the veeamsnap service and started Veeam with veeamconfig recoveryui.

Connecting to a VBR and recovering files worked. However, after I disconnected the mount from inside the utility and tried to recover from a volume, I get the following error:



Unfortunately it doesn't seem possible for me to do any type of bare metal restore. Am I just having really bad luck, or is there perhaps a bug here?

Thanks for any tips.

[PTide]

Ok, now that doesn't look right at all. Right now I don't have any ideas what that could be. Let us try to reproduce it in our lab, I'll get back to you in case more info is required.

If you could grab system logs and agent logs (the usual veeamconfig grabLogs) that would be great.

Thanks!

[PTide]

@aj_potc,

Besides logs getting the logs, we also need to know whether you used veeam-recovery-media-5.0.0.iso (163 MB), or veeam-recovery-amd64-5.0.0.iso (490 MB).

Thanks!

[aj_potc]

Hi @PTide,

The recovery media that failed to boot was the 64-bit version:
veeam-recovery-amd64-5.0.0.iso (490 MB)

I didn't yet try the other one, but would be glad to do so if it would help.
My case related to this problem is: Case #04668680.

On the other error:
I've generated a log export. Should I upload it to the case number above?

Thanks.

[aj_potc]

Small correction:

The actual name of the recovery media file is:
veeam-recovery-media-5.0.0.4318_x86_64.iso

[PTide]

I've generated a log export. Should I upload it to the case number above?

Yes, please do.

veeam-recovery-media-5.0.0.4318_x86_64.iso

Ok, that's the new recovery. I wonder if the issues persists if you use the legacy version?

Here is the link:
https://repository.veeam.com/backup/lin ... -5.0.0.iso

Also, what is the liveCD (distro and kernel) that you used when you got "Object of type [7] has no parent" error?

Thanks!

[aj_potc]

Hi @PTide,

Unfortunately it seems that the support case I created has disappeared. I can no longer access it, and when I create new ones they are never replied to and automatically closed. So I don't think I'll continue with that unless you can suggest something better.

I haven't yet tried to use an older recovery version, but I can try it when I have some spare time. I've already dumped dozens of hours into evaluating B&R for Linux machines, and so far it's been nothing but frustration. I keep hitting problems, but can't get any support for them. Should I purchase licensing in the hope that I'll get the support I need? That doesn't sound logical to me.

The LiveCD version I used was Debian 9 (4.9.190).

Thanks for any help.

[solarjdp69]

See my solution that I worked through and have used numerous times to restore RAID1.

topic67734.html

[PTide]

Unfortunately it seems that the support case I created has disappeared. I can no longer access it, and when I create new ones they are never replied to and automatically closed. So I don't think I'll continue with that unless you can suggest something better.

Well, let's get a fast-track then ; )
Please upload logs to any filesharing resource of your choice and PM me the link.

Should I purchase licensing in the hope that I'll get the support I need? That doesn't sound logical to me.

Correct. Support for Customers with the Community Edition/Free License/NFR licensed products is provided on a best-effort basis.

Thanks!

[aj_potc]

See my solution that I worked through and have used numerous times to restore RAID1.

topic67734.html

Yes, indeed. I linked to your solution in my initial post above. Well done! It was a great help, but it really points out the complexity of using Veeam to do such a restoration.

My servers are remote, and I really need to be able to connect to them via SSH to handle the various pre- and post-restore tasks. Veeam's latest recovery media should allow this, but I haven't been able to boot with it on my testbed.

[PTide]

@aj_potc,

When should I expect to receive the logs from you? Unfortunately we cannot do much without logs, so please send it over.

Thanks!

[aj_potc]

Booting from the following media works on my system:
https://repository.veeam.com/backup/lin ... -5.0.0.iso

However, there are issues:

First, when doing a file restore, which mounts the backups to /mnt/backup, the unmount process is unreliable. See screenshots below.

Here is the dialog where you can choose to unmount the backup:


And here is what happens when I choose that option:


There's nothing mounted to /mnt/backup, so I can't observe any obvious issues like a stuck mount, or some process that's holding the files open, etc. It isn't obvious to me how to fix this other than by rebooting the system.
I've captured logs from this if there's any interest.

Second, when I do a volume restore, the transfer process runs very slowly. It runs at approximately one-quarter transfer speed when compared to booting the system via a different Debian-based LiveCD, installing the Veeam client, and running the restore from there. Maybe this has something to do with network drivers or something else that isn't optimized in the limited recovery environment -- not sure.

@PTide has suggested building a custom recovery ISO based on the following ISO:
https://repository.veeam.com/backup/lin ... -5.0.0.iso
I've not had time to try this yet, but may do so.

That's all for now.

[PTide]

Unfortunately it seems that the support case I created has disappeared. I can no longer access it, and when I create new ones they are never replied to and automatically closed.

It seems that the support team has located your case. Can you confirm that you still cannot access it?

Thanks!

[aj_potc]

@PTide,

Yes, it's back in my account, thank you.

All relevant logs and screenshots have been uploaded/referenced there.

Any progress on your end so far?

[PTide]

Any progress on your end so far?

Not yet, unfotunately. We are still trying to reproduce the issue.

Thanks

[aj_potc]

We are still trying to reproduce the issue.

I've updated the ticket with as many details as possible, so please check that if you haven't already. I'd be interested in knowing what parts don't match my experience.

A quick summary for those following along:

Both unpatched and patched versions of this ISO won't boot at all on my system:
https://repository.veeam.com/backup/lin ... -5.0.0.iso

The following ISO will boot:
https://repository.veeam.com/backup/lin ... -5.0.0.iso

However, it has other weird issues:

1. Sometimes it's not possible to gracefully end file recovery (i.e., unmount /mnt/backup using the Veeam UI)
2. Network transfer speed is extremely poor, which means a bare metal recovery would take too long for real world use

[PTide]

I wonder whether setting "acpi=off" kernel parameter will help. Have you tried?

[aj_potc]

I wonder whether setting "acpi=off" kernel parameter will help. Have you tried?

Yeah, this was one of the first things I tried based on my research of boot issues. No help, unfortunately.

[PTide]

How was the media created? I mean, do you use some sort of external HDD enclosure which emulates DVD drive, or you created a bootable USB?

Thanks!

[aj_potc]

I don't have physical access to the server, so the staff at the datacenter are writing the ISO to a bootable USB stick and plugging that into the server.

This is a very common procedure for them. If a customer requests any OS for which they don't provide a template, then they set up remote access via IPMI and let you boot the server via USB flash drive.

Is there any way I can make the startup process more verbose? I get the feeling that the ACPI errors I see on the console (see screenshot above) are not the true cause, as they are displayed when I boot the same server from other distributions and kernel versions.

[PTide]

his is a very common procedure for them. If a customer requests any OS for which they don't provide a template, then they set up remote access via IPMI and let you boot the server via USB flash drive.

There are multiple ways to do so, and I wonder which one did they use. Is it possible to ask them?

Is there any way I can make the startup process more verbose?

You can remove 'quite' parameter.

Thanks!

[aj_potc]

There are multiple ways to do so, and I wonder which one did they use. Is it possible to ask them?

I'm afraid I don't know what particular detail you're after. The datacenter downloads the ISO from the URL I give them. Then they write it to a USB flash memory drive to create bootable media. (I don't know which software they use for this, but I'm sure it's one of the open source options out there.) Then they physically attach that USB flash drive to the server. Next, they set up an IP-based KVM and connect that to the server. Finally, they send me access details to the KVM. After connecting to it, I can view and send input to the server console.

Considering this is a very common task for them, and that they've done it many times for me successfully, I don't believe the method of creating the boot drive is the issue. I think it's just something weird about this particular server and the combination of OS used on the Veeam recovery media.

I'm continuing to work with support on this and other issues related to bare metal restore, but it's very slow going.

Thanks.

[PTide]

I'm afraid I don't know what particular detail you're after.

The name of the tool, for example. I also wonder whether you could plug an ISO via IPMI from your side, and if that would work.

Our team is also still digging, please let us know of you see anything interesting in the extended boot log.

Thanks!