Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
Post Reply
GaryW
Influencer
Posts: 11
Liked: 5 times
Joined: Apr 12, 2016 11:07 pm
Full Name: Gary Williams
Location: Kent, United Kingdom
Contact:

Linux Agent BMR ISO issue

Post by GaryW »

Hi all,

I'm trying to recover a server using an ISO generated by the agent and it's failing during boot:

Image

I've not been able to find any articles on this so would appreciate any pointers - the backup and restore and being attempted on Dell servers that have RAID1 so I'm wondering if that's part of the issue.

thanks.
PetrM
Veeam Software
Posts: 3229
Liked: 519 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Linux Agent BMR ISO issue

Post by PetrM »

Hi Gary,

Please provide a support case ID for the issue as requested when you click New Topic. It's difficult to say what is the reason without having a look at debug logs.

Thanks!
GaryW
Influencer
Posts: 11
Liked: 5 times
Joined: Apr 12, 2016 11:07 pm
Full Name: Gary Williams
Location: Kent, United Kingdom
Contact:

Re: Linux Agent BMR ISO issue

Post by GaryW »

Case ID is 04923703

There is an issue with booting the veeam backup generated ISO file as per the screenshot.
GaryW
Influencer
Posts: 11
Liked: 5 times
Joined: Apr 12, 2016 11:07 pm
Full Name: Gary Williams
Location: Kent, United Kingdom
Contact:

Re: Linux Agent BMR ISO issue

Post by GaryW »

I'm pretty certain that your ISO is using a broken version of Ubuntu as documented here https://bugs.launchpad.net/ubuntu/+sour ... ug/1873614
GaryW
Influencer
Posts: 11
Liked: 5 times
Joined: Apr 12, 2016 11:07 pm
Full Name: Gary Williams
Location: Kent, United Kingdom
Contact:

Re: Linux Agent BMR ISO issue

Post by GaryW »

So a bit more testing - I downloaded version 4 of the linux agent boot ISO and it worked first go. I'm now certain that you have a bug in your version 5.x veeam agent boot ISO.
aj_potc
Expert
Posts: 138
Liked: 33 times
Joined: Mar 17, 2018 12:43 pm
Contact:

Re: Linux Agent BMR ISO issue

Post by aj_potc » 2 people like this post

I've had plenty of issues with the v5 ISO as well. Please see my post here, and later posts in that thread:

post402971.html#p402971

Edit to add:

My issue was never truly resolved in a satisfactory way. I found a crazy workaround that I posted in that thread, but it makes no sense why it works.
GaryW
Influencer
Posts: 11
Liked: 5 times
Joined: Apr 12, 2016 11:07 pm
Full Name: Gary Williams
Location: Kent, United Kingdom
Contact:

Re: Linux Agent BMR ISO issue

Post by GaryW » 1 person likes this post

I wish I'd found this last night! I ended up giving up on version 5 and downgrading to v4. I'm glad it's not just me who has hit this issue.

thank you!
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Linux Agent BMR ISO issue

Post by PTide »

@GaryW,

What kind of RAID controller does the server have? Is it built-in (aka "fakeRAID"), or it's a separate card?

Thanks!
GaryW
Influencer
Posts: 11
Liked: 5 times
Joined: Apr 12, 2016 11:07 pm
Full Name: Gary Williams
Location: Kent, United Kingdom
Contact:

Re: Linux Agent BMR ISO issue

Post by GaryW » 1 person likes this post

It's a Dell R920 with a PERC. It's a proper RAID card.

As above, the issue isn't hardware or RAID card. It's the recovery ISO in veeam version 5. I rolled the agent back to V4, backed up, booted from the recovery ISO and did a full restore. It worked first go.
The Veeam v5 recovery iso is broken.

@ptide - try downloading the v5 recovery iso and booting from it. You'll see the issue.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Linux Agent BMR ISO issue

Post by PTide »

Noted, thanks! Actually, we already have a working item for that issue, and it is scheduled for fixing.
GaryW
Influencer
Posts: 11
Liked: 5 times
Joined: Apr 12, 2016 11:07 pm
Full Name: Gary Williams
Location: Kent, United Kingdom
Contact:

Re: Linux Agent BMR ISO issue

Post by GaryW » 1 person likes this post

Okay - a few things concern me here:

1. How did this slip past QA?
2. If you have a schedule for fixing, why is it not listed on the know issues list? I checked that before doing some in depth testing and this has been known about since Februray when aj_potc posted about it.

While this issue persists BMR recovery of linux is not possible. This is something that really should not have passed Q&A. I'm shocked it did and it gives me concern for the rest of the veeam suite.
aj_potc
Expert
Posts: 138
Liked: 33 times
Joined: Mar 17, 2018 12:43 pm
Contact:

Re: Linux Agent BMR ISO issue

Post by aj_potc » 1 person likes this post

My impression is that Veeam's Linux product line isn't nearly as popular as the others. As a result, while Veeam is at the top of the game for Windows, Hyper-V, and VMWare platforms, their Linux offering isn't close to that level.

If you and I are the only ones who noticed in the last 6 months that there's a critical problem booting the recovery ISO on (some?) bare metal, and posted about it here, then I think that lends some credence to my theory. As shocking as it is, maybe Veeam really doesn't have that many customers backing up Linux bare metal and actually testing whether a restore works.

Having said that, I see that Veeam's product managers and support are serious about the product, and they seem to take suggestions enthusiastically, so I hope that things will get better. I don't mean this as criticism, but rather to point out that there is room for improvement.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Linux Agent BMR ISO issue

Post by PTide »

@GaryW,

1. Because this bug mostly shows up when the media (ISO) is not initializing very fast (for whatever reason) or when it is mounted remotely with a slow/unstable connection to the server.
That is, some specific circumstances were required to reproduce it, and so far you seem to be the second to stumble on that thing ever.

2. It should have been added, must have slipped through the cracks. Will check.

UPDATE: It was described as a KB article and in the context of third-party ISO tools. Will add it in the Known Issues.

@aj_potc
Interesting version! It could even be sound had we not had certain large customers who are very picky when it comes to backup products : )
Jokes aside, the bug is indeed rarely seen and because of that it wasn't detected by pretty much anyone, even by those who test BMR (QA and the customers).
It will be fixed in the upcoming minor update.

For now, please try increasing "rootdelay" parameter kernel before booting (set it to 20, for example).

Next, since we are here discussing relevant problems, I'd like to ask you if you could give us an example of some Linux backup product (including OSS) which you truly like.

Thanks!
aj_potc
Expert
Posts: 138
Liked: 33 times
Joined: Mar 17, 2018 12:43 pm
Contact:

Re: Linux Agent BMR ISO issue

Post by aj_potc » 1 person likes this post

Interesting version! It could even be sound had we not had certain large customers who are very picky when it comes to backup products : )
Jokes aside, the bug is indeed rarely seen and because of that it wasn't detected by pretty much anyone, even by those who test BMR (QA and the customers).
I'm glad to hear that the big customers are happy. I'm a really small (but paying) customer who needs bare metal recovery to work on a few dedicated servers in a remote datacenter, and unfortunately Veeam does not make this easy. I've written a lot about my experiences on this forum. I don't know why "enterprise" customers would have an easier time than me, but I suppose that's just how it is. :)

It's good that the ISO boot bug is rare. I have no idea why it would crop up in certain situations, but I'm genuinely happy that you've isolated the cause.
Next, since we are here discussing relevant problems, I'd like to ask you if you could give us an example of some Linux backup product (including OSS) which you truly like.
No solution is perfect, and I haven't tested all of them. For Linux I use a mix of open source products such as rsync/rsnapshot and UrBackup as well as commercial ones like R1Soft and Veeam. I don't rely on any single product.

R1Soft: This is the solution I was looking to replace with Veeam. R1Soft runs the UI on the backup servers, but with Veeam I had to set up a Windows instance for this purpose. While I like the power of the VBR interface, it's a pain to have to set up a Windows instance just for it. I also found out that Veeam doesn't support backing up MariaDB databases. Since this is the default database for my Linux distro (RHEL/CentOS), I can't switch to Veeam for my database backups. R1Soft still seems to support MariaDB.

Acronis Cyber Backup: I see this as your most serious competitor. If you haven't already, I would strongly suggest signing up for their trial product and seeing how it works. Their bare metal restore process for Linux is silky smooth compared to all the manual steps I need to carry out with Veeam (mostly because Acronis handles disk partitioning and mdraid in a superior way). Also, after an Acronis restore, my systems are immediately bootable. With Veeam, that's never the case. I always have to regenerate initramfs, check mdadm.conf, or fiddle around with some other critical things that I'd rather not be doing in a DR scenario. In fact, I would probably use Acronis if not for their odd infrastructure requirements. With Veeam, I can build my own repositories on various hardware or VMs, and I can achieve redundancy by creating backup copy jobs. Acronis lets you create just a single backup destination on your own infrastructure (similar to a Veeam repo). If you want more, then you need to license the software as a service provider, which lets you set up your own complex Acronis-centered infrastructure. Otherwise, you've got to buy backup storage space from a provider and hope that it's reliable.
Post Reply

Who is online

Users browsing this forum: No registered users and 11 guests