Comprehensive data protection for all workloads
Post Reply
albertwt
Expert
Posts: 646
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Linux guest file level recovery

Post by albertwt » Oct 28, 2010 7:55 am

Hi Everyone,

I'm trying to recover my Office document from Linux VM which has got 1 TB of datastore, somehow it failed with the error message: "Timed out waiting for FLR helper appliance to start."

my current VBR 5 server got no more than 200 GB of hard drive space, could this be the problem ?
--
/* Veeam software enthusiast user & supporter ! */

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Linux guest file level recovery

Post by Vitaliy S. » Oct 28, 2010 9:34 am

Hello Albert,

No, I don't think that it should cause any problems for FLR to work. Could you please tell if you have a DHCP server in your environment? It seems like the appliance VM failed to get an IP address. I would suggest opening a support ticket with our technical team, so we could have a look at your logs or make a webex session if needed.

Thanks!

albertwt
Expert
Posts: 646
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Linux guest file level recovery

Post by albertwt » Oct 28, 2010 9:59 am

thanks for the quick reply Vitaliy, yes I do have DHCP server in place, but somehow I've never seen the FLR appliance running :-| because it is always time out.
--
/* Veeam software enthusiast user & supporter ! */

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Linux guest file level recovery

Post by Vitaliy S. » Oct 28, 2010 10:15 am

Well... we could possibly tell you more about this issue after investigating your log files, have you already sent those to our support? By the way, just to cut out a DHCP issue, I would double-check if the Appliance actually receives the IP address or not.

albertwt
Expert
Posts: 646
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Linux guest file level recovery

Post by albertwt » Oct 28, 2010 10:30 am

yes I did that already, however I've never seen the appliance looks like since i upgraded this into v5 so i wonder how does it looks like is it a VM in the ESXi or VMware player ?
--
/* Veeam software enthusiast user & supporter ! */

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Linux guest file level recovery

Post by Vitaliy S. » Oct 28, 2010 11:15 am

VMware player is no longer needed for Multi-OS FLR operations, as we use an appliance VM which is located on the host to perform all the requests. Please have a look at our F.A.Q. section for additional information:
http://www.veeam.com/forums/viewtopic.p ... 449#p20765

On top of that, could you please tell me your ticket number, so I could update the topic with the resolution, as soon as our technical team helps you.

albertwt
Expert
Posts: 646
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Linux guest file level recovery

Post by albertwt » Oct 28, 2010 1:14 pm

thanks Vitaliy for your willingness to help, here it is the ticket number [ID#536831] Timed out waiting for FLR helper appliance to start.
--
/* Veeam software enthusiast user & supporter ! */

Gostev
SVP, Product Management
Posts: 25149
Liked: 3700 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Linux guest file level recovery

Post by Gostev » Oct 28, 2010 1:46 pm

Did you configure FLR appliance to start on the same network as Veeam Backup server on the last step of Linux FLR wizard (same place where you select host to run appliance on)?

albertwt
Expert
Posts: 646
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Linux guest file level recovery

Post by albertwt » Oct 29, 2010 12:41 am

Gostev, thank you for the direction, suddenly i realize that i select the vSwitch into TestNetwork since i don't want to cause IP conflict with the existing VM in the production,
so what I'm I suppose to do in case I just want to restore one single file only inside this big 1 TB file server VM ?
--
/* Veeam software enthusiast user & supporter ! */

tsightler
VP, Product Management
Posts: 5476
Liked: 2294 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Linux guest file level recovery

Post by tsightler » Oct 29, 2010 3:23 am

Performing file level restores does not power on the original VM, it power on a FLR appliance which pulls a DHCP address and mounts the volumes from your original server. No chance of a IP conflict, although you do need a DHCP server on the subnet which is sometimes uncommon for server subnets.

albertwt
Expert
Posts: 646
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Linux guest file level recovery

Post by albertwt » Oct 29, 2010 3:45 am

Great, thanks for your clarification Mr. Sightler,
I did the testing on the smaller Linux guest and it works (80 GB) but then when I was trying to recover the file from larger Fileserver VM 1 TB, it failed with time out error ?
--
/* Veeam software enthusiast user & supporter ! */

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Linux guest file level recovery

Post by Vitaliy S. » Oct 29, 2010 7:53 am

Hi Albert,

I didn't know that your Multi-OS FLR wizard does work on smaller VMs... In this case try setting the registry key below, to extend the timeout, should help:
http://www.veeam.com/forums/viewtopic.p ... out#p20621

albertwt
Expert
Posts: 646
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Linux guest file level recovery

Post by albertwt » Oct 29, 2010 11:44 am

thanks for your reply Vitaliy, Ekaterina Erofeeva already guide me in doing this and I'm leaving it overnight to see how it goes.
--
/* Veeam software enthusiast user & supporter ! */

tsightler
VP, Product Management
Posts: 5476
Liked: 2294 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Linux guest file level recovery

Post by tsightler » Oct 29, 2010 1:53 pm

albertwt wrote:Great, thanks for your clarification Mr. Sightler,
I did the testing on the smaller Linux guest and it works (80 GB) but then when I was trying to recover the file from larger Fileserver VM 1 TB, it failed with time out error ?
Interesting, to be honest, I might not have tried it with a Linux VM that large. I wouldn't think it would make much difference (maybe a little longer to perform journal playback. Are these ext3 volumes? Have you watched the appliance console while it's booting to see what's going on?

albertwt
Expert
Posts: 646
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Linux guest file level recovery

Post by albertwt » Oct 29, 2010 3:20 pm

Unfortunately, it is timed out still :-|
but this time i could see there is window from the FLR is opened but with nothing inside it.

FYI: I've increased the time out to be 120000 but no luck yet.
--
/* Veeam software enthusiast user & supporter ! */

tsightler
VP, Product Management
Posts: 5476
Liked: 2294 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Linux guest file level recovery

Post by tsightler » Oct 29, 2010 3:48 pm

albertwt wrote:Unfortunately, it is timed out still :-|
but this time i could see there is window from the FLR is opened but with nothing inside it.

FYI: I've increased the time out to be 120000 but no luck yet.
Was the FLR powered on? Did you actually watch it power on? It should show some messages while it boots.

albertwt
Expert
Posts: 646
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Linux guest file level recovery

Post by albertwt » Oct 30, 2010 6:33 am

yes it was, but then there is nothing in the window (eg. empty selections / nothing is mounted).
I'll update this thread after i got the solution from the Dev. team.
--
/* Veeam software enthusiast user & supporter ! */

vmbackupuser
Enthusiast
Posts: 48
Liked: 2 times
Joined: Nov 09, 2009 5:18 pm
Contact:

Re: Linux guest file level recovery

Post by vmbackupuser » Jul 29, 2011 4:45 pm

Was there any update to problem?

I'm having similar problem case currently logged as: 5139339.
Sometimes the VeeamFLR appliance is pingable during power-on and initializes fine, and shows the FLR Brower.
But most times the Veeam FLR appliance is un-pingable from the Veeam server and its initialization times-out, and the appliance powers-off and removes itself from inventory.

The address of the appliance is static and on the production network (re: same vlan where the target VMs reside).
Ping/ICMP enabled between Veeam server, FLR Appliance and target VMs
No firewall on Veeam server
Ports 2500 - 2510 are open on target VMs & ESX Hosts
IP Tables currently disabled on target VMs, for testing purposes.
I can successfully and consistently ping the target VMs from the Veeam server.
I can only sometimes ping the Veeam FLR Appliance successfully from the Veeam server. Weird!

Admittedly it would be easier to troubleshoot if it never worked. But it has worked rarely, but often does not. And no changes have been made in between the success and subsequent failures, and success, and failure, etc.. :)

Seems to be unreliable & random whether the Veeam FLR Appliance boots successfully or not.
Are there any settings to tweak responsiveness/reliability of the FLR appliance itself?
Have other come across this scenario. re: intermittent initialization of veeam FLR appliance?
Clutching at straws here, but any chance the VMware Player FLR solution can be configured as a workaround? This worked consistently for us before :)

Any ideas are welcome & much appreciated!

rbrambley
Veeam Software
Posts: 463
Liked: 55 times
Joined: Jun 16, 2009 1:23 pm
Full Name: Rich Brambley
Contact:

Re: Linux guest file level recovery

Post by rbrambley » Jul 30, 2011 12:03 am

I've had some similar issues with the Veeam FLR VM in my Veeam demo lab, but not an exact scenario match to what you describe. Maybe my workaround will help.

When you run the FLR restore job, be sure to pick a vSwitch / portgroup that will let the Veeam BaR server and the FLR VM communicate. The FLR VM does not have to talk to the original production VMs. The process is between the FLR VM and the Veeam server. The assembled VMDK(s) from the Backup files are attached to Veeam BaR via the FLR as a proxy.

The only time you would need to reach the production VM(s) is if you copied the restored file(s) back to it, and that process would be between the Veeam server and the production VM - still not the FLR VM.

MOST IMPORTANTLY - browse and pick the vSwitch/portgroup every time you run the FLR job. In my experience, even though I want to use the same FLR config from the last restore and it is already populated, it seems to only work consistently if I pick the vSwitch again. So now I do it again every time, and I no longer get the failures.

Now, if you want to use the FTP service from the FLR VM, then you need to provide the connectivity to the FLR VM.

cby
Expert
Posts: 109
Liked: 6 times
Joined: Feb 24, 2009 5:02 pm
Contact:

Re: Linux guest file level recovery

Post by cby » Aug 01, 2011 3:11 pm

Agree with Rich having recently been through the major FLR changes applied by Veeam to 5.0.2, even though 4.1.2 was working fine for us. Rgressing to 4.1.2 from 5.0.2 was not an option so bit the bullet...

Three things that sorted FLR at our place:

1. Added VMkernel network on the Veeam backup server

2 .Put the FLR appliance on the same subnet as the Veeam server

3. Used fixed IP because not running DHCP for main servers

I captured the screenshots as an aide memoire!

vmbackupuser
Enthusiast
Posts: 48
Liked: 2 times
Joined: Nov 09, 2009 5:18 pm
Contact:

Re: Linux guest file level recovery

Post by vmbackupuser » Aug 02, 2011 11:51 am

1. Thanks Rich & CBY. That was some very useful information.
I have now tried the steps as advised by Rich, specifically placing the FLR appliance on the same subnet as the Veeam Server (instead of the target VM subnet), and re-selecting the vSwitch/portgroup each time. This now appears to work fine and the FLR VM is now consistently pingable and the FLR Browser opens each time. Thanks again! The VeeamBackup User Guide needs to elaborate on the this, as you guys did ;) Hopefully FLR appliance functionality will be fine-tuned in the upcoming v6.

2. Hi Rich, my overall problems with the FLR appliance was two-fold. Part 1 has been solved by your suggestions above (Cheers!). And Part 2 is that once the FLR Browser is open and I attempt to restore a file (either onto the Veeam server or the target/live VM) I get the error:
'Failed to start client agent on host 'veeam flr appliance', Failed to install agent sourcePath 'C:\Program Files\Veeam\Backup and Replications\VeeamAgent', targetPath '/tmp/VeeamAgent...' scp:/tmpVeeamAgent..' No space left on device.'

-Have you come across this before?
-Is this a genuine space restriction error or a write permission error?
-Where precisely is it referring to?
-All datastores and disks show free space of atleast 20GB or more. The file being restored is only 174Kb.
-How much space does the VeeamAgent require to install?

I understand these are many questions, but I'm so close yet so far... :(

Advice welcomed!

Gostev
SVP, Product Management
Posts: 25149
Liked: 3700 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Linux guest file level recovery

Post by Gostev » Aug 02, 2011 11:55 am

Do you have a support case open for (2)?

vmbackupuser
Enthusiast
Posts: 48
Liked: 2 times
Joined: Nov 09, 2009 5:18 pm
Contact:

Re: Linux guest file level recovery

Post by vmbackupuser » Aug 02, 2011 12:08 pm

I do, its ref:5139339 its been open for approx 3 weeks.

Gostev
SVP, Product Management
Posts: 25149
Liked: 3700 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Linux guest file level recovery

Post by Gostev » Aug 02, 2011 12:31 pm

Support case for (2) was opened on 07/28/2011, which is far less than 3 weeks ;)
Opened as 5140576 and does refer to the previous support case for (1) in description.

rbrambley
Veeam Software
Posts: 463
Liked: 55 times
Joined: Jun 16, 2009 1:23 pm
Full Name: Rich Brambley
Contact:

Re: Linux guest file level recovery

Post by rbrambley » Aug 03, 2011 12:48 am

vmbackupuser wrote: I get the error:
'Failed to start client agent on host 'veeam flr appliance', Failed to install agent sourcePath 'C:\Program Files\Veeam\Backup and Replications\VeeamAgent', targetPath '/tmp/VeeamAgent...' scp:/tmpVeeamAgent..' No space left on device.'
Glad to hear you are getting the FLR appliance to work more consistently.

I can't say I've run across this error exactly, but my first instinct is asking are you logged on and/or using an account that has local administrator priveledges on the Veeam BaR server? Also, are all the Veeam Services running as an account that is a local admin?

The C:\ path is the Veeam BaR server - which makes we wonder if your account can't access/use that location. Not sure why space would be an issue unless you are in fact low .

I've never had to access or troubleshoot the FLR appliance. Couldn't even tell you the root account or password. You should never have to troubleshoot or worry about that either.

Veeam Support would definitely be the best avenue to get to the bottom of this. Post a quick summary here when you figure it out.

Thanks for using Veeam!

tsightler
VP, Product Management
Posts: 5476
Liked: 2294 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Linux guest file level recovery

Post by tsightler » Aug 03, 2011 1:51 am

I'd actually be suspicious that the users root and or /tmp volume on the remote target is low/out of space. I know he said that the volume he's attempting to restore to has plenty of space, but when Veeam performs a direct restore it actually pushes out a small agent to the /tmp directory and runs it from there. Another possibility, if the Linux system definitely has enough space on /tmp, is that the system might be running SElinux or some similar policy based engine, and is locked down enough to not allow SSH/SCP direct access to /tmp. I know it's generally considered an unsafe practice to allow remote users to write arbitrary files to /tmp and then execute them.

Still, I'd start with the basics of verifying that /tmp has enough space and go from there.

vmbackupuser
Enthusiast
Posts: 48
Liked: 2 times
Joined: Nov 09, 2009 5:18 pm
Contact:

Re: Linux guest file level recovery

Post by vmbackupuser » Aug 10, 2011 1:00 pm

The support agent discovered that a bug exists, in the way the FLR appliance handles errors during deployment of the agent. Apparently the agent deployed to the FLR isn't removed/deleted if the appliance had an error during its deployment. This in turn fills up the disk space in the appliance causing the error above. This is to be addressed in a future release.

I've decided to work-around the FLR Browser Window restores by using the FTP functionality of the appliance. This works fine for me.
Many thanks for all your assistance/suggesstions.

Gostev
SVP, Product Management
Posts: 25149
Liked: 3700 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Linux guest file level recovery

Post by Gostev » Aug 10, 2011 2:36 pm

Yes, I can confirm that /tmp volume issue with FLR appliance is addressed in v6. Thanks for the update!

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 48 guests