FLR doesn't work on a linux machine with GIT repo / symlinks

VMware specific discussions

FLR doesn't work on a linux machine with GIT repo / symlinks

Veeam Logoby Morgenstern72 » Wed Aug 05, 2015 7:23 am

We tried a File Level Restore on a linux VM. The job run for 24 hours, restoring nothing and showing no error, so I contacted the support: Case #00977876

It was investigated and FLR had a problem with this path
Code: Select all
/media/sda7/cm/shared/apps/env/sysconf/DefaultMachine/cm/local/apps/environment-modules/current/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/3.2.10/3.2.10/Modules/default/Modules/3.2.10/default/default/default/3.2.10/init?


The path is not that long, but those folders contains symlinks which refers to the parent directory, the real path is below:
Code: Select all
/cm/shared/apps/env/sysconf/DefaultMachine/cm/local/apps/environment-modules/current


Then, the folder is composed by the following tree structure:
lrwxrwxrwx 1 root root 1 May 5 2014 3.2.10 -> .
drwxr-xr-x 2 root root 4096 May 5 2014 bin
lrwxrwxrwx 1 root root 1 May 5 2014 default -> .
drwxr-xr-x 2 root root 4096 May 5 2014 init
lrwxrwxrwx 1 root root 1 May 5 2014 Modules -> .
drwxr-xr-x 3 root root 4096 Mar 6 2014 share

The only workaround was to follow http://www.veeam.com/kb1459

The support wanted to close the case but I wanted more investigation, since we use symlinks regularly on linux VMs.

This is the last answer from support in 5th of August 2015:
I have done some research regarding FLR. Current realization of FLR for Linux is focused towards user friendly interface. Once you mount the file system, FLR enumerates all the files in that system to display them in the GUI. Because of this enumeration complicated files with deep nested structure such as git repositories are hard to enumerate, resulting in FLR failure.
I am afraid with current realization the best way to restore such files would be instant recovery.
Veeam version 9 which is about to release is bringing new features for file restore, which might meet your needs for file level restore.


So it seems to be impossible to use FLR when symlinks are present. This is an old problem I encountered many times in other backup products, but up to date I do know only veeam that still cannot handle symlinks. I hope this will be resolved fast, but be informed that FLR will not work for you in that case. I did not test with junctions/mklink on Windows but I guess it will be similar.

To be clear about the support: I was quite satisfied with the support analyzing the problem and helping me very fast. Now it's a either a bug to be fixed by the developers or something to be officially documented that veeams does not support.
Morgenstern72
Enthusiast
 
Posts: 64
Liked: 5 times
Joined: Thu Jan 30, 2014 3:37 pm
Full Name: Joachim

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Vitaliy S. » Fri Aug 07, 2015 5:28 pm

Hi Joachim,

I have passed this information to our dev team for review. I will update this topic, once I have further details.

Thanks for the detailed post!
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Morgenstern72 » Fri Aug 07, 2015 6:05 pm

Thx Vitaliy!
I hope this will be fixed :)

I was just a little shocked when the support came up with the workaround and then asked to close the ticket, since the problem was solved. Well....no. It's either a bug or symlinks are not supported. Either way I cannot use FLR anymore for linux VMs and that's quite sad.
Morgenstern72
Enthusiast
 
Posts: 64
Liked: 5 times
Joined: Thu Jan 30, 2014 3:37 pm
Full Name: Joachim

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Andreas Neufert » Mon Aug 10, 2015 8:17 pm

There where many Linux related enhancements in v8 Update 2.
Specifically with Sym Links.

Are you sure that you run on lates Patch Level v8 Update2b ? 8.0.0.2030
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Morgenstern72 » Tue Aug 11, 2015 7:18 am

Andreas Neufert wrote:Are you sure that you run on lates Patch Level v8 Update2b ? 8.0.0.2030


Yes, early adopter (got it from support prior to release) :)
Morgenstern72
Enthusiast
 
Posts: 64
Liked: 5 times
Joined: Thu Jan 30, 2014 3:37 pm
Full Name: Joachim

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Andreas Neufert » Tue Aug 11, 2015 8:16 am

Ok, then let´s wait for Vitaliys feedback, he is in the Produсt Management team.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Vitaliy S. » Fri Aug 14, 2015 11:52 am

Joachim, our QA team is trying to reproduce this behavior in order to see what can be done to fix that. Can you please share some details on how you have everything setup, so that we could catch the same error message during FLR process?
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Andreas Neufert » Fri Aug 14, 2015 11:55 am 2 people like this post

Create Linux VM
Create this folder:
Code: Select all
/media/sda7/cm/shared/apps/env/sysconf/DefaultMachine/cm/local/apps/environment-modules/current/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/3.2.10/3.2.10/Modules/default/Modules/3.2.10/default/default/default/3.2.10/init?


Create a sym link to the following:
Code: Select all
/cm/shared/apps/env/sysconf/DefaultMachine/cm/local/apps/environment-modules/current


Backup
Try Linux FLR of it
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Vitaliy S. » Sun Aug 16, 2015 8:45 pm

Ok, thanks! Our QA team is building a lab to verify some possible fixes.
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Morgenstern72 » Mon Aug 17, 2015 7:51 am

Vitaliy S. wrote:Joachim, our QA team is trying to reproduce this behavior in order to see what can be done to fix that. Can you please share some details on how you have everything setup, so that we could catch the same error message during FLR process?


Hi Vitaly

I can either offer you a remote session or a copy of the VM to test with. The VM itself was setup by a colleague of mine but I will ask him to give you some insight of what he has done.
Morgenstern72
Enthusiast
 
Posts: 64
Liked: 5 times
Joined: Thu Jan 30, 2014 3:37 pm
Full Name: Joachim

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Morgenstern72 » Mon Aug 17, 2015 7:56 am

And thank your for selecting my post as "best post of the week" :)
Morgenstern72
Enthusiast
 
Posts: 64
Liked: 5 times
Joined: Thu Jan 30, 2014 3:37 pm
Full Name: Joachim

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Andreas Neufert » Mon Aug 17, 2015 7:59 am

Was a bit scary that I saw my answer as this :?
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Vitaliy S. » Mon Aug 17, 2015 9:06 am

Morgenstern72 wrote:I can either offer you a remote session or a copy of the VM to test with. The VM itself was setup by a colleague of mine but I will ask him to give you some insight of what he has done.

Yes, that would be helpful.
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby Morgenstern72 » Mon Aug 17, 2015 1:57 pm 1 person likes this post

Here you are more details about the folder structure from my colleague Jose:

This folder is related to the “environment modules” software (http://modules.sourceforge.net/), if they want to reproduce this scenario, they can do the following:
Create a folder with the following structure
environment-modules/3.2.10

lrwxrwxrwx 3.2.10 -> .
drwxr-xr-x bin
lrwxrwxrwx default -> .
drwxr-xr-x init
lrwxrwxrwx Modules -> .
drwxr-xr-x 3 share

where 3.2.10, default and Modules are symbolic links pointing to the current folder.

More information from the bright cluster support:

“This is the default structure of the environment modules directory. The links point to the current directory.
This is required to cover references like "/cm/local/apps/environment-modules/3.2.10/Modules/default/init/bash" for exporting the modules function.”
Morgenstern72
Enthusiast
 
Posts: 64
Liked: 5 times
Joined: Thu Jan 30, 2014 3:37 pm
Full Name: Joachim

Re: FLR doesn't work on a linux machine with GIT repo / syml

Veeam Logoby danielch » Fri Sep 04, 2015 3:29 pm

Morgenstern72, I suspect I'm having a similar problem with symlinks in Windows. One of the server backups I attempted to open in FLR that makes use of symlinks heavily caused the Backup Browser to hang indefinitely (I left it open for close to 20 hours once, no results), but any other server that doesn't make use of symlinks open up in the Backup Browser nearly instantly.

One thing I noticed was the volumes in the backup actually do get mounted on the file system despite the Backup Browser staying in a perpetual "Please Wait" state. I have a C:\VeeamFLR folder on my Veeam backup server, and in here have found the mounted disks from the FLR request. Access the files directly in the file explorer from here and you can copy/paste them wherever you need.

As I was typing this, I tried FLR from a server with less symlinks than the server I was originally trying, and the Backup Browser came up after 30 minutes. I have a feeling if I had left my FLR job running longer than 20 hours for the server that had tons of symlinks it would eventually come up.

In any case, the VeeamFLR folder has the mounted volumes available after a few seconds of launching the FLR job. I've stopped the FLR job while the Backup Browser is in "Please Wait" state, and needed to restart the Veeam Backup Service to clear out the job, and used rmdir in a cmd prompt to properly remove the mounted volumes (Windows junctions) in the VeeamFLR folder.

Just wanted to share some information on what I found in my testing, using Veeam B&R 8.0.0.2030.
danielch
Influencer
 
Posts: 15
Liked: 2 times
Joined: Fri Sep 04, 2015 3:08 pm

Next

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot] and 15 guests