FLR doesn't work on a linux machine with GIT repo / symlinks

[Morgenstern72]

We tried a File Level Restore on a linux VM. The job run for 24 hours, restoring nothing and showing no error, so I contacted the support: Case #00977876

It was investigated and FLR had a problem with this path

Code: Select all

/media/sda7/cm/shared/apps/env/sysconf/DefaultMachine/cm/local/apps/environment-modules/current/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/3.2.10/3.2.10/Modules/default/Modules/3.2.10/default/default/default/3.2.10/init?

The path is not that long, but those folders contains symlinks which refers to the parent directory, the real path is below:

Code: Select all

/cm/shared/apps/env/sysconf/DefaultMachine/cm/local/apps/environment-modules/current

Then, the folder is composed by the following tree structure:
lrwxrwxrwx 1 root root 1 May 5 2014 3.2.10 -> .
drwxr-xr-x 2 root root 4096 May 5 2014 bin
lrwxrwxrwx 1 root root 1 May 5 2014 default -> .
drwxr-xr-x 2 root root 4096 May 5 2014 init
lrwxrwxrwx 1 root root 1 May 5 2014 Modules -> .
drwxr-xr-x 3 root root 4096 Mar 6 2014 share

The only workaround was to follow http://www.veeam.com/kb1459

The support wanted to close the case but I wanted more investigation, since we use symlinks regularly on linux VMs.

This is the last answer from support in 5th of August 2015:

I have done some research regarding FLR. Current realization of FLR for Linux is focused towards user friendly interface. Once you mount the file system, FLR enumerates all the files in that system to display them in the GUI. Because of this enumeration complicated files with deep nested structure such as git repositories are hard to enumerate, resulting in FLR failure.
I am afraid with current realization the best way to restore such files would be instant recovery.
Veeam version 9 which is about to release is bringing new features for file restore, which might meet your needs for file level restore.

So it seems to be impossible to use FLR when symlinks are present. This is an old problem I encountered many times in other backup products, but up to date I do know only veeam that still cannot handle symlinks. I hope this will be resolved fast, but be informed that FLR will not work for you in that case. I did not test with junctions/mklink on Windows but I guess it will be similar.

To be clear about the support: I was quite satisfied with the support analyzing the problem and helping me very fast. Now it's a either a bug to be fixed by the developers or something to be officially documented that veeams does not support.

[Vitaliy S.]

Hi Joachim,

I have passed this information to our dev team for review. I will update this topic, once I have further details.

Thanks for the detailed post!

[Morgenstern72]

Thx Vitaliy!
I hope this will be fixed

I was just a little shocked when the support came up with the workaround and then asked to close the ticket, since the problem was solved. Well....no. It's either a bug or symlinks are not supported. Either way I cannot use FLR anymore for linux VMs and that's quite sad.

[Andreas Neufert]

There where many Linux related enhancements in v8 Update 2.
Specifically with Sym Links.

Are you sure that you run on lates Patch Level v8 Update2b ? 8.0.0.2030

[Morgenstern72]

Are you sure that you run on lates Patch Level v8 Update2b ? 8.0.0.2030

Yes, early adopter (got it from support prior to release)

[Andreas Neufert]

Ok, then let´s wait for Vitaliys feedback, he is in the Produсt Management team.

[Vitaliy S.]

Joachim, our QA team is trying to reproduce this behavior in order to see what can be done to fix that. Can you please share some details on how you have everything setup, so that we could catch the same error message during FLR process?

[Andreas Neufert]

Create Linux VM
Create this folder:

Code: Select all

/media/sda7/cm/shared/apps/env/sysconf/DefaultMachine/cm/local/apps/environment-modules/current/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/default/3.2.10/3.2.10/Modules/default/Modules/3.2.10/default/default/default/3.2.10/init?

Create a sym link to the following:

Code: Select all

/cm/shared/apps/env/sysconf/DefaultMachine/cm/local/apps/environment-modules/current

Backup
Try Linux FLR of it

[Vitaliy S.]

Ok, thanks! Our QA team is building a lab to verify some possible fixes.

[Morgenstern72]

Joachim, our QA team is trying to reproduce this behavior in order to see what can be done to fix that. Can you please share some details on how you have everything setup, so that we could catch the same error message during FLR process?

Hi Vitaly

I can either offer you a remote session or a copy of the VM to test with. The VM itself was setup by a colleague of mine but I will ask him to give you some insight of what he has done.

[Morgenstern72]

And thank your for selecting my post as "best post of the week"

[Andreas Neufert]

Was a bit scary that I saw my answer as this

[Vitaliy S.]

I can either offer you a remote session or a copy of the VM to test with. The VM itself was setup by a colleague of mine but I will ask him to give you some insight of what he has done.

Yes, that would be helpful.

[Morgenstern72]

Here you are more details about the folder structure from my colleague Jose:

This folder is related to the “environment modules” software (http://modules.sourceforge.net/), if they want to reproduce this scenario, they can do the following:
Create a folder with the following structure
environment-modules/3.2.10

lrwxrwxrwx 3.2.10 -> .
drwxr-xr-x bin
lrwxrwxrwx default -> .
drwxr-xr-x init
lrwxrwxrwx Modules -> .
drwxr-xr-x 3 share

where 3.2.10, default and Modules are symbolic links pointing to the current folder.

More information from the bright cluster support:

“This is the default structure of the environment modules directory. The links point to the current directory.
This is required to cover references like "/cm/local/apps/environment-modules/3.2.10/Modules/default/init/bash" for exporting the modules function.”

[danielch]

Morgenstern72, I suspect I'm having a similar problem with symlinks in Windows. One of the server backups I attempted to open in FLR that makes use of symlinks heavily caused the Backup Browser to hang indefinitely (I left it open for close to 20 hours once, no results), but any other server that doesn't make use of symlinks open up in the Backup Browser nearly instantly.

One thing I noticed was the volumes in the backup actually do get mounted on the file system despite the Backup Browser staying in a perpetual "Please Wait" state. I have a C:\VeeamFLR folder on my Veeam backup server, and in here have found the mounted disks from the FLR request. Access the files directly in the file explorer from here and you can copy/paste them wherever you need.

As I was typing this, I tried FLR from a server with less symlinks than the server I was originally trying, and the Backup Browser came up after 30 minutes. I have a feeling if I had left my FLR job running longer than 20 hours for the server that had tons of symlinks it would eventually come up.

In any case, the VeeamFLR folder has the mounted volumes available after a few seconds of launching the FLR job. I've stopped the FLR job while the Backup Browser is in "Please Wait" state, and needed to restart the Veeam Backup Service to clear out the job, and used rmdir in a cmd prompt to properly remove the mounted volumes (Windows junctions) in the VeeamFLR folder.

Just wanted to share some information on what I found in my testing, using Veeam B&R 8.0.0.2030.

[danielch]

I opened a case with support and they informed me what I was seeing was due to the large number of reparse points (in my case, symlinks) in the VMs being backed up. Here's their response:

I looked over your restore logs, and it looks like we can do about 133 reparse points per second. This should be more than adequate for the vast majority of systems, but **********10, **********11, and **********12 have 16,288,902 (volume 2), 10,600,456 (volume 2), and 427,418 (volume 1) reparse points respectively.

Going based on the averages I calculated from your successful restore jobs, it should take about:

34 hours to handle the 16,288,902 reparse points for **********10's Volume2,
22 hours to handle the 10,600,456 reparse points for **********11's Volume2, and
54 minutes to handle the 472,418 reparse points for **********12's Volume 1.

By contrast, **********P1's Volume1's 395 reparse points take 3 seconds to process. Putting more RAM and/or processing cores on the Veeam server could help, potentially, as could giving it access to faster backup storage or closer proximity to the repository (if any of these are applicable to your environment).

To pose a question to any Veeam Dev's on the forum, would it be possible to process the reparse points on a folder per folder basis when file systems have more than a certain amount (say 10,000 or more)? When I ran the FLR job for these VMs, it wasn’t apparent to me what was going on, and whether the job was hung or not. At the very least, processing reparse points for a folder’s contents (not recursively for subfolders, only for that folder) when it is accessed could reduce confusion in these situations and help the end user out.

I was able to use the "Other OS FLR" job and get the VM mounted on a Hyper-V host to restore files with less of a headache, but I’d much rather get the volumes mounted locally as with the Windows FLR job than to set up the VM in a remote Hyper-V server.

[Vitaliy S.]

Our QA team is still trying to reproduce OP's issue, so far with no results, unfortunately. If you could provide your case ID, I could also escalate this internally.

[danielch]

Sure, it's Case # 01016949.