File Level Restore Wizard - problem mounting UFS2/EXT3

[HenrikS]

Hello!

I'm trying to use the File Level Restore Wizard on Veeam b&r v4.
I choose the right backup through the wizard, and the appliance starts.
Now, the appliance never finishes the restore process and I can not get the file browser.

The only output I can get is by hitting alt-f2, and it says:
-
Time: pit clocksource has been installed.
EXT2-fs warning: checktime reached, running e2fsck is recommended
VFS: Mounted root (ext2 filesystem).
Freeing unused kernel memory: 148k freed.
-
Nothing more happens.

Does anyone got an idea for what can cause the appliance not to proceed? I've tested with freshly installed UFS2 and EXT3 (FreeBSD and ubuntu) backups.
And on the other hand, is there a way to get a shell on the file-level restore appliance? it would probably help a lot to debug

-Henrik

[Vitaliy S.]

Hello Henrik,

We need to see a File_Level_Restore.log from Help | Support Information to tell you what causes the issue, please send us that log file to support@veeam.com.

Thank you

[HenrikS]

I have sent you the log file.
What are the requirements for the appliance? I see that the .tpl/.vmx file is set up with bridged network, and i don't have any dhcp server running on that network.

[HenrikS]

There was indeed a problem with the .tpl/.vmx.
After putting it to host-only networking, the appliance booted up, gave me a shell and i could connect through scp using the username root and no password.
The backup was lying in /media/sd1

Now the only thing i'm not getting is the file browser automaticly opening. I have to manually connect to the appliance.

[Gostev]

Henrik, this is not really a problem - the appliance is designed to use bridged networking. Sounds like you either do not have DHCP in your environment, or it does not function correctly, so the appliance cannot go on the network. Which is why the Backup Browser window never appears.

[HenrikS]

Correct, as i stated in my second post: i don't have a dhcp server running and i don't want to.
I could live with a local dhcp like if the appliance where to use NAT behind the veeam server, would that work?
I guess i should try..

[Gostev]

Oh, sorry, I missed that DHCP part.

[HenrikS]

Ok, i've now tried with NAT - same result no browser windows but i can still connect.
I have enabled a DHCP server on the network, put the appliance to bridged network and booted it.
I can see that it receives the IP, but still there is no browser window popping up.

[Gostev]

Sure, it is impossible for backup browser to establish the connection with NAT'ed appliance. This is how NAT concept works: NAT'ed computer always has to establish the connection with the server first, then server can "see" him and connect back to specific port of "parent" IP. NAT will then translate the request to NAT'ed computer.

[HenrikS]

Sorry, let me rephrase:

I tried with NAT - no go.

I _then_ went back to bridged mode, enabled a DHCP server on the network (even though i find this a bit annoying) with a scope of 1 IP,
reserved that for the appliance and went through the file level restore wizard as presented to me in the user guide.

Just to close the NAT thread, and why i tried it:
NAT performed on the HOST running the appliance (the veeam server) would make the host able to initiate a session with the appliance. (yes, it worked).
I have no technical documents telling me how the intended backup browser should be run, but when i "add" the appliance in the veeam backup and fastscp window, i can browse the whole filetrees if i'm running bridged, nat'ed or host-only networking on the appliance.

[Gostev]

Yes, this works because you are adding it into Veeam Backup tree with NAT IP (internal) address. But our current design requires DHCP and is not aware about NAT and that internal address which you can lookup inside appliance and manually specify in connection settings.

[HenrikS]

Ok, forget about me mentioning NAT.
I also said that i have now put the appliance on the host-network and it get's its IP through DHCP without NAT.
Still, i cannot see any browser windows popping up. what more is needed for this to work?

You talk about your current design: could you please tell me about it?
If i where told how this is ment to work in detail, i would solve my issue in no time

[Gostev]

Henrik, the host network will not work either - it has to be bridged network plus DHCP for backup browser to autmatically popup. When the appliance boots up, it sends special broadcast packet to port 22222, the packet contains its IP address. Veeam Backup consoles listens to this, this is how it gets the IP address of the appliance. After that, direct connection to the give IP address is installed.

[HenrikS]

Thank you for the information!

I have now used both wireshark and tcpdump on the network to see if the broadcast packet arrives, and it does not.
I can se the DHCP discover, offer, request and ack packages, but then it goes silent.
Are there any way to see if the appliance actually sends the broadcast? does it log anything?

-Henrik

[Gostev]

Yes, this is logged in /var/log/init.log, look for heartbeep... most likely in your case it would say "sendmsg: operation not permitted".

[HenrikS]

Hi again!

What i would have needed before was the broadcast address that the package is sent to: 255.255.255.255.
I thought at once that it would broadcast to the subnet it was a member of.
The log had said: sendto: network unreachable (no router)
Quite right, i had not set a default router in the dhcpd, on purpose.
After this was specified and udp 22222 was opened in the windows firewall on the veeam host, the browser pops up like it should: PROBLEM SOLVED


Now a security concern.....
With the appliance connected to the network, and also containing a default route, it is easy to:
1. know about when the appliance is online - just keep an eye on the network for broadcast packets to port udp:22222
2. connect to the appliance using scp, username root and a blank password.
3. copy all mounted backed up data on the appliance.

I would recommend you to put this in your manual so that people are aware that they need to think about the security themselves.

Alternatively, I don't get it why it is designed this way in the first place..
The browser only let's you copy the files to the servers added to the veeam backup and scp, meaning that the data transfers through the host eitherhow.
resolution: set the appliance to host-only networking. Make it send the broadcast package to the subnet-broadcast address that it get's from vmware player while booting up.

[tsightler]

Hi again!
Now a security concern.....
With the appliance connected to the network, and also containing a default route, it is easy to:
1. know about when the appliance is online - just keep an eye on the network for broadcast packets to port udp:22222
2. connect to the appliance using scp, username root and a blank password.
3. copy all mounted backed up data on the appliance.

As a user, I agree that this is a concern, I had meant to mention this during the beta. Probably the ssh server shouldn't be running by default, certainly with no password, it can be started manually if needed.

Alternatively, I don't get it why it is designed this way in the first place..
The browser only let's you copy the files to the servers added to the veeam backup and scp, meaning that the data transfers through the host eitherhow.
resolution: set the appliance to host-only networking. Make it send the broadcast package to the subnet-broadcast address that it get's from vmware player while booting up.

This part isn't quite true and can probably be blamed on me. In our testing the FLR appliance with the browser Windows is only minimally useful, it doesn't allow for the preservations of permissions and ownership info, and is VERY slow in restoring large directory trees with many files (i.e. 10's or 100's of thousands). We asked for a way to more directly access and restore these files that would allow for the preservation of permissions/ownership and work for large directory tree's like Oracle home's, etc. With direct access to the FLR appliance we can use tar over SSH, or an FTP client like lftp (after manually starting the FTP daemon on the FLR) to restore these files while preserving all file attributes and ownership, and with excellent performance. We tested restores of Oracle home directories, and large applications directories and the restores are many times faster. For example, restoring an Oracle 11g home took ~11 minutes with lftp directly from the appliance, while it took over two hours with the "file browser" and broke many symbolic links and permissions.

[HenrikS]

Hello and thank you for your reply!

It could be solved by the appliance asking you to set a password for root when it is started.
On the other hand, the browser could get a way to export files to a .tar archive as you self mentioned, to preserve file- and ownerrights.
Eitherhow, i don't find it good that this is default..

Btw: u don't need to start the ftpd to use a scripted lftp:)
$ lftp fish://root:@IP
where IP is the appliance' IP.

[tsightler]

It could be solved by the appliance asking you to set a password for root when it is started.
On the other hand, the browser could get a way to export files to a .tar archive as you self mentioned, to preserve file- and ownerrights.
Eitherhow, i don't find it good that this is default..

Btw: u don't need to start the ftpd to use a scripted lftp:)
$ lftp fish://root:@IP
where IP is the appliance' IP.

The browser exporting to .tar wouldn't be the only fix, the browser would also need to be fixed to handle a large directory tree. Right now it simply doesn't do this well.

Regarding not needing to start the ftpd service, that's certainly a change from the beta release. With the beta I had to start inetd to preform a restore via ftp, and I don't think SSH was running by default either. I'd agree that this is somewhat of a security issue.

[HenrikS]

Well, the browser is working against the SSH. it's the only service listening as default.

[tsightler]

I guess NAT mode is a reasonable option. With NAT mode the FLR appliance still works, but is relatively protected from the outsite world since the SSH and FTP ports wouldn't be forwarded to the FLR by the Windows host. Simply change the LiveCD.tpl file from "bridged" to "nat", everything will still work but there won't be any access to the FLR host from the outside.

I may very well change this myself. If I want to perform a direct restore to a Linux host I can simply switch the mode to "bridged" from the VMplayer menu, and kill/rerun udhcpcd within the VM to pull a new IP on the bridged network. This works, I just tested it.

[albertwt]

wow, thanks for sharing this steps Mr. Sightler,

I also facing the same problem in my current condition of not being able to see the browser.

I have DHCP in my domain and now i shall change the networking options in the VMWare player into NAT.

Cheers.