problems on ubuntu 16.04 LTS (linode.com VPS)

[dgomes]

When installing Veeamsnap .deb package (x64) on an ubuntu 16.04 LTS virtual machine hosted on the linode.com service I get the following warnings.
If I'm not mistaken the VMs at linode.com run on KVM virtualization.

Code: Select all

root@ubuntu:/tmp# dpkg -i veeamsnap*
(Reading database ... 32281 files and directories currently installed.)
Preparing to unpack veeamsnap_1.0.0.678_all.deb ...

------------------------------
Deleting module version: 1.0.0.678
completely from the DKMS tree.
------------------------------
Done.
Unpacking veeamsnap (1.0.0.678) over (1.0.0.678) ...
Setting up veeamsnap (1.0.0.678) ...
Loading new veeamsnap-1.0.0.678 DKMS files...
dpkg: warning: version '4.8.6-x86_64' has bad syntax: invalid character in revision number
It is likely that 4.8.6-x86_64-linode78 belongs to a chroot's host
Module build for the currently running kernel was skipped since the
kernel source for this kernel does not seem to be installed.

Also had dependency errors but the suggested "apt-get install -f" resolved it.

The packages still seem to get installed:

Code: Select all

root@ubuntu:/tmp# dpkg -l | grep veeam
ii  veeam                          1.0.0.678                       amd64        Veeam Agent for Linux
ii  veeamsnap                      1.0.0.678                       all          Veeam Agent for Linux kernel module

However the backup job then immediately fails at creating snapshot as I can see in my veeam B&R console:


Everything else seems to function as I was able to create to job and point it to my lab install of veeam B&R on the other side of the planet with no issues (ubuntu is in tokyo, veeam B&R in montreal). I'm assuming because of the veeamsnap install warnings, it's not properly installed/configured so can't do any snapshots.

If you need logs or other system info please provide a forum username or email I can send to, do not feel like putting that info in public.
Thanks,

[nielsengelen]

We'll need the logs from /var/log/veeam to see why the loading of the module failed.

[PTide]

Also please provide dmesg -T output as kernel logs are also needed.

Thanks

[dgomes]

I've sent the log folder and a text file with the output of dmesg -T to both of you.
Thanks,

[tsightler]

I believe this error message is as simple as the fact that the kernel source package is not installed so the module could not be compiled:

Module build for the currently running kernel was skipped since the
kernel source for this kernel does not seem to be installed.

I'd try running the following to install the proper headers:

Code: Select all

sudo apt-get install linux-headers-`uname -r`

Then running the following to attempt to rebuild the module:

Code: Select all

sudo dpkg-reconfigure veeamsnap

[nielsengelen]

Please try what Tom stated, could you also try the following afterwards:
lsmod | grep veeam
modprobe veeamsnap

And post the output from that? Did you by any chance reboot the server after the install?

[PTide]

I believe this error message is as simple as the fact that the kernel source package is not installed so the module could not be compiled

Oh, I must have overlooked that in the error message...

However, even if you manage to install the module it won't work because BETA2 does not support kernel 4.8, which, I believe, is exactly the one that DKMS thinks you are using ATM:

It is likely that 4.8.6-x86_64-linode78 belongs to a chroot's host

Are you trying to install it in a chrooted environment?

If you'd like to test out the version that supports kernel 4.8 I could PM share a link with you, just let me know.

Thanks

[dgomes]

I would be interested in testing the one compatible 4.8, but not sure I will be able to test immediately. On linode.com it is not too fun to get the kernel source package installed, a lot of steps to go through and they suggest doing a backup first... which is what I want Veeam for. Catch 22 here

[PTide]

Then let's try to install the BETA2 module manually and see if it will work:

- extract all source files from the veeamsnap package and place them into a directory on the VM
- cd into the directory and enter the following:

Code: Select all

make all
make load

- after that try to install 'veeam' package manually and check what dependencies are not met, install all of them manually (except 'veeamsnap')
- install 'veeam' package manually (make dpkg to ignore 'veeamsnap' dependency):

Code: Select all

dpkg --install --ignore-depends=veeamsnap veeam_1.0.0.678_amd64.deb

(synthax from the top of my head, you'll need to check the man page for dpkg)

Please let me know how it goes.

Thanks

[dgomes]

Before seeing your message I did the linode.com procedure for changing the kernel version

Code: Select all

root@ubuntu:/var/log/veeam# uname -a
Linux ubuntu 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

And install was a success without errors.

Only issue I have now is for some reason the linux agent is trying to connect to the LAN IP of my test VBR server (port 2500) instead of public, but only after the backup starts. During the configuration, connection is successful and allows me to select the repo.

Code: Select all

     2016-11-29 20:58:26 Creating volume snapshot                                                                                                                  00:00:00
     2016-11-29 20:58:32 [error] Failed to connect to the port [192.168.5.33:2500].
     2016-11-29 20:58:32 [error] Failed to perform backup

192.168.5.33 is the LAN IP of our lab's VBR server.
I will look at logs on both ends to try and see what is going on.

[tsightler]

Yeah, I just fired up a Linode so that I could poke at this and I came to the conclusion that switching to the distro provided kernel is definitely the easiest way. It's nice that Linode provides a simple way to use newer mainline kernels, but the way they do it makes it really difficult to use 3rd party modules. The procedure to switch to the distro kernel is quite straightforward.

Regarding your other issue, that's somewhat expected if you've either a) added the repository/gateway by the IP address, or b) the VBR server itself resolves it's name to the LAN address. The problem is that the VBR server resolves the IP of the repository and passes that to VAL. You can make this work, but you have to trick the local VBR server into thinking it lives at the public IP address so that it sends that address to VAL when it tells it to connect back.

[dgomes]

Regarding your other issue, that's somewhat expected if you've either a) added the repository/gateway by the IP address, or b) the VBR server itself resolves it's name to the LAN address. The problem is that the VBR server resolves the IP of the repository and passes that to VAL. You can make this work, but you have to trick the local VBR server into thinking it lives at the public IP address so that it sends that address to VAL when it tells it to connect back.

In our test scenario the VBR is itself the backup repo with local storage, and all is done by DNS names rather than IP. Is the linux agent not intended to use traditional backup repos on the VBR server and instead only use the VBR as a bridge to cloud connect repos? The only way I can see it working like you said is to trick it to resolve the public IP, which would be either playing with local DNS zones or the VBR's hosts file, both of which don't seem like something anyone would ever want to do in a production environment, especially if the VBR is managing local backups as well.

Added note: only documentation we've been able to find for setup is this: https://www.veeam.com/blog/how-to-backup-linux.html which sadly doesn't cover the option of pointing to a VBR server.

Thanks,

[nielsengelen]

You can configure the agent to send backup to a Veeam repository and then configure a backup copy job to your cloud connect service provider. All you need to configure on the repository is agent settings and make sure you are running VBR 9.5

[tsightler]

I might be mistaken, but I was under the impression that he was just trying to get his cloud based Linode server backed up to his local VBR server, not to a service provider.

VBR is really designed to work on a LAN, it doesn't really deal well with NAT or dual-homed public/private setups. That's part of what Cloud Connect is designed to address, however, I don't believe that v1 of the Linux agent will have direct support for Cloud Connect repo's, although my understanding is that it is planned.

However, you can get creative and make it work, but you have to do something so that the VBR server sends the public IP address to the VAL client to tell it to connect. Here's a document I put together that describes a way to do direct backup from Endpoint/VAL to a VBR repo behind NAT:

https://drive.google.com/open?id=0B8s_E ... 0FNUHhKNVk

I put this together to allow my laptop to backup while I roam around without me having to bring up my VPN connection to my home network, but I've had a couple of other clients set this up for cases where they have NAT between endpoints and the VBR server, which sometimes happens even internally.

The same technique can be used for public/private IP, but it's a little more difficult when the VBR server is the repo. You can probably still be creative and do something like manually add a secondary repo to the VBR server itself using it's own public IP, but I haven't tried this.

[dgomes]

This seems to do exactly what I was trying to test. Although I don't think it's something I will be able to sell to our customers having to mess around so much with public IP and such. They currently back up through SSH their files / configs from cloud servers -> local using scripts. It's fun to play around with for a person like me in the lab, but I will likely not get this to pass in majority of production environment that I can think of.

Is there any plans to make VBR understand that the agent is not local and to pass the correct info? I guess the best way to describe it would be a reverse of a cloud connect for agents.

[backup_wonder]

[...]
VBR is really designed to work on a LAN, it doesn't really deal well with NAT or dual-homed public/private setups.
[...]

Thanks for that hint, unfortunatley this is currently the showstopper for us.

[simeon.hemus]

I am trying to backup a CentOS Server located in a datacentre that i have no access to the network.
I have asked the support people to open up specific ports, such as TCP 10002, 4444 & 2500 so i can Backup this server to our Veeam Backup Server, and it starts backing up successfully, but after a while it fails because i'm using openvpn to connect this CentOS box to our firewall using an SSL VPN.
CentOS Kernal: 2.6.32-504.30.3.el6.x86_64
Firewall: WatchGuard XTM850 using a Mobile SSL VPN

It appears that the backup job is failing because the VPN is dropping out.

Is there any way i can tweak the vpn or maybe come up with another solution where i can backup this server to our Veeam Backup Server in a different site?

[PTide]

Hi Simeon,

What was the error? Also how far did the job go before failure?

Is there any way i can tweak the vpn or maybe come up with another solution where i can backup this server to our Veeam Backup Server in a different site?

You need to ensure that the connection is stable throughout the whole backup job. VBR can resist to very bad high latency connections and minor packet loss but if the connection is dropped, the job fails. Cloud Connect is designed to address such problems, however Cloud Connect support will be added to VAL later.

Thanks

Thanks

[nielsengelen]

OpenVPN does offer a start-up option which might help called --tun-mtu
You can increase the MTU size of the tun adapter to an amount of bytes both on the server & client side. This resembles Jumbo frames on a regular Ethernet LAN. Note that the MTU size on the underlying network switches was not altered.

You can try to find the best parameters by using --mtu-test and monitoring iperf to see how much mbit you utilize and need.

[dgomes]

I am trying to backup a CentOS Server located in a datacentre that i have no access to the network.
I have asked the support people to open up specific ports, such as TCP 10002, 4444 & 2500 so i can Backup this server to our Veeam Backup Server, and it starts backing up successfully, but after a while it fails because i'm using openvpn to connect this CentOS box to our firewall using an SSL VPN.
CentOS Kernal: 2.6.32-504.30.3.el6.x86_64
Firewall: WatchGuard XTM850 using a Mobile SSL VPN

It appears that the backup job is failing because the VPN is dropping out.

Is there any way i can tweak the vpn or maybe come up with another solution where i can backup this server to our Veeam Backup Server in a different site?

In watchguard there is not much you can tweak for SSLVPN settings.
You might try to increase the timeout settings:

And auto-reconnect:

Hopefully Veeam will just see it as a packet drop if the re-connection is fast enough.
Also be sure you don't have any rules limiting the inbound VPN traffic.

Other than that you can attempt to modify client-side settings as suggested by vmniels, but I do not know if the VPN service on the watchguard box will take them into account.

[simeon.hemus]

i sent the error logs to Veeam support and this is what they replied with:

Code: Select all

Hello Simeon,

Thank you for the log files.  Unfortunately the only error message in Veeam logs is:
..
[10.01.2017 07:16:25] <140595021821696> lpbcore| WARN|Method invocation was not finalized. Method id [12]. Class: [N10lpbcorelib11interaction11ILpbServiceE]
[10.01.2017 07:16:25] <140595021821696> lpbcore| ERR |Failed to connect to the port [192.168.1.199:10002].
[10.01.2017 07:16:25] <140595021821696> lpbcore| >>  |Unable connect to backup server 192.168.1.199:10002.
[10.01.2017 07:16:25] <140595021821696> lpbcore| >>  |--tr:Failed to get client for VBR server [{7fc246e4-4ed5-4a3c-ad76-27ca79e1b533}].
..
Since this is an internal IP address, I can suggest that it could be a VPN issue. Hovewer, I'd recommend checking vpn logs (usually it is located in /var/log/syslog, depending on your VPN software).

dmesg also doesn\t show anything useful, only some firewall blocked ports:
..
Jan 10 07:16:19 dse-vmmxq3bn kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9a:04:f8:00:23:e9:34:fe:83:08:00 SRC=113.20.17.238 DST=113.20.7.235 LEN=44 TOS=0x00 PREC=0x00 TTL=238 ID=19639 PROTO=TCP SPT=22058 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 
Jan 10 07:16:20 dse-vmmxq3bn kernel: Firewall: *TCP_IN Blocked* IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:23:e9:34:fe:83:08:00 SRC=113.20.17.238 DST=113.20.7.239 LEN=44 TOS=0x00 PREC=0x00 TTL=239 ID=28193 PROTO=TCP SPT=32572 DPT=23 WINDOW=14600 RES=0x00 SYN URGP=0 
Jan 10 07:16:32 dse-vmmxq3bn xinetd[1945]: START: ftp pid=24304 from=::ffff:203.88.112.237
..


Kind regards,
Konstantin Primakov
Veeam Technical Support

So i would assume that would confirm my suspicions that the VPN is dropping out.

The other thing that i'll mention is that my Veeam Backup & Replication version is: 9.5.0.711 (the first release of 9.5, the Agent for Linux says it is only supported by 9.5 update 1, but i have actually got a test CentOS VPN residing at another site that i control the hardware firewall and when i back this up, it is fine.

So i still suspect that it is probably openvpn being a bit unstable.
I'll try some of the suggestions above and get back to you on how i get on.

[simeon.hemus]

If i need to use the openvpn --mtu-test option, what values do i need to use with this and what iperf commands should i use to test the stability?
I do have the same timeout values as in gnomes pictured post above, but i don't have Authentication MD5 & Encyption Blowfish.
Should i try this with openvpn to be more stable??

[simeon.hemus]

I changed the watchguard SSL VPN Settings to Authentication MD5 & Encyption Blowfish and changed the time out to 120 Seconds, and started the backup again. This time it uploaded 82GB of files before it failed.
Any other ideas of how i can get the Mobile SSL VPN more stable? or do you think it is better that i setup an IPSEC Mobile VPN?? would that be more stable?

[PTide]

I don't think anyone on this forum has ever compared those two in terms of transmitting such amounts of data, I suggest you to try it.

Thanks

[dgomes]

I changed the watchguard SSL VPN Settings to Authentication MD5 & Encyption Blowfish and changed the time out to 120 Seconds, and started the backup again. This time it uploaded 82GB of files before it failed.
Any other ideas of how i can get the Mobile SSL VPN more stable? or do you think it is better that i setup an IPSEC Mobile VPN?? would that be more stable?

You also need to keep in mind the watchguard mobile VPNs are labeled as such because of their intended use: mobile devices, laptops, or out-of-office PCs.
For the kind of thing you are doing it would need their branch office VPN functionality (BOVPN) that establishes a permanent tunnel between 2 sites. We do multi-terabyte backups via BOVPN for several customers with a watchguard at each site and it works very well.

[prehcm]

I'd like ot backup a physical server in the cloud to a local Repository.
I can setup my Firewall to redirect TCP traffic coming from my external server's IP only on ports 100002, 2500 to 5000, 49152-65535 to the Server where the Repository runs on according to: https://helpcenter.veeam.com/docs/agent ... tml?ver=10

I was however wondering if this is considered "safe" or if there is a "better practice"?

[nielsengelen]

I would suggest creating a VPN connection between the servers instead of opening it up to the world wide web.

[PTide]

@prehcm

Niels is spot on - the number of ports that needs to be opened is way too high. Please review the thread - it contains some considerations that might be useful.

Thanks

[prehcm]

@PTide @vmniels

I have now read this whole thread and I'm not worried about getting it to work through the FW as much as I was curious to know if there is a more elegant way.

I guess I'll use the 30 days demo to test if I can get a VPN tunnel up with the pre-job scripting and shutting the tunnel down with the post-job script.
If that works alright I'll buy the server version of the agent.

[dgomes]

@PTide @vmniels

I have now read this whole thread and I'm not worried about getting it to work through the FW as much as I was curious to know if there is a more elegant way.

I guess I'll use the 30 days demo to test if I can get a VPN tunnel up with the pre-job scripting and shutting the tunnel down with the post-job script.
If that works alright I'll buy the server version of the agent.

I haven't found a way to do it more elegantly. Customer has hundreds of linux VMs at different providers like linode. He's considering dropping veeam soon, he does not want to have to do weird "hacks" even if they work, which I guess makes sense at this scale. Currently he has veeam for local backups and I told him to wait for 9.5 to check this out but he's losing patience now. He doesn't accept that he can't just simply point his cloud linux endpoints to the VBR server via WAN IP and have it work out of the box. So I've pretty much dropped this project and our boss no longer wants us offering this feature to customers when doing our infrastructure consulting.