Host-based backup of VMware vSphere VMs.
Post Reply
davecla
Enthusiast
Posts: 26
Liked: 4 times
Joined: Feb 03, 2016 9:40 pm
Full Name: Dave Clarke
Contact:

Backup Copy Jobs not working when repository is Linux

Post by davecla »

I've got a local veeam repository on linux have been running backup copy job to a NTFS based repository at secondary site for years - all was working well.

Wanted to move repository at remote site to linux. Created new repository and copied over local backups. Remapped Backup Job. Backup of servers at remote site to repository at remote site are working fine.

Want to move backup copy jobs to new remote linux repository. Copied over files and remapped copy job. Copy job now failed with error "No connection could be made because the target machine actively refused it xx.xx.xx.xx:2500.

I've got no idea how the backup job can work, but they backup copy does not. Anyone got any pointers?

thx
dave
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by nitramd »

Firewall issue? Veeam B&R needs access to ports 2500-5000 on the repository.

Have you checked your Linux log files for clues?
zx81
Service Provider
Posts: 13
Liked: 1 time
Joined: Nov 24, 2016 6:57 am
Location: Perth, Australia
Contact:

[MERGED] Linux Backup repository refusing connections

Post by zx81 »

Case ID# 03124279

I'm having intermittent problems with a Veeam linux repository. This repository is shared between a Veeam Backup and Replication server and a separate Veeam Cloud Connect server. I understand that you can share a repository in this way, but must take into account the number of concurrent tasks configured for this repository on each Veeam Backup server. And that the two servers obviously can't know how many tasks the other is running. I understand that the recommendation is for 1 task per core.

The repository in question is a physical machine with 2 CPUs, 12 cores and 96GB of RAM. I'm seeing intermittent backup copy job failures with the error No connection could be made because the target machine actively refused it 10.x.x.x:2514.

Is there a way to debug this issue on the linux repository server itself? When the problem is happening I can't find any issues on the linux server. It has plenty of free RAM, CPU and disk IO. It's clear that at some point the backup repository is refusing backup copy job data for some reason, but I've searched the syslog and auth logs for issues related to ssh or anything else and can't find any problems. I also notice that each backup copy job even when idle maintains a running process in the form /tmp/VeeamAgentxxxx, so I'm not sure how to work out if I'm exceeding the recommended number of tasks at a given time.

In short, does Veeam log anything on the linux repository that could help me determine why it seems to be refusing connections some of the time? How can I identify which processes to count to ensure I'm not exceeding 1 task per core?

Thanks,
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Linux Backup repository refusing connections

Post by PTide »

Hi,

Have you tried to run a backup job while the repo is refusing backup copy jobs? What is your source repository?

Thanks
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by PTide »

Hi Dave, sorry for the delay with response.

Have you contacted our support team already? If not then please do so and let us know your case ID.

Thanks
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by nitramd »

@zx81, have you double checked your firewall settings? Ports 2500-5000 should be open on the repository.
zx81
Service Provider
Posts: 13
Liked: 1 time
Joined: Nov 24, 2016 6:57 am
Location: Perth, Australia
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by zx81 »

@nitramd There aren't any firewall rules between the Veeam Server and the repository. The problem is intermittent, the backups work most of the time. I could see how a firewall error that didn't open the range completely could cause this issue, but unfortunately that's not the case here.
zx81
Service Provider
Posts: 13
Liked: 1 time
Joined: Nov 24, 2016 6:57 am
Location: Perth, Australia
Contact:

Re: Linux Backup repository refusing connections

Post by zx81 »

PTide wrote: Have you tried to run a backup job while the repo is refusing backup copy jobs? What is your source repository?
Hi,

Unfortunately I've never managed to catch the problem while it's live. Whenever I login to check and then force the job to Sync Now, it works. The source repository is another Linux based repository at a different site.
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by nitramd »

@zx81,

Have a look at this post regarding a socket being forcibly closed - it may or may not apply: https://forums.veeam.com/veeam-backup-r ... 46532.html
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by PTide »

zx81,

I've briefly reviewed your case and noticed that there were actually two support engineers who handled your case, and one of them (from VCSP support team) asked you to send them the logs so they could find out why exactly the target datamover process wasn't listening. Have you done that? The thing is that there are many reasons why the error could have occurred, including the target datamover process agent crash. On the other hand, if it was really due to lack of CPU, the datamover shouldn't have crashed. Please let the support team investigate the root cause.

Thank you.
zx81
Service Provider
Posts: 13
Liked: 1 time
Joined: Nov 24, 2016 6:57 am
Location: Perth, Australia
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by zx81 »

There are no outstanding requests for logs for the case (ID# 03124279) I referenced? I've already provided logs from the backup server, including that of the target repository that is refusing connections. So I'm not sure where you are getting that I haven't proactively been working with support on this case?

Maybe you're talking about yet another case for Veeam Cloud Connect? I'm also battling several Veeam problems that aren't the subject of a case (including ReFS hard hangs) and you're right, I haven't found time to get those logs yet. I have to prioritize the work on several Veeam problems.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by PTide »

I'm referencing case 03124279. Right, there seemingly were no outstanding requests, however the engineers have switched at some point, so I wanted to clarify whether or not the substitute engineer got your logs (I've contacted him already, so he will review the case again).

Thanks
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by Andreas Neufert »

Default SSHD Service on Linux is not optimized for the Veeam usage (multiple connections). Please change:
etc/ssh/sshd_conf settings:
ClientAliveInterval 30
TCPKeepAlive yes
ClientAliveCountMax 99999
MaxSessions 200
MaxStartups 100:30:200
davecla
Enthusiast
Posts: 26
Liked: 4 times
Joined: Feb 03, 2016 9:40 pm
Full Name: Dave Clarke
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by davecla » 2 people like this post

Changes to /ssh/sshd_conf per @Andreas fixed our "target machine actively refused it xx.xx.xx.xx:2500" error.

All jobs completing without error now. Thanks all.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by Andreas Neufert »

Hi Dave, thanks. Which Linux OS Version do you use?
davecla
Enthusiast
Posts: 26
Liked: 4 times
Joined: Feb 03, 2016 9:40 pm
Full Name: Dave Clarke
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by davecla »

Debian Stretch for the repo servers
jeff.richeson
Enthusiast
Posts: 25
Liked: 2 times
Joined: Apr 25, 2017 1:58 am
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by jeff.richeson »

Is this information in the documentation? I've been fighting this exact problem for months. When I opened a ticket, I was told that it was a connectivity issue (network, firewall). Never was I directed to look at sshd configs.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by PTide »

Hi,

The issue will be described in a KB article soon. Also we're thinking about adding sshd auto-tweak function into the current linux repository deployment routine.

Thanks!
RubinCompServ
Service Provider
Posts: 259
Liked: 65 times
Joined: Mar 16, 2015 4:00 pm
Full Name: David Rubin
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by RubinCompServ »

Here we are 9 months later, and this is still an issue. I spent countless hours over the course of a month with Veeam support and my SA/SE before we discovered this forum post. Has it been made into a KB article that we all just missed? If not, are there still plans to do so? What about the "auto-tweak" that was mentioned?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by foggy »

Hi David, the KB should go online in a couple of days.
chris.arceneaux
VeeaMVP
Posts: 667
Liked: 358 times
Joined: Jun 24, 2019 1:39 pm
Full Name: Chris Arceneaux
Location: Georgia, USA
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by chris.arceneaux »

RubinCompServ
Service Provider
Posts: 259
Liked: 65 times
Joined: Mar 16, 2015 4:00 pm
Full Name: David Rubin
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by RubinCompServ »

Thanks Chris but, FYI, this isn't just with Backup Copy jobs, but with regular Backup jobs as well.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by PTide »

@RubinCompServ,

The KB should apply regardless of a job type. If it doesn't, please reach out to our support team so they can pinpoint the problem. Once you open a ticket, please update this thread with the ticket ID.

Thanks!
RubinCompServ
Service Provider
Posts: 259
Liked: 65 times
Joined: Mar 16, 2015 4:00 pm
Full Name: David Rubin
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by RubinCompServ »

It does, indeed, apply (and has been in place in my environment for 5 months now). I'd just like to see the KB updated to reflect that this is not limited to Copy jobs.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup Copy Jobs not working when repository is Linux

Post by PTide » 1 person likes this post

No problem, we'll do that, thank you for the feedback!

Cheers
Post Reply

Who is online

Users browsing this forum: No registered users and 81 guests