Backup Copy Jobs not working when repository is Linux

[davecla]

I've got a local veeam repository on linux have been running backup copy job to a NTFS based repository at secondary site for years - all was working well.

Wanted to move repository at remote site to linux. Created new repository and copied over local backups. Remapped Backup Job. Backup of servers at remote site to repository at remote site are working fine.

Want to move backup copy jobs to new remote linux repository. Copied over files and remapped copy job. Copy job now failed with error "No connection could be made because the target machine actively refused it xx.xx.xx.xx:2500.

I've got no idea how the backup job can work, but they backup copy does not. Anyone got any pointers?

thx
dave

[nitramd]

Firewall issue? Veeam B&R needs access to ports 2500-5000 on the repository.

Have you checked your Linux log files for clues?

[zx81]

Case ID# 03124279

I'm having intermittent problems with a Veeam linux repository. This repository is shared between a Veeam Backup and Replication server and a separate Veeam Cloud Connect server. I understand that you can share a repository in this way, but must take into account the number of concurrent tasks configured for this repository on each Veeam Backup server. And that the two servers obviously can't know how many tasks the other is running. I understand that the recommendation is for 1 task per core.

The repository in question is a physical machine with 2 CPUs, 12 cores and 96GB of RAM. I'm seeing intermittent backup copy job failures with the error No connection could be made because the target machine actively refused it 10.x.x.x:2514.

Is there a way to debug this issue on the linux repository server itself? When the problem is happening I can't find any issues on the linux server. It has plenty of free RAM, CPU and disk IO. It's clear that at some point the backup repository is refusing backup copy job data for some reason, but I've searched the syslog and auth logs for issues related to ssh or anything else and can't find any problems. I also notice that each backup copy job even when idle maintains a running process in the form /tmp/VeeamAgentxxxx, so I'm not sure how to work out if I'm exceeding the recommended number of tasks at a given time.

In short, does Veeam log anything on the linux repository that could help me determine why it seems to be refusing connections some of the time? How can I identify which processes to count to ensure I'm not exceeding 1 task per core?

Thanks,

[PTide]

Hi,

Have you tried to run a backup job while the repo is refusing backup copy jobs? What is your source repository?

Thanks

[PTide]

Hi Dave, sorry for the delay with response.

Have you contacted our support team already? If not then please do so and let us know your case ID.

Thanks

[nitramd]

@zx81, have you double checked your firewall settings? Ports 2500-5000 should be open on the repository.

[zx81]

@nitramd There aren't any firewall rules between the Veeam Server and the repository. The problem is intermittent, the backups work most of the time. I could see how a firewall error that didn't open the range completely could cause this issue, but unfortunately that's not the case here.

[zx81]

Have you tried to run a backup job while the repo is refusing backup copy jobs? What is your source repository?

Hi,

Unfortunately I've never managed to catch the problem while it's live. Whenever I login to check and then force the job to Sync Now, it works. The source repository is another Linux based repository at a different site.

[nitramd]

@zx81,

Have a look at this post regarding a socket being forcibly closed - it may or may not apply: https://forums.veeam.com/veeam-backup-r ... 46532.html

[PTide]

zx81,

I've briefly reviewed your case and noticed that there were actually two support engineers who handled your case, and one of them (from VCSP support team) asked you to send them the logs so they could find out why exactly the target datamover process wasn't listening. Have you done that? The thing is that there are many reasons why the error could have occurred, including the target datamover process agent crash. On the other hand, if it was really due to lack of CPU, the datamover shouldn't have crashed. Please let the support team investigate the root cause.

Thank you.

[zx81]

There are no outstanding requests for logs for the case (ID# 03124279) I referenced? I've already provided logs from the backup server, including that of the target repository that is refusing connections. So I'm not sure where you are getting that I haven't proactively been working with support on this case?

Maybe you're talking about yet another case for Veeam Cloud Connect? I'm also battling several Veeam problems that aren't the subject of a case (including ReFS hard hangs) and you're right, I haven't found time to get those logs yet. I have to prioritize the work on several Veeam problems.

[PTide]

I'm referencing case 03124279. Right, there seemingly were no outstanding requests, however the engineers have switched at some point, so I wanted to clarify whether or not the substitute engineer got your logs (I've contacted him already, so he will review the case again).

Thanks

[Andreas Neufert]

Default SSHD Service on Linux is not optimized for the Veeam usage (multiple connections). Please change:
etc/ssh/sshd_conf settings:
ClientAliveInterval 30
TCPKeepAlive yes
ClientAliveCountMax 99999
MaxSessions 200
MaxStartups 100:30:200

[davecla]

Changes to /ssh/sshd_conf per @Andreas fixed our "target machine actively refused it xx.xx.xx.xx:2500" error.

All jobs completing without error now. Thanks all.

[Andreas Neufert]

Hi Dave, thanks. Which Linux OS Version do you use?

[davecla]

Debian Stretch for the repo servers

[jeff.richeson]

Is this information in the documentation? I've been fighting this exact problem for months. When I opened a ticket, I was told that it was a connectivity issue (network, firewall). Never was I directed to look at sshd configs.

[PTide]

Hi,

The issue will be described in a KB article soon. Also we're thinking about adding sshd auto-tweak function into the current linux repository deployment routine.

Thanks!

[RubinCompServ]

Here we are 9 months later, and this is still an issue. I spent countless hours over the course of a month with Veeam support and my SA/SE before we discovered this forum post. Has it been made into a KB article that we all just missed? If not, are there still plans to do so? What about the "auto-tweak" that was mentioned?

[foggy]

Hi David, the KB should go online in a couple of days.

[chris.arceneaux]

KB Article: https://www.veeam.com/kb2985

[RubinCompServ]

Thanks Chris but, FYI, this isn't just with Backup Copy jobs, but with regular Backup jobs as well.

[PTide]

@RubinCompServ,

The KB should apply regardless of a job type. If it doesn't, please reach out to our support team so they can pinpoint the problem. Once you open a ticket, please update this thread with the ticket ID.

Thanks!

[RubinCompServ]

It does, indeed, apply (and has been in place in my environment for 5 months now). I'd just like to see the KB updated to reflect that this is not limited to Copy jobs.

[PTide]

No problem, we'll do that, thank you for the feedback!

Cheers