Case #05520611 Copy job fails with "Failed to start a repository agent"

[mboswell]

We're seeing increased numbers of copy jobs failing with "Failed to start a repository agent." We've scrutinized network connectivity at the request of support and haven't found anything in packet captures or monitoring. We're seeing failures at different client sites with most eventually succeeding after a number of retries. We have jobs targeting two Linux repositories and this problem seems confined to jobs targeting just one of them, so I'm looking for advice on what to look for on the Linux repo that might be causing these intermittent failures.

Thanks in advance!

[HannesK]

Hello,
and welcome to the forums.

As you say "increased"... does that mean it worked fine for some time and it starts getting worse? You as service provider... is it possible that you are onboarding more customers and sizing / configuration does not meet the requirements anymore?

I read about antivirus in the case. It's expected that antivirus causes issues. Antivirus is a common reason for problems. Excluding everything from AV that influence Veeam components is important.

That's my two guesses.

Best regards,
Hannes

[mboswell]

Yes, the case predates my time with the company but my understanding is that the errors have increased over time. I'm the Linux guy so I've been asked to look at that angle since apparently only jobs targeting one of two hosts have the error. Is there some verbose logging I can look at?

[LickABrick]

You can find more detailed job logs on the VBR server in: C:\ProgramData\Veeam\Backup\<Job Name>

EDIT: Also have a look at https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[mboswell]

Thanks. Currently exfiltrating logs from the repo for analysis. As requested, we have tested with AV and firewalls disabled on all Veeam components and the issue recurs.

[mboswell]

I have an update, and some progress on diagnosis. We have been observing the issue on a particular job that runs every 2 hours and thus a) gives us a lot of data points since we don't have to wait a day for an update and b) occurs outside of the busiest backup window, so resource utilization is negligible during this time.

The job in question is a copy job containing 3 VMs. We noticed the following pattern:
1. The job starts and the first VM completes while the next two fail immediately.
2. The job retries, skips the first VM, completes the second, and the third fails immediately.
3. The job retries, skips the first two VMs and completes the third.

We tried changing the "Max Concurrent Tasks" setting on the bandwidth tab of the tenant to 1 and now the job will complete without any failures, running each VM in sequence.

Obviously this isn't a long term solution but should help us narrow down what is causing this.