Veeam B&R V13 Upgrade Experience Troubleshooting

[john.chaney]

Hello all,

I have been using the Veeam product since it was just a sand alone esxi 2.0 instance backup for my machines. I have used Veeam Backup and Replication since version 9. My company did an upgrade from version 12 to v13 a month ago. And lie i have been reading with others, we too have been experiencing issues with the product. We have the latest version installed and mainly outside of AIX systems, just do VCENTER backups of our systems with replication going off site.

Mainly errors with

"Failed to save backup meta to NFS%" to finish off a 99% complete job.
Some jobs would lose access to our infinidat NFS share and just continue writing and reading at the peak proxy transfer rate level (1gb per second in our example) and would not let go of a job 18 hours later when the job should have completed in 10-20 minutes.
"Nothing to process: all objects were excluded from the task list" where in finding that veeam backup job has an empty list of servers to backup even though you go and edit the job and all of the servers would be listed there.
"Error: Time is out Failed to create nfs file stream. Path NFS%" just not communicating at all after starting the backup.
" Error: Unknown network error" just random

So we have support and they usually say the usual thing. is antivirus running and do you have the exclusions added - Yes
Is there any firewall or network blocking you between the NFS storage and the Veeam backup server.... No
The usual stuff and no nothing changed between V12 and V13. But complete random backup jobs would fail for us.

So finally for two days in a row, we have gotten good backups of everything and I wanted to share what I did to maybe help others. This is not a complete all and a fix for everyone. I know this but please try these few items as your own risk. I am not responsible for what you do to your systems.

With that said here is what I ended up finding out and helped us.

#1 Making sure that if you are running an infinidat system or something with a large NFS share on t with multiple IP addresses assigned, check to see if someone had a old hack put into your host file that static assigned that nfs share name with ip addresses. Get it to use DNS instead for this. We had a bunch of them in there for our infinidat and maybe that confused things with the new system.

#2 Understand that the system has lowered the amount of ports it uses for working with the veeam backup software. Mainly look at port 443 for https tcp traffic. if you are doing any inspections anywhere, you might need to lower that down a bit to allow the process to go through.
Once all components are upgraded to v13, only the following inbound ports are required: There are a few more but these are the main ones.
- TCP 443
- TCP 6160
- TCP 6162
- TCP 2500–3300
- TCP 135, 445 (Windows components only)

#3 If your network cannot handle IPv6, then disable IPv6 from the network adapter and only run ipv4. Reboot the server so only the services run under IPv4. This can cause some issues when updating or upgrading so remember to switch it back on to do the upgrades as needed.

#4 This last one might be a odd one and still dont understand why it might work. But I created a powershell script to export out what ports the veeam.backup.*.exe services are doing. This powershell will export to a .csv file what is running at the exact time. Look up how to run this command every 1 minute for 24 hours. After 1 hour if you were like what happened to me, the veeam.backup.satellite.exe service (which is checking licensing) stops running like half of the processes with veeam finally. This service mainly verifies that you are licensed for what you are doing in veeam mainly along with some other things. Once i got this script to run every minute for 1 hour, this service stopped over night and my backups continued to work without any issues (knock on wood) Create a scheduled task and run this every minute for 24 hours to the temp location. And notice the service above does not continue to use up all of your ports. See if that helps you.

I got some more things I did that helps me but see if this last one is not some sot of major bug with that service causing an issue with the system backing up things.

Powershell script (look up now to make a powershell script run every 1 minute for 24 hours)

# 1. Define the process name pattern and the output folder
$ProcessPattern = "veeam.backup.*"
$OutputPath = "C:\temp\veeamlogs" # Change this to your desired folder

# 2. Generate a dynamic filename with current date and time
$Timestamp = Get-Date -Format "yyyy-MM-dd_HH-mm-ss"
$FileName = "VeeamConnections_$Timestamp.csv"
$FullPath = Join-Path -Path $OutputPath -ChildPath $FileName

# 3. Ensure the output directory exists
if (!(Test-Path $OutputPath)) { New-Item -ItemType Directory -Path $OutputPath }

# 4. Get processes matching the pattern and find their TCP connections
$VeeamProcesses = Get-Process -Name $ProcessPattern -ErrorAction SilentlyContinue

if ($VeeamProcesses) {
$VeeamProcesses | ForEach-Object {
$Proc = $_
Get-NetTCPConnection -OwningProcess $Proc.Id -ErrorAction SilentlyContinue |
Select-Object `
@{Name="ProcessName"; Expression={$Proc.ProcessName}},
@{Name="PID"; Expression={$Proc.Id}},
LocalAddress,
LocalPort,
RemoteAddress,
RemotePort,
State
} | Export-Csv -Path $FullPath -NoTypeInformation

Write-Host "Connection data exported to: $FullPath"
} else {
Write-Warning "No processes matching '$ProcessPattern' were found running."
}