Linux Backup repository SSHD issue since 6.5 upgrade

[AlainRussell]

We currently use a Linux server (ReadyNAS Pro) as a backup repository. Since the 6.5 upgrade the NAS has stopped responding (full hang) a couple of times - if I can log in fast enough before a forced restart I can see a lot of sshd processes running (2180 at current count). This NAS has no external access and apart from Veeam backup I am the only person with a root login.. Has anyone else experienced something similar?

Netstat output shows all the connections are from the Veeam Server

Code: Select all

tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:21070 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:19235 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:21263 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:21361 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:19213 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:21049 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:19530 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:25312 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:23377 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:17446 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:19406 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:17484 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:27040 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:22699 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:26282 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:23528 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:19282 ESTABLISHED
tcp6       0      0 NASNAME:ssh             ip-IP-OF-VEEM-SERVER.:21339 ESTABLISHED

Example ps -aux output shows

Code: Select all

root     19455  0.0  0.0   8720   248 ?        Ss   Nov11   0:01 sshd: root@notty 
root     19457  0.0  0.0   8900   200 ?        Ss   Nov12   0:00 sshd: root@notty 
root     19465  0.0  0.0   8716   156 ?        Ss   Nov12   0:01 sshd: root@notty 
root     19471  0.0  0.0   8756   104 ?        Ss   Nov12   0:01 sshd: root@notty 
root     19483  0.0  0.0   8752   128 ?        Ss   Nov11   0:01 sshd: root@notty 
root     19495  0.0  0.0   8792    96 ?        Ss   06:38   0:00 sshd: root@notty 
root     19512  0.0  0.0   8860   232 ?        Ss   14:31   0:00 sshd: root@notty 
root     19525  0.0  0.0   8616   180 ?        Ss   Nov12   0:01 sshd: root@notty 
root     19545  0.0  0.0   8720   128 ?        Ss   Nov12   0:01 sshd: root@notty 
root     19551  0.0  0.0   8784   216 ?        Ss   Nov12   0:01 sshd: root@notty 
root     19559  0.0  0.0   8840   172 ?        Ss   Nov11   0:01 sshd: root@notty 
root     19564  0.0  0.0   8720   252 ?        Ss   Nov12   0:01 sshd: root@notty 
root     19568  0.0  0.0   8716   256 ?        Ss   Nov11   0:01 sshd: root@notty 
root     19597  0.0  0.0   8772   184 ?        Ss   06:39   0:00 sshd: root@notty

[AlainRussell]

Update on this issue - I had to restart the NAS late this afternoon - since restart I can see a new connection established every few minutes..

Code: Select all

root      6766  0.0  0.2   8752  2952 ?        Ss   18:31   0:00 sshd: root@notty 
root      6850  0.0  0.2   8748  2948 ?        Ss   18:33   0:00 sshd: root@notty 
root      7017  0.0  0.2   8752  2952 ?        Ss   18:35   0:00 sshd: root@notty 
root      7101  0.0  0.2   8716  2920 ?        Ss   18:37   0:00 sshd: root@notty 
root      7272  0.0  0.2   8780  2984 ?        Ss   18:40   0:00 sshd: root@notty 
root      7356  0.0  0.2   8720  2928 ?        Ss   18:41   0:00 sshd: root@notty 
root      7522  0.0  0.2   8716  2924 ?        Ss   18:44   0:00 sshd: root@notty 
root      7689  0.0  0.2   8716  2924 ?        Ss   18:46   0:00 sshd: root@notty 
root      7773  0.0  0.3   8932  3120 ?        Ss   18:48   0:00 sshd: root@notty 
root      7857  0.0  0.2   8716  2920 ?        Ss   18:49   0:00 sshd: root@notty 
root      8023  0.0  0.3   8896  3152 ?        Ss   18:52   0:00 sshd: root@notty 
root      8112  0.0  0.2   8720  2916 ?        Ss   18:53   0:00 sshd: root@notty 
root      8278  0.0  0.2   8716  2924 ?        Ss   18:56   0:00 sshd: root@notty 
root      8444  0.0  0.2   8716  2920 ?        Ss   18:59   0:00 sshd: root@notty 
root      8529  0.0  0.2   8716  2916 ?        Ss   19:00   0:00 sshd: root@notty 
root      8613  0.0  0.2   8716  2924 ?        Ss   19:01   0:00 sshd: root@notty 
root      8697  0.0  0.2   8808  3032 ?        Ss   19:03   0:00 sshd: root@notty 
root      8781  0.0  0.2   8744  2956 ?        Ss   19:04   0:00 sshd: root@notty 
root      8865  0.0  0.2   8900  3032 ?        Ss   19:05   0:00 sshd: root@notty 
root      8949  0.0  0.2   8756  2956 ?        Ss   19:07   0:00 sshd: root@notty 
root      9038  0.0  0.3   8964  3160 ?        Ss   19:08   0:00 sshd: root@notty 
root      9122  0.0  0.2   8788  3024 ?        Ss   19:09   0:00 sshd: root@notty 
root      9288  0.0  0.2   8768  3016 ?        Ss   19:12   0:00 sshd: root@notty 
root      9373  0.0  0.2   8772  3016 ?        Ss   19:13   0:00 sshd: root@notty 
root      9539  0.0  0.2   8720  2924 ?        Ss   19:16   0:00 sshd: root@notty 
root      9626  0.0  0.2   8716  2924 ?        Ss   19:18   0:00 sshd: root@notty 
root      9710  0.0  0.2   8720  2968 ?        Ss   19:19   0:00 sshd: root@notty 
root     10045  0.0  0.2   8764  3016 ?        Ss   19:24   0:00 sshd: root@notty 
root     10130  0.0  0.2   8716  2924 ?        Ss   19:26   0:00 sshd: root@notty 
root     10214  0.0  0.2   8752  2952 ?        Ss   19:27   0:00 sshd: root@notty 
root     10380  0.0  0.3   8904  3136 ?        Ss   19:30   0:00 sshd: root@notty 
root     10546  0.2  0.2   8716  2920 ?        Ss   19:32   0:00 sshd: root@notty

[AlainRussell]

A bit more info - it appears that this is caused by the new monitoring in Veeam one 6.5 - as soon as I remove the Veeam Backup server from the Backup protection monitoring the number of SSH processes stops increasing, if I re-add the Veeam server the number of connections continues to increase... assuming the polling is every 2-3 minutes?

[Vitaliy S.]

If I recall correctly Veeam ONE is asking Veeam B&R to update Veeam backup repository services health state and other information every 5 minutes or so. Can you please contact our support team to verify why the number of connections is increasing all the time?

[tsightler]

I'm able to confirm this behavior in my lab as well. This actually appears to be a bug in B&R in that it doesn't appear to be closing connections properly.

[AlainRussell]

I've submitted a case to support on this (#00155518) but to date they have not requested B&R logs, only Veeam One logs.

[Vitaliy S.]

Hi Alain, we were able to reproduce this behavior internally and our dev team has confirmed that this is a bug. Please keep the ticket open to be notified when the hotfix for B&R becomes available. Thanks!

[tsightler]

I recently had the Linux server in my lab run out of memory due to thousands of SSHD sessions from this (it didn't crash, but became very slow with tons of swapped out memory). I was able to tell that the Veeam.Backup.WmiServer.exe process was holding the connections open so I implemented a simple "workaround" of my own. I scheduled a tasks via Windows that runs the following command once an hour:

Code: Select all

taskkill /IM Veeam.Backup.WmiServer.exe /F

The process will automatically respawn the next time Veeam ONE polls so it doesn't seem to cause any major issues. One minor issue that I did notice, I initially scheduled this task to run every hour at the top of the hour. Because the default Veeam collection was scheduled for 3:00AM, this collection failed because the Windows system would kill the process just as it was starting. I changed the Task Scheduler to run the taskkill job at 1/2 hour and so far I've found no other impact.

This is not an official workaround from support, but it seems to work for me so I just wanted to share it as a possible option until an actual hotfix/patch is produced to correct the issue.

[gse]

Yesterday we also experienced a full hang of the Linux Backup repository. Also Veeam One is installed to monitor B&R.
We will implement the "workaround" and keep an eye on the to be released fix.

[flakpyro]

Also having this issue, took me awhile to trace it back to VeeamOne but glad its not just me.

[Gostev]

By the way, this is resolved in Patch 1 which release is imminent.

[AlainRussell]

Great, thanks for the update.