cleanup of /tmp on linux repos

[selva]

Sometimes Veeam can fill up the /tmp of a linux repo if, somehow, an error occurs, no cleanup happens, and the job gets retried multiple times. Happened again to me this morning where one job filled up the /tmp and then multiple jobs started failing with unexpected errors, all actually caused by the /tmp fillup. Far too many copies of VeeamAgent and veam_soap files were left behind in /tmp.

I use a dedicated 1 or 2GB partition for /tmp or tmpfs --- about 1GB which is usually more than adequate except when this happens.

As its not easy for a program to cleanup itself after certain errors, not asking for technical support, but looking to hear what others do to avoid this.

[HannesK]

Hello,
normally that should not happen. Of course, you could just run a cron-job that cleans it up. But that does not solve the root course.

I know a customer that had a similar issue because the had core dumps from time to time. They are still investigating the reason (case 04096410), but I recommend to ask support why it happens in your situation.

If you open a case, could you please post the case number for reference?

Thanks,
Hannes

[DonZoomik]

I've experienced the same problem occasionally (maybe once a month) on a repository that is the target of a single backup copy job (~100 VMs, ~15TB of data). In that case, tmp is 4GB.
I haven't really looked into why it fills up, rm /tmp/Veeam* -rf cleans it up again.

[selva]

Will open case when it happens next time -- or if I can trigger one.

No core-dumps in my case -- its just /tmp filling up and the job doesn't end with "no space left on device" which is easily attended to, but with strange errors, multiple retries in a succession and no cleanup exasperating the problem.

A cronjob is not that straightforward as one has to isolate files of running jobs and any processes about to be started etc. One cant just do a rm -rf /tmp/Veam* /tmp/vee* . In normal operation I see several /tmp/VeeamAgent......./veeamagent jobs idling 24x7 on the Linux repo host (one per backup job?). The executables are different and with each at about 62MB, my 1GB /tmp is filled to ~60% by these "legitmate" processes. Add to that lots of logs retained by Veeam in /tmp.

So on a weekend when some additional jobs trigger, it doesn't take much to fill up the /tmp. Not sure exactly when that goes out of control leading to pile up of files, no cleanup and unhelpful errors in the console.

/tmp is not a good place for large files and logs. Is it possible to designate a different folder for executables transferred on demand to the host and for local logs?

[tsightler]

Yes, you can use LinAgentFolder registry key described here to change the path.

[selva]

Great to know. Thanks.

[selva]

LinAgentFolder works and env vars like $HOME is recognized, but its a bit erratic.

Based on scavenging logs, what I see is that when veeam uses scp to transfer files, env vars in the path work, but not when sftp is in use. I cant figure when veeam decides to use sftp instead of scp --- probably based on the susbsytem availability on the target host. Why not use scp always?

[Gostev]

Frankly, I'm surprised you see Veeam using SFTP to transfer the data mover files, because we don't have such functionality. Are you sure you're reading the logs correctly? I suggest you have our support take another look at them. Thanks!

[selva]

I'm not sure of the right terminology, but this is for transferring the agent to a Linux repository host -- something that VBR seems to do at the start of every job which normally works fine. Not the "data mover" you have in mind, I guess..

Sorry, asking for support is time consuming and I prefer to do that only for absolutely critical stuff -- especially for a community version setup where its even less attractive when response is not guaranteed. Not complaining at all -- my experience with Veeam has been great and most issues have been from my own misunderstanding. Help on the forum has been great as well.

Hopefully I'm not violating any rules by posting a snippet from the logs below to show where SFTP is used (see $HOME in the path and how it fails). I've another Linux repo where exactly the same thing happens via scp and works.

This is absolutely not critical and I can live without env vars in LinAgentFolder definition (its cools if it works), but still curious about this.

Code: Select all

[19.06.2020 12:55:00] <14> Info     [CSshRenciConnectionImpl] SFTP Renci connection '2cc1cca1-3d89-4eeb-9f9a-a1f3a08673a2' has been established. Session: [SSH Session; Local: 192.168.1.125:62425; Remote: 192.168.1.5:22]
[19.06.2020 12:55:00] <14> Error    Failed to upload file C:\Program Files\Veeam\Backup and Replication\Backup\VeeamAgent64 to $HOME/veeam-tmp/VeeamAgent4e4fa6f6-873b-4461-80c3-9165def3cff8
[19.06.2020 12:55:00] <14> Error    No such file (Renci.SshNet.Common.SftpPathNotFoundException)

[Gostev]

Yes, this is the data mover I was talking about.

Indeed, from this line it would appear as if SFTP is used within the SSH session, if we're reading it correctly.

But yes, it does actually violate the forum rules to post log snippets as explained there, the reason is that the debug log investigation is handled by our support engineers in collaboration with developers, and they will need the entire log package - not just a few lines you think might be relevant.

So if this becomes critical at some point, I recommend you follow the regular support procedure to get this sorted. It's not time consuming at all, you just open a case on the customer portal, and upload the debug log package there. May be there's some environment-specific logic or bug around this functionality that makes it switch to SFTP, and the registry key then causes it to be ignored in this case. In which case, we will want to understand what is causing the switch to SFTP for this specific machine. And even more importantly, why SFTP subsequently fails.

Thanks!

[tsightler]

I had communicated that variables worked because I had seen others report success with them and had seen it work in some cases but, after thinking about this, I have to admit that I probably didn't think through all of cases where we use this value. I can see how Veeam would not be able to use $HOME (or other variables) when doing things like uploading the initial files because the path is determined from the Veeam server, so it has no method to resolve $HOME.

Summary version is, variables will work when we are running code on the Linux host, because Linux will resolve them as part of normal path resolving steps, but, if the path variable is being used on the Veeam side (like for initial uploading of files) then they will not work. Based on this, I'm going to edit the post above to remove the statement that variables work properly so that others are mislead in the future.

[Gostev]

Update after discussing with the devs. Due to large variety of Linux servers we support, we have two SSH clients in the product (Granados and Renci). The first one uses SCP, and the second one uses SFTP for file transfer. The default SSH client is Granados, and if that SSH connection fails - then we failover to Renci.

[selva]

Thanks for the follow up. That explains it -- the newer server does not have "diffie-hellman-group1-sha1" for key exchange enabled, and Grandos seems not to be happy with that. I won't change the ssh server Kext's just to get env vars in the path working, but glad to know why the two repositories behaved so differently.

[Gostev]

Sounds good. We will probably make SCP the default transport in both, with failover to SFTP in both.