VEEAM Linux agent backup start freezes server

[CatSpirent]

Just wondering if there is a link to best practices for configuring the VEEAM Linux Agent to use reduced resources.
The backup has been running fine for a month but then 3 days ago would get so far in the setup and then just hang the entire server.
The developers are facing the consequences of installing a new application on this server that is not appropriate for the server's limited resources.
In the meantime, I haven't gotten a backup in 3 days.
I am hoping there are some configuration adjustments I might be able to make to the agent to reduce its load on the server and still get a backup while the developers find an alternative server for their new application.

[CatSpirent]

I have an update after running some statistics. I still am going through the /var/log/messages file line by line and will find the VEEAM log to go through it.

The top and the dstat that I was running still continued to run till the server was hard booted.
The applications showed as still running in the top output which of course stopped running when the server was hardbooted.

The freeze as I called it seemed to have affected some OS processes.
The VEEAM backup started at 9:30 am. The messages file is filled with mostly VEEAM messages and stops getting updated after these few lines.

Sep 13 21:57:33 spcfrksasol01 kernel: veeamsnap:blk_dev_utile.c:blk_thaw_bdev thaw_bdev for device=253:0
Sep 13 21:57:33 spcfrksasol01 kernel: veeamsnap:tracker.c:_tracker_freeze dev_id=253:2
Sep 13 21:57:33 spcfrksasol01 kernel: veeamsnap:defer_io.c:defer_io_work_thread started.

Yesterday morning after the previous nights 'freeze' I had left a terminal window up.
The command I ran was a tail command, not entirely sure which file but my guess would be it was probably /var/log/messages, which then froze. I couldn't ctrl-C or ctrl-Z or get out of it at all.
The terminal did echo back my command so those OS whatever did work.

For last night's 'freeze' as I mentioned the top, dstat, and other developer applications appeared to remain running.
I am assuming the syslog daemon died or froze since messages stopped getting updated.
The main applications are Jira, Confluence, and docker. Would any of them have conflicts with VEEAM Linux agent?

The dstat showd almost 100% I/O wait when VEEAM started up though it eventually settled down to about 50% I/O wait. Memory usage seemed reasonable.
Disk I/O increased significantly, especially reads, and network I/O increased significantly both are not unexpected.

So now I am curious if the problem is really a resource issue or if there was a conflict elsewhere I haven't found.

[CatSpirent]

So we are moving the recently added app to a new VM.

Hopefully that will resolve any issues. I still would be interested in any best practices for performance if there are any or if any do get created down the road for the Linux agent.

[PTide]

Hi,

am hoping there are some configuration adjustments I might be able to make to the agent to reduce its load on the server and still get a backup while the developers find an alternative server for their new application.

Yes, you can adjust load settings via parameters in /etc/veeam/veeam.ini

Code: Select all

# IO rate limit, from 0.01 to 1.0
# ioRateLimit= 0.9

# CPU priority for veeamagents, from 0 to 19
# priority= 10

The main applications are Jira, Confluence, and docker. Would any of them have conflicts with VEEAM Linux agent?

None of these should cause problems, however if the application introduces some heavy IO on the system there might be issues. Do you use the Agent to backup docker containers?

Thanks

[CatSpirent]

I have it back up everything for a bare metal restore capability, so I would guess the answer to that is yes.

[PTide]

Hi,

Have you managed to tweak the parameters that I've mentioned, or you just moved the app to another server?

The reason why I'm asked about containers is that VAL hasn't been fully tested with containers, so I'd like to know if you ever tried to restore any containers already.

Thanks

[CatSpirent]

I did tweak the settings, but did not try to start the backup to see if it makes a difference.
The engineer started the move effort, I do not know if he completed it.

I haven't tried to restore anything at this point.

[CatSpirent]

Just a quick update. Engineer moved the Jira/confluence etc application off of the server to a new VM.

He has asked me to hold off on starting the backup schedule / test till next week as he has some important tasks scheduled that he doesn't want to risk any problems.
I will update once I get a chance to run the backup again.

[CatSpirent]

I have an update. I finally was able to start the backup last night which the cron was set for 9:00 pm.
I received the email that the backup started, but no email for completion.
This morning I had a terminal still up so system was available.
I initially tried to start a new terminal session since I had forgotten about the one I had up. I received the login prompt and password prompt, but it hung after I entered the password.

I was able to run a ps command and grep for veeam. There were numerous VEEAM processes displayed.
I ran a systemctl command just to see if there were any obvious failed processes.
The only obvious red one was mongod.service which I ran a systemctl status mongod.service which showed it had failed several days ago after which the command prompt never returned and I was unable to <ctrl> c out of it.

Not sure if anything in the systemctl would be helpful. The SUB column shows some waiting, exits and such services, but those may be normal.

I will look at reducing those parameters you mentioned some more, but wonder if any suggestions for ideas to focus on? I can see if anything else would be helpful with the dstat command. I am not very familiar with sar if anyone has suggestions for something to run while attempting the backup again. The alt-sysrq keys didn't seem to have an effect nor the F# keys to get a text console.

[PTide]

I suggest you to contact our support team directly so they can confirm if it's Docker to blame or there is another reason, please post your case ID once you open a case.

Thanks

[CatSpirent]

I created case 02341814. Will see how it goes.

[CatSpirent]

Just updating this with a conclusion.

After working with support they believed that the issue was I/O contention so since I had the disk space, I created a large file system large enough to hold the server's data on a different disk.
They also had me update the /etc/veeam/veeam.ini file by uncommenting a few items and making changes to their default values. The default file has all items commented out.
I also scheduled the backup to run at a time when there would be the least amount of activity.

ioRateLimit= 0.5

priority= 8

location=/veeamsnap

maxSize= 182536110080

minSize= 536870912

[PTide]

Thank you for the update!