Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
Post Reply
CatSpirent
Enthusiast
Posts: 53
Liked: 3 times
Joined: Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer
Contact:

VEEAM Linux agent backup start freezes server

Post by CatSpirent »

Just wondering if there is a link to best practices for configuring the VEEAM Linux Agent to use reduced resources.
The backup has been running fine for a month but then 3 days ago would get so far in the setup and then just hang the entire server.
The developers are facing the consequences of installing a new application on this server that is not appropriate for the server's limited resources.
In the meantime, I haven't gotten a backup in 3 days.
I am hoping there are some configuration adjustments I might be able to make to the agent to reduce its load on the server and still get a backup while the developers find an alternative server for their new application.
CatSpirent
Enthusiast
Posts: 53
Liked: 3 times
Joined: Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by CatSpirent »

I have an update after running some statistics. I still am going through the /var/log/messages file line by line and will find the VEEAM log to go through it.

The top and the dstat that I was running still continued to run till the server was hard booted.
The applications showed as still running in the top output which of course stopped running when the server was hardbooted.

The freeze as I called it seemed to have affected some OS processes.
The VEEAM backup started at 9:30 am. The messages file is filled with mostly VEEAM messages and stops getting updated after these few lines.

Sep 13 21:57:33 spcfrksasol01 kernel: veeamsnap:blk_dev_utile.c:blk_thaw_bdev thaw_bdev for device=253:0
Sep 13 21:57:33 spcfrksasol01 kernel: veeamsnap:tracker.c:_tracker_freeze dev_id=253:2
Sep 13 21:57:33 spcfrksasol01 kernel: veeamsnap:defer_io.c:defer_io_work_thread started.

Yesterday morning after the previous nights 'freeze' I had left a terminal window up.
The command I ran was a tail command, not entirely sure which file but my guess would be it was probably /var/log/messages, which then froze. I couldn't ctrl-C or ctrl-Z or get out of it at all.
The terminal did echo back my command so those OS whatever did work.

For last night's 'freeze' as I mentioned the top, dstat, and other developer applications appeared to remain running.
I am assuming the syslog daemon died or froze since messages stopped getting updated.
The main applications are Jira, Confluence, and docker. Would any of them have conflicts with VEEAM Linux agent?

The dstat showd almost 100% I/O wait when VEEAM started up though it eventually settled down to about 50% I/O wait. Memory usage seemed reasonable.
Disk I/O increased significantly, especially reads, and network I/O increased significantly both are not unexpected.

So now I am curious if the problem is really a resource issue or if there was a conflict elsewhere I haven't found.
CatSpirent
Enthusiast
Posts: 53
Liked: 3 times
Joined: Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by CatSpirent »

So we are moving the recently added app to a new VM.

Hopefully that will resolve any issues. I still would be interested in any best practices for performance if there are any or if any do get created down the road for the Linux agent.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by PTide »

Hi,
am hoping there are some configuration adjustments I might be able to make to the agent to reduce its load on the server and still get a backup while the developers find an alternative server for their new application.
Yes, you can adjust load settings via parameters in /etc/veeam/veeam.ini

Code: Select all

# IO rate limit, from 0.01 to 1.0
# ioRateLimit= 0.9

# CPU priority for veeamagents, from 0 to 19
# priority= 10
The main applications are Jira, Confluence, and docker. Would any of them have conflicts with VEEAM Linux agent?
None of these should cause problems, however if the application introduces some heavy IO on the system there might be issues. Do you use the Agent to backup docker containers?

Thanks
CatSpirent
Enthusiast
Posts: 53
Liked: 3 times
Joined: Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by CatSpirent »

I have it back up everything for a bare metal restore capability, so I would guess the answer to that is yes.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by PTide »

Hi,

Have you managed to tweak the parameters that I've mentioned, or you just moved the app to another server?

The reason why I'm asked about containers is that VAL hasn't been fully tested with containers, so I'd like to know if you ever tried to restore any containers already.

Thanks
CatSpirent
Enthusiast
Posts: 53
Liked: 3 times
Joined: Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by CatSpirent »

I did tweak the settings, but did not try to start the backup to see if it makes a difference.
The engineer started the move effort, I do not know if he completed it.

I haven't tried to restore anything at this point.
CatSpirent
Enthusiast
Posts: 53
Liked: 3 times
Joined: Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by CatSpirent » 1 person likes this post

Just a quick update. Engineer moved the Jira/confluence etc application off of the server to a new VM.

He has asked me to hold off on starting the backup schedule / test till next week as he has some important tasks scheduled that he doesn't want to risk any problems.
I will update once I get a chance to run the backup again.
CatSpirent
Enthusiast
Posts: 53
Liked: 3 times
Joined: Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by CatSpirent »

I have an update. I finally was able to start the backup last night which the cron was set for 9:00 pm.
I received the email that the backup started, but no email for completion.
This morning I had a terminal still up so system was available.
I initially tried to start a new terminal session since I had forgotten about the one I had up. I received the login prompt and password prompt, but it hung after I entered the password.

I was able to run a ps command and grep for veeam. There were numerous VEEAM processes displayed.
I ran a systemctl command just to see if there were any obvious failed processes.
The only obvious red one was mongod.service which I ran a systemctl status mongod.service which showed it had failed several days ago after which the command prompt never returned and I was unable to <ctrl> c out of it.

Not sure if anything in the systemctl would be helpful. The SUB column shows some waiting, exits and such services, but those may be normal.

I will look at reducing those parameters you mentioned some more, but wonder if any suggestions for ideas to focus on? I can see if anything else would be helpful with the dstat command. I am not very familiar with sar if anyone has suggestions for something to run while attempting the backup again. The alt-sysrq keys didn't seem to have an effect nor the F# keys to get a text console.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by PTide »

I suggest you to contact our support team directly so they can confirm if it's Docker to blame or there is another reason, please post your case ID once you open a case.

Thanks
CatSpirent
Enthusiast
Posts: 53
Liked: 3 times
Joined: Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by CatSpirent »

I created case 02341814. Will see how it goes.
CatSpirent
Enthusiast
Posts: 53
Liked: 3 times
Joined: Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by CatSpirent »

Just updating this with a conclusion.

After working with support they believed that the issue was I/O contention so since I had the disk space, I created a large file system large enough to hold the server's data on a different disk.
They also had me update the /etc/veeam/veeam.ini file by uncommenting a few items and making changes to their default values. The default file has all items commented out.
I also scheduled the backup to run at a time when there would be the least amount of activity.

ioRateLimit= 0.5

priority= 8

location=/veeamsnap

maxSize= 182536110080

minSize= 536870912
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: VEEAM Linux agent backup start freezes server

Post by PTide »

Thank you for the update!
Post Reply

Who is online

Users browsing this forum: No registered users and 14 guests