VEEAM Linux agent backup start freezes server

Backup agent for Linux servers and workstations on-premises or in the public cloud

VEEAM Linux agent backup start freezes server

Veeam Logoby CatSpirent » Wed Sep 13, 2017 6:40 pm

Just wondering if there is a link to best practices for configuring the VEEAM Linux Agent to use reduced resources.
The backup has been running fine for a month but then 3 days ago would get so far in the setup and then just hang the entire server.
The developers are facing the consequences of installing a new application on this server that is not appropriate for the server's limited resources.
In the meantime, I haven't gotten a backup in 3 days.
I am hoping there are some configuration adjustments I might be able to make to the agent to reduce its load on the server and still get a backup while the developers find an alternative server for their new application.
CatSpirent
Influencer
 
Posts: 19
Liked: 2 times
Joined: Fri Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer

Re: VEEAM Linux agent backup start freezes server

Veeam Logoby CatSpirent » Thu Sep 14, 2017 2:29 pm

I have an update after running some statistics. I still am going through the /var/log/messages file line by line and will find the VEEAM log to go through it.

The top and the dstat that I was running still continued to run till the server was hard booted.
The applications showed as still running in the top output which of course stopped running when the server was hardbooted.

The freeze as I called it seemed to have affected some OS processes.
The VEEAM backup started at 9:30 am. The messages file is filled with mostly VEEAM messages and stops getting updated after these few lines.

Sep 13 21:57:33 spcfrksasol01 kernel: veeamsnap:blk_dev_utile.c:blk_thaw_bdev thaw_bdev for device=253:0
Sep 13 21:57:33 spcfrksasol01 kernel: veeamsnap:tracker.c:_tracker_freeze dev_id=253:2
Sep 13 21:57:33 spcfrksasol01 kernel: veeamsnap:defer_io.c:defer_io_work_thread started.

Yesterday morning after the previous nights 'freeze' I had left a terminal window up.
The command I ran was a tail command, not entirely sure which file but my guess would be it was probably /var/log/messages, which then froze. I couldn't ctrl-C or ctrl-Z or get out of it at all.
The terminal did echo back my command so those OS whatever did work.

For last night's 'freeze' as I mentioned the top, dstat, and other developer applications appeared to remain running.
I am assuming the syslog daemon died or froze since messages stopped getting updated.
The main applications are Jira, Confluence, and docker. Would any of them have conflicts with VEEAM Linux agent?

The dstat showd almost 100% I/O wait when VEEAM started up though it eventually settled down to about 50% I/O wait. Memory usage seemed reasonable.
Disk I/O increased significantly, especially reads, and network I/O increased significantly both are not unexpected.

So now I am curious if the problem is really a resource issue or if there was a conflict elsewhere I haven't found.
CatSpirent
Influencer
 
Posts: 19
Liked: 2 times
Joined: Fri Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer

Re: VEEAM Linux agent backup start freezes server

Veeam Logoby CatSpirent » Thu Sep 14, 2017 8:37 pm

So we are moving the recently added app to a new VM.

Hopefully that will resolve any issues. I still would be interested in any best practices for performance if there are any or if any do get created down the road for the Linux agent.
CatSpirent
Influencer
 
Posts: 19
Liked: 2 times
Joined: Fri Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer

Re: VEEAM Linux agent backup start freezes server

Veeam Logoby PTide » Mon Sep 18, 2017 10:47 am

Hi,

am hoping there are some configuration adjustments I might be able to make to the agent to reduce its load on the server and still get a backup while the developers find an alternative server for their new application.
Yes, you can adjust load settings via parameters in /etc/veeam/veeam.ini

Code: Select all
# IO rate limit, from 0.01 to 1.0
# ioRateLimit= 0.9

# CPU priority for veeamagents, from 0 to 19
# priority= 10


The main applications are Jira, Confluence, and docker. Would any of them have conflicts with VEEAM Linux agent?
None of these should cause problems, however if the application introduces some heavy IO on the system there might be issues. Do you use the Agent to backup docker containers?

Thanks
PTide
Veeam Software
 
Posts: 3147
Liked: 262 times
Joined: Tue May 19, 2015 1:46 pm

Re: VEEAM Linux agent backup start freezes server

Veeam Logoby CatSpirent » Mon Sep 18, 2017 8:36 pm

I have it back up everything for a bare metal restore capability, so I would guess the answer to that is yes.
CatSpirent
Influencer
 
Posts: 19
Liked: 2 times
Joined: Fri Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer

Re: VEEAM Linux agent backup start freezes server

Veeam Logoby PTide » Sun Sep 24, 2017 5:55 pm

Hi,

Have you managed to tweak the parameters that I've mentioned, or you just moved the app to another server?

The reason why I'm asked about containers is that VAL hasn't been fully tested with containers, so I'd like to know if you ever tried to restore any containers already.

Thanks
PTide
Veeam Software
 
Posts: 3147
Liked: 262 times
Joined: Tue May 19, 2015 1:46 pm

Re: VEEAM Linux agent backup start freezes server

Veeam Logoby CatSpirent » Mon Sep 25, 2017 2:35 pm

I did tweak the settings, but did not try to start the backup to see if it makes a difference.
The engineer started the move effort, I do not know if he completed it.

I haven't tried to restore anything at this point.
CatSpirent
Influencer
 
Posts: 19
Liked: 2 times
Joined: Fri Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer

Re: VEEAM Linux agent backup start freezes server

Veeam Logoby CatSpirent » Thu Sep 28, 2017 6:55 pm 1 person likes this post

Just a quick update. Engineer moved the Jira/confluence etc application off of the server to a new VM.

He has asked me to hold off on starting the backup schedule / test till next week as he has some important tasks scheduled that he doesn't want to risk any problems.
I will update once I get a chance to run the backup again.
CatSpirent
Influencer
 
Posts: 19
Liked: 2 times
Joined: Fri Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer

Re: VEEAM Linux agent backup start freezes server

Veeam Logoby CatSpirent » Tue Oct 10, 2017 3:42 pm

I have an update. I finally was able to start the backup last night which the cron was set for 9:00 pm.
I received the email that the backup started, but no email for completion.
This morning I had a terminal still up so system was available.
I initially tried to start a new terminal session since I had forgotten about the one I had up. I received the login prompt and password prompt, but it hung after I entered the password.

I was able to run a ps command and grep for veeam. There were numerous VEEAM processes displayed.
I ran a systemctl command just to see if there were any obvious failed processes.
The only obvious red one was mongod.service which I ran a systemctl status mongod.service which showed it had failed several days ago after which the command prompt never returned and I was unable to <ctrl> c out of it.

Not sure if anything in the systemctl would be helpful. The SUB column shows some waiting, exits and such services, but those may be normal.

I will look at reducing those parameters you mentioned some more, but wonder if any suggestions for ideas to focus on? I can see if anything else would be helpful with the dstat command. I am not very familiar with sar if anyone has suggestions for something to run while attempting the backup again. The alt-sysrq keys didn't seem to have an effect nor the F# keys to get a text console.
CatSpirent
Influencer
 
Posts: 19
Liked: 2 times
Joined: Fri Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer

Re: VEEAM Linux agent backup start freezes server

Veeam Logoby PTide » Tue Oct 10, 2017 5:12 pm

I suggest you to contact our support team directly so they can confirm if it's Docker to blame or there is another reason, please post your case ID once you open a case.

Thanks
PTide
Veeam Software
 
Posts: 3147
Liked: 262 times
Joined: Tue May 19, 2015 1:46 pm

Re: VEEAM Linux agent backup start freezes server

Veeam Logoby CatSpirent » Wed Oct 11, 2017 1:46 pm

I created case 02341814. Will see how it goes.
CatSpirent
Influencer
 
Posts: 19
Liked: 2 times
Joined: Fri Dec 30, 2016 4:10 pm
Full Name: Caterine Kieffer


Return to Veeam Agent for Linux



Who is online

Users browsing this forum: No registered users and 2 guests