Standalone backup agents for Linux, Mac, AIX & Solaris workloads on-premises or in the public cloud
Post Reply
Sebbo
Influencer
Posts: 16
Liked: never
Joined: Aug 11, 2016 12:08 pm
Contact:

Veeam Agent for Linux: Performance and session summary

Post by Sebbo »

Hey,
I've just tested VAL on one of our dev servers, where are multiple websites in /var/www/ and the result is very mad in my opinion. :(

The session is running since ~45 minutes and no data were transfered yet - as the summary says:
Image

But on the backup server, I can already see something:

Code: Select all

backup@p-v00283:/var/local/backup/veeam# ls -lh p-v00050/
total 31M
-rw-r--r-- 1 backup backup 31M Aug 11 16:25 p-v00050_2016-08-11T162522.vbk
backup@p-v00283:/var/local/backup/veeam#
Ok, that can be 0 % of the transferred data, but the summary should already display, that at least 31 MB were transferred, or not?

Secondly, the process is killing the servers CPU:

Code: Select all

root@p-v00050:~# uptime
 17:16:18 up 1 day,  7:10,  2 users,  load average: 5.11, 5.28, 5.36
The reason for that, is an error, which is being logged by syslog:
Aug 11 16:25:50 p-v00050 kernel: [109200.268563] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268565] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268566] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268568] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268570] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268572] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268573] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268575] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268577] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268578] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
In the meanwhile, these lines were logged so often, that my disk space is now filled to 100 %. :(
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by nielsengelen »

Could you present us the veeam logs from /var/log/veeam ?

Also could you give us the following information:
- Distro you are running
- Amount of CPU's, memory
- Kernel version (uname -a)
- Job configuration (I notice a file job but what is the target)?

As you are getting kernel allocation errors it looks like it's going into a deadlock.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
Sebbo
Influencer
Posts: 16
Liked: never
Joined: Aug 11, 2016 12:08 pm
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by Sebbo »

Sure, sorry.

Here you go: http://46.16.75.141/veeam_logs.zip

I've also added a file called "syslog" to the ZIP file. It includes necessary lines regarding veeam.

- Ubuntu 16.04.1 LTS
- CPU Cores: 1
- Memory: 3 GB
- Linux p-v00050 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

And the job configuration:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?><Config Version="1" /><Repositories><Repository Name="Repository_1" Type="Share" RepositoryOptions='<OnShareRepositoryOptions Version="1" Address="p-v00283/backup/veeam" FsType="cifs" Login="backup" Password="" Domain="" />' /></Repositories><Jobs><Job Name="p-v00050" creation_time="2016-08-11 14:24:04" RepoName="Repository_1"><JobOptions Compression="Lz4" BlockSize="KbBlockSize1024" PrefreezeCommand="" PostthawCommand="" MaxPoints="14" IsDedupEnabled="false" Schedule="true"><Schedule Hours="6" Minutes="0"><WeekMask Sunday="true" Monday="true" Tuesday="true" Wednesday="true" Thursday="true" Friday="true" Saturday="true" /></Schedule></JobOptions><ObjectsForBackup><Object ObjectType="Directory" RecordType="Include" Value="/var" /></ObjectsForBackup></Job></Jobs><VbrServers />
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by PTide »

Hi,

Please also provide lsblk -af and dmesg output.

Thanks
Sebbo
Influencer
Posts: 16
Liked: never
Joined: Aug 11, 2016 12:08 pm
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by Sebbo »

Since Ubuntu 16.04 LTS, dmesg is always emtpy, due of syslog is used instead. This log file was added to the ZIP file.

Unfortunately, I'm not in the office anymore, why I don't have access to the server anymore. I'll provide the information of the command tomorrow.
Sebbo
Influencer
Posts: 16
Liked: never
Joined: Aug 11, 2016 12:08 pm
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by Sebbo »

Here you go:

Code: Select all

root@p-v00050:~# lsblk -af
NAME   FSTYPE LABEL UUID                                 MOUNTPOINT
vda
|-vda1 ext4         a0ddaaea-c23c-4a6d-afb7-0dabe7bb893d /
`-vda2 swap         9b7584ec-e8a2-4df0-bf9d-a0e81e300a1b [SWAP]
vdb
`-vdb1 xfs          81e9a936-53e2-4479-a1f3-2368a642d29a /var
loop0
loop1
loop2
loop3
loop4
loop5
loop6
loop7
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by nielsengelen »

Is this a KVM VM ? I notice you have vd disks which means virtio block device and thus is para-virtualization. Might be related as this doesn't 100% offer the device to perform the correct steps to back it up?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
Sebbo
Influencer
Posts: 16
Liked: never
Joined: Aug 11, 2016 12:08 pm
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by Sebbo »

I'm not sure, what exactly it is. Yes, it's a VM, but I don't know, if it's VMware, Proxmox, Xen or anything else.

It's hosted by Profitbricks: https://www.profitbricks.com/

The storage is connected via bustype "Virt IO". The other available option would be "IDE".

I only can say you, that we're able to fully manage the operating system. We're able to install own kernels, we can use iptables and so on. There aren't any limitations.
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by nielsengelen »

According to their website it is KVM based. They use the para-virtualization part (nothing wrong with it) which gives you a full OS however the disks are in a way linked to the host for several reasons. We'll have to look into this if this is currently working as it should in the beta.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by PTide »

One more question - did you kill the service manually or just perfromed VM hard-reset?
Sebbo
Influencer
Posts: 16
Liked: never
Joined: Aug 11, 2016 12:08 pm
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by Sebbo »

I've stopped the job and service, what took around 5-10 minutes per command.

Due of that, that it didn't stopped all Veeam processes and the CPU load was still very high (>5.00 last 5 minutes), because syslog wrote hundred of further above mentioned messages to the log file, I had to reboot the server.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by PTide »

How much free RAM did you have before starting the job? Please run slabtop -o prior to starting the job and during the job and send us the results. Also please try to lower the veeam processes' priorities by using renice.

Thank you
Sebbo
Influencer
Posts: 16
Liked: never
Joined: Aug 11, 2016 12:08 pm
Contact:

Re: Veeam Agent for Linux: Performance and session summary

Post by Sebbo »

Sorry for the delayed response - I'm very busy this week. I had around 3.5 GB free RAM before.
Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests