Veeam Agent for Linux: Performance and session summary

[Sebbo]

Hey,
I've just tested VAL on one of our dev servers, where are multiple websites in /var/www/ and the result is very mad in my opinion.

The session is running since ~45 minutes and no data were transfered yet - as the summary says:


But on the backup server, I can already see something:

Code: Select all

backup@p-v00283:/var/local/backup/veeam# ls -lh p-v00050/
total 31M
-rw-r--r-- 1 backup backup 31M Aug 11 16:25 p-v00050_2016-08-11T162522.vbk
backup@p-v00283:/var/local/backup/veeam#

Ok, that can be 0 % of the transferred data, but the summary should already display, that at least 31 MB were transferred, or not?

Secondly, the process is killing the servers CPU:

Code: Select all

root@p-v00050:~# uptime
 17:16:18 up 1 day,  7:10,  2 users,  load average: 5.11, 5.28, 5.36

The reason for that, is an error, which is being logged by syslog:

Aug 11 16:25:50 p-v00050 kernel: [109200.268563] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268565] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268566] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268568] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268570] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268572] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268573] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268575] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268577] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268578] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257

In the meanwhile, these lines were logged so often, that my disk space is now filled to 100 %.

[nielsengelen]

Could you present us the veeam logs from /var/log/veeam ?

Also could you give us the following information:
- Distro you are running
- Amount of CPU's, memory
- Kernel version (uname -a)
- Job configuration (I notice a file job but what is the target)?

As you are getting kernel allocation errors it looks like it's going into a deadlock.

[Sebbo]

Sure, sorry.

Here you go: http://46.16.75.141/veeam_logs.zip

I've also added a file called "syslog" to the ZIP file. It includes necessary lines regarding veeam.

- Ubuntu 16.04.1 LTS
- CPU Cores: 1
- Memory: 3 GB
- Linux p-v00050 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

And the job configuration:

Code: Select all

<?xml version="1.0" encoding="UTF-8"?><Config Version="1" /><Repositories><Repository Name="Repository_1" Type="Share" RepositoryOptions='<OnShareRepositoryOptions Version="1" Address="p-v00283/backup/veeam" FsType="cifs" Login="backup" Password="" Domain="" />' /></Repositories><Jobs><Job Name="p-v00050" creation_time="2016-08-11 14:24:04" RepoName="Repository_1"><JobOptions Compression="Lz4" BlockSize="KbBlockSize1024" PrefreezeCommand="" PostthawCommand="" MaxPoints="14" IsDedupEnabled="false" Schedule="true"><Schedule Hours="6" Minutes="0"><WeekMask Sunday="true" Monday="true" Tuesday="true" Wednesday="true" Thursday="true" Friday="true" Saturday="true" /></Schedule></JobOptions><ObjectsForBackup><Object ObjectType="Directory" RecordType="Include" Value="/var" /></ObjectsForBackup></Job></Jobs><VbrServers />

[PTide]

Hi,

Please also provide lsblk -af and dmesg output.

Thanks

[Sebbo]

Since Ubuntu 16.04 LTS, dmesg is always emtpy, due of syslog is used instead. This log file was added to the ZIP file.

Unfortunately, I'm not in the office anymore, why I don't have access to the server anymore. I'll provide the information of the command tomorrow.

[Sebbo]

Here you go:

Code: Select all

root@p-v00050:~# lsblk -af
NAME   FSTYPE LABEL UUID                                 MOUNTPOINT
vda
|-vda1 ext4         a0ddaaea-c23c-4a6d-afb7-0dabe7bb893d /
`-vda2 swap         9b7584ec-e8a2-4df0-bf9d-a0e81e300a1b [SWAP]
vdb
`-vdb1 xfs          81e9a936-53e2-4479-a1f3-2368a642d29a /var
loop0
loop1
loop2
loop3
loop4
loop5
loop6
loop7

[nielsengelen]

Is this a KVM VM ? I notice you have vd disks which means virtio block device and thus is para-virtualization. Might be related as this doesn't 100% offer the device to perform the correct steps to back it up?

[Sebbo]

I'm not sure, what exactly it is. Yes, it's a VM, but I don't know, if it's VMware, Proxmox, Xen or anything else.

It's hosted by Profitbricks: https://www.profitbricks.com/

The storage is connected via bustype "Virt IO". The other available option would be "IDE".

I only can say you, that we're able to fully manage the operating system. We're able to install own kernels, we can use iptables and so on. There aren't any limitations.

[nielsengelen]

According to their website it is KVM based. They use the para-virtualization part (nothing wrong with it) which gives you a full OS however the disks are in a way linked to the host for several reasons. We'll have to look into this if this is currently working as it should in the beta.

[PTide]

One more question - did you kill the service manually or just perfromed VM hard-reset?

[Sebbo]

I've stopped the job and service, what took around 5-10 minutes per command.

Due of that, that it didn't stopped all Veeam processes and the CPU load was still very high (>5.00 last 5 minutes), because syslog wrote hundred of further above mentioned messages to the log file, I had to reboot the server.

[PTide]

How much free RAM did you have before starting the job? Please run slabtop -o prior to starting the job and during the job and send us the results. Also please try to lower the veeam processes' priorities by using renice.

Thank you

[Sebbo]

Sorry for the delayed response - I'm very busy this week. I had around 3.5 GB free RAM before.