Veeam Agent for Linux: Performance and session summary

Backup agent for Linux servers and workstations on-premises or in the public cloud

Veeam Agent for Linux: Performance and session summary

Veeam Logoby Sebbo » Thu Aug 11, 2016 3:22 pm

Hey,
I've just tested VAL on one of our dev servers, where are multiple websites in /var/www/ and the result is very mad in my opinion. :(

The session is running since ~45 minutes and no data were transfered yet - as the summary says:
Image

But on the backup server, I can already see something:
Code: Select all
backup@p-v00283:/var/local/backup/veeam# ls -lh p-v00050/
total 31M
-rw-r--r-- 1 backup backup 31M Aug 11 16:25 p-v00050_2016-08-11T162522.vbk
backup@p-v00283:/var/local/backup/veeam#

Ok, that can be 0 % of the transferred data, but the summary should already display, that at least 31 MB were transferred, or not?

Secondly, the process is killing the servers CPU:
Code: Select all
root@p-v00050:~# uptime
 17:16:18 up 1 day,  7:10,  2 users,  load average: 5.11, 5.28, 5.36


The reason for that, is an error, which is being logged by syslog:
Aug 11 16:25:50 p-v00050 kernel: [109200.268563] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268565] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268566] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268568] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268570] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268572] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268573] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268575] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268577] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257
Aug 11 16:25:50 p-v00050 kernel: [109200.268578] ERR veeamsnap:blk_dev_utile.c:__dev_direct_submit_pages bio_alloc failed. Schedule. nr_iovecs=257


In the meanwhile, these lines were logged so often, that my disk space is now filled to 100 %. :(
Sebbo
Influencer
 
Posts: 16
Liked: never
Joined: Thu Aug 11, 2016 12:08 pm

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby vmniels » Thu Aug 11, 2016 3:33 pm

Could you present us the veeam logs from /var/log/veeam ?

Also could you give us the following information:
- Distro you are running
- Amount of CPU's, memory
- Kernel version (uname -a)
- Job configuration (I notice a file job but what is the target)?

As you are getting kernel allocation errors it looks like it's going into a deadlock.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be
vmniels
Veeam Software
 
Posts: 1633
Liked: 362 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby Sebbo » Thu Aug 11, 2016 3:52 pm

Sure, sorry.

Here you go: http://46.16.75.141/veeam_logs.zip

I've also added a file called "syslog" to the ZIP file. It includes necessary lines regarding veeam.

- Ubuntu 16.04.1 LTS
- CPU Cores: 1
- Memory: 3 GB
- Linux p-v00050 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

And the job configuration:
Code: Select all
<?xml version="1.0" encoding="UTF-8"?><Config Version="1" /><Repositories><Repository Name="Repository_1" Type="Share" RepositoryOptions='&lt;OnShareRepositoryOptions Version=&quot;1&quot; Address=&quot;p-v00283/backup/veeam&quot; FsType=&quot;cifs&quot; Login=&quot;backup&quot; Password=&quot;&quot; Domain=&quot;&quot; /&gt;' /></Repositories><Jobs><Job Name="p-v00050" creation_time="2016-08-11 14:24:04" RepoName="Repository_1"><JobOptions Compression="Lz4" BlockSize="KbBlockSize1024" PrefreezeCommand="" PostthawCommand="" MaxPoints="14" IsDedupEnabled="false" Schedule="true"><Schedule Hours="6" Minutes="0"><WeekMask Sunday="true" Monday="true" Tuesday="true" Wednesday="true" Thursday="true" Friday="true" Saturday="true" /></Schedule></JobOptions><ObjectsForBackup><Object ObjectType="Directory" RecordType="Include" Value="/var" /></ObjectsForBackup></Job></Jobs><VbrServers />
Sebbo
Influencer
 
Posts: 16
Liked: never
Joined: Thu Aug 11, 2016 12:08 pm

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby PTide » Thu Aug 11, 2016 4:51 pm

Hi,

Please also provide lsblk -af and dmesg output.

Thanks
PTide
Veeam Software
 
Posts: 3252
Liked: 273 times
Joined: Tue May 19, 2015 1:46 pm

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby Sebbo » Thu Aug 11, 2016 6:15 pm

Since Ubuntu 16.04 LTS, dmesg is always emtpy, due of syslog is used instead. This log file was added to the ZIP file.

Unfortunately, I'm not in the office anymore, why I don't have access to the server anymore. I'll provide the information of the command tomorrow.
Sebbo
Influencer
 
Posts: 16
Liked: never
Joined: Thu Aug 11, 2016 12:08 pm

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby Sebbo » Fri Aug 12, 2016 7:43 am

Here you go:
Code: Select all
root@p-v00050:~# lsblk -af
NAME   FSTYPE LABEL UUID                                 MOUNTPOINT
vda
|-vda1 ext4         a0ddaaea-c23c-4a6d-afb7-0dabe7bb893d /
`-vda2 swap         9b7584ec-e8a2-4df0-bf9d-a0e81e300a1b [SWAP]
vdb
`-vdb1 xfs          81e9a936-53e2-4479-a1f3-2368a642d29a /var
loop0
loop1
loop2
loop3
loop4
loop5
loop6
loop7
Sebbo
Influencer
 
Posts: 16
Liked: never
Joined: Thu Aug 11, 2016 12:08 pm

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby vmniels » Fri Aug 12, 2016 9:09 am

Is this a KVM VM ? I notice you have vd disks which means virtio block device and thus is para-virtualization. Might be related as this doesn't 100% offer the device to perform the correct steps to back it up?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be
vmniels
Veeam Software
 
Posts: 1633
Liked: 362 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby Sebbo » Fri Aug 12, 2016 11:26 am

I'm not sure, what exactly it is. Yes, it's a VM, but I don't know, if it's VMware, Proxmox, Xen or anything else.

It's hosted by Profitbricks: https://www.profitbricks.com/

The storage is connected via bustype "Virt IO". The other available option would be "IDE".

I only can say you, that we're able to fully manage the operating system. We're able to install own kernels, we can use iptables and so on. There aren't any limitations.
Sebbo
Influencer
 
Posts: 16
Liked: never
Joined: Thu Aug 11, 2016 12:08 pm

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby vmniels » Fri Aug 12, 2016 11:31 am

According to their website it is KVM based. They use the para-virtualization part (nothing wrong with it) which gives you a full OS however the disks are in a way linked to the host for several reasons. We'll have to look into this if this is currently working as it should in the beta.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be
vmniels
Veeam Software
 
Posts: 1633
Liked: 362 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby PTide » Fri Aug 12, 2016 2:21 pm

One more question - did you kill the service manually or just perfromed VM hard-reset?
PTide
Veeam Software
 
Posts: 3252
Liked: 273 times
Joined: Tue May 19, 2015 1:46 pm

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby Sebbo » Fri Aug 12, 2016 6:23 pm

I've stopped the job and service, what took around 5-10 minutes per command.

Due of that, that it didn't stopped all Veeam processes and the CPU load was still very high (>5.00 last 5 minutes), because syslog wrote hundred of further above mentioned messages to the log file, I had to reboot the server.
Sebbo
Influencer
 
Posts: 16
Liked: never
Joined: Thu Aug 11, 2016 12:08 pm

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby PTide » Mon Aug 15, 2016 6:57 pm

How much free RAM did you have before starting the job? Please run slabtop -o prior to starting the job and during the job and send us the results. Also please try to lower the veeam processes' priorities by using renice.

Thank you
PTide
Veeam Software
 
Posts: 3252
Liked: 273 times
Joined: Tue May 19, 2015 1:46 pm

Re: Veeam Agent for Linux: Performance and session summary

Veeam Logoby Sebbo » Tue Aug 16, 2016 4:53 pm

Sorry for the delayed response - I'm very busy this week. I had around 3.5 GB free RAM before.
Sebbo
Influencer
 
Posts: 16
Liked: never
Joined: Thu Aug 11, 2016 12:08 pm


Return to Veeam Agent for Linux



Who is online

Users browsing this forum: No registered users and 1 guest