Terribly, terribly slow!!!

Backup agent for Linux servers and workstations on-premises or in the public cloud

Terribly, terribly slow!!!

Veeam Logoby pama » Fri Jul 15, 2016 1:30 pm

I have tried several times, but I am finding Veeam for Linux terribly slow.

I have never finished a backup.
I use CIFS folder, used before with scp file transfer from the same server without any problem. I attach here my backup HDD benchmark (I Know, it is not faster than other, but it is for backup).
Image

Each time I run the backup I have different bottlecages: one time proxy, one time destination, one time source..... so for me it is impossible to understand.
Here I attach also veeam logs, syslog and kernel dump. link

Thank you for your investigation
Andrea
pama
Enthusiast
 
Posts: 40
Liked: 1 time
Joined: Wed Apr 20, 2011 1:02 pm
Full Name: Andrea de Lutti

Re: Terribly, terribly slow!!!

Veeam Logoby PTide » Fri Jul 15, 2016 4:07 pm

Did I get it right - the very same job shows different bottlenecks in stats? Where is CIFS share located?

Thanks
PTide
Veeam Software
 
Posts: 3252
Liked: 273 times
Joined: Tue May 19, 2015 1:46 pm

Re: Terribly, terribly slow!!!

Veeam Logoby pama » Mon Jul 18, 2016 6:29 am

Thanks man,
the CIFS folder is on a Windows 2008R2 server, connected in Gigabit LAN.

Two days ago the bottleneck was the target, with speed of 800KB/s, and backup failed after 3GB.
Yesterday I have seen 11MB/s, bottlenek source,but failed after 20GB of backup, freezing the source.

Currently I am disabling Veeam job.
Thank you
pama
Enthusiast
 
Posts: 40
Liked: 1 time
Joined: Wed Apr 20, 2011 1:02 pm
Full Name: Andrea de Lutti

Re: Terribly, terribly slow!!!

Veeam Logoby vmniels » Mon Jul 18, 2016 7:40 am

Just wondering a few things to help here. Could you tell us how big the server is you are trying to backup (or give the output from df -h). I am guessing the digital signing of files by Windows could be causing an issue. Is this server part of a domain or is this a solo setup?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be
vmniels
Veeam Software
 
Posts: 1633
Liked: 362 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Terribly, terribly slow!!!

Veeam Logoby pama » Mon Jul 18, 2016 8:01 am

vmniels wrote:Just wondering a few things to help here. Could you tell us how big the server is you are trying to backup (or give the output from df -h). I am guessing the digital signing of files by Windows could be causing an issue. Is this server part of a domain or is this a solo setup?


System is not joined to domain, and it is 58GB big:

Code: Select all
~# df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                          1.9G     0  1.9G   0% /dev
tmpfs                         382M  5.6M  377M   2% /run
/dev/mapper/nvidia_fbcfiaca1  226G   58G  157G  27% /
tmpfs                         1.9G     0  1.9G   0% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                         1.9G     0  1.9G   0% /sys/fs/cgroup
cgmfs                         100K     0  100K   0% /run/cgmanager/fs
tmpfs                         382M     0  382M   0% /run/user/0


It runs Mysql and Apache, it is an Owncloud installation over Ubuntu 14.04 LTS 64bit.
pama
Enthusiast
 
Posts: 40
Liked: 1 time
Joined: Wed Apr 20, 2011 1:02 pm
Full Name: Andrea de Lutti

Re: Terribly, terribly slow!!!

Veeam Logoby vmniels » Mon Jul 18, 2016 8:33 am

So if I understand it correct, the Linux VM is running in owncloud and the Windows server as well? The Windows server is isolated from a domain as well so it is a standalone install?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be
vmniels
Veeam Software
 
Posts: 1633
Liked: 362 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Terribly, terribly slow!!!

Veeam Logoby pama » Mon Jul 18, 2016 9:03 am

vmniels wrote:So if I understand it correct, the Linux VM is running in owncloud and the Windows server as well? The Windows server is isolated from a domain as well so it is a standalone install?


I use two HP Microserver, both are physical installations. One is Linux with Owncloud, the other one is a Windows 2008 R2 Domain controller.
pama
Enthusiast
 
Posts: 40
Liked: 1 time
Joined: Wed Apr 20, 2011 1:02 pm
Full Name: Andrea de Lutti

Re: Terribly, terribly slow!!!

Veeam Logoby vmniels » Mon Jul 18, 2016 9:11 am

Could you try the following on the domain controller:
- Run GPMC.MSC
- Right click on the default domain policy and select edit
- Navigate through the Group Policy Management Editor to Computer Configuration | Policies | Windows Settings | Security Settings | Local Policies | Security Options
- Once there disable the Microsoft Network Server: Digitally Sign Communications (Always) policy. You will also need to disable the Domain Member: Digitally Encrypt or Sign Secure Channel Data (Always) policy.

Then save and close everything and try to run the backup again and see if it improves.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be
vmniels
Veeam Software
 
Posts: 1633
Liked: 362 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Terribly, terribly slow!!!

Veeam Logoby pama » Mon Jul 18, 2016 12:06 pm

vmniels wrote:Could you try the following on the domain controller:
- Run GPMC.MSC
- Right click on the default domain policy and select edit
- Navigate through the Group Policy Management Editor to Computer Configuration | Policies | Windows Settings | Security Settings | Local Policies | Security Options
- Once there disable the Microsoft Network Server: Digitally Sign Communications (Always) policy. You will also need to disable the Domain Member: Digitally Encrypt or Sign Secure Channel Data (Always) policy.

Then save and close everything and try to run the backup again and see if it improves.


Thank you, have tried and backupped 12,3GB in 2h30, then freezed again...
pama
Enthusiast
 
Posts: 40
Liked: 1 time
Joined: Wed Apr 20, 2011 1:02 pm
Full Name: Andrea de Lutti

Re: Terribly, terribly slow!!!

Veeam Logoby PTide » Mon Jul 18, 2016 12:17 pm

/dev/mapper/nvidia_fbcfiaca1 226G 58G 157G 27% /
<...>I use two HP Microserver, both are physical installations<...>
AFAIR you had a bulit-in RAID configured on at least one of them. Do you have a built-in RAID enabled on this machine too?
PTide
Veeam Software
 
Posts: 3252
Liked: 273 times
Joined: Tue May 19, 2015 1:46 pm

Re: Terribly, terribly slow!!!

Veeam Logoby pama » Mon Jul 18, 2016 12:32 pm

PTide wrote:AFAIR you had a bulit-in RAID configured on at least one of them. Do you have a built-in RAID enabled on this machine too?


Yes, I have a native RAID (onboard device) activated on the Linux system, not in the Windows one.
pama
Enthusiast
 
Posts: 40
Liked: 1 time
Joined: Wed Apr 20, 2011 1:02 pm
Full Name: Andrea de Lutti

Re: Terribly, terribly slow!!!

Veeam Logoby vmniels » Mon Jul 18, 2016 1:59 pm

I have been checking your logs and I notice a few things. How much memory does this Linux VM have? Also does it have any swap memory? If you can give us the output of 'free -m' we can try to simulate it as well to see what the minimum treshold would be.

I am seeing kills happening on the veeamagent since you are running out of memory so this seems to be the major issue.
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be
vmniels
Veeam Software
 
Posts: 1633
Liked: 362 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Terribly, terribly slow!!!

Veeam Logoby pama » Mon Jul 18, 2016 2:59 pm

vmniels wrote:I have been checking your logs and I notice a few things. How much memory does this Linux VM have? Also does it have any swap memory? If you can give us the output of 'free -m' we can try to simulate it as well to see what the minimum treshold would be.

I am seeing kills happening on the veeamagent since you are running out of memory so this seems to be the major issue.


The server has 4GB ram, it is not a VM.
Here is the result of "free -m"

Code: Select all
# free -m
              total        used        free      shared  buff/cache   available
Mem:           3818         160        3092          16         565        3593
Swap:          3958         115        3843


I will look to increase memory, But I have never had problem with it...
pama
Enthusiast
 
Posts: 40
Liked: 1 time
Joined: Wed Apr 20, 2011 1:02 pm
Full Name: Andrea de Lutti

Re: Terribly, terribly slow!!!

Veeam Logoby vmniels » Mon Jul 18, 2016 3:03 pm 1 person likes this post

Can you try the following as a matter of test:
sysctl vm.overcommit_memory=2
echo "vm.overcommit_memory=2" >> /etc/sysctl.conf

And then try to run the backup to see what happens? If it would fail can you provide your syslog, dmesg -T and veeam log files?
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be
vmniels
Veeam Software
 
Posts: 1633
Liked: 362 times
Joined: Mon Jul 15, 2013 11:09 am
Full Name: Niels Engelen

Re: Terribly, terribly slow!!!

Veeam Logoby pama » Tue Jul 19, 2016 2:30 pm

I have had some hope....After 25% it fails. Here are the logs

TY
pama
Enthusiast
 
Posts: 40
Liked: 1 time
Joined: Wed Apr 20, 2011 1:02 pm
Full Name: Andrea de Lutti

Next

Return to Veeam Agent for Linux



Who is online

Users browsing this forum: No registered users and 1 guest