Terribly, terribly slow!!!

[pama]

I have tried several times, but I am finding Veeam for Linux terribly slow.

I have never finished a backup.
I use CIFS folder, used before with scp file transfer from the same server without any problem. I attach here my backup HDD benchmark (I Know, it is not faster than other, but it is for backup).


Each time I run the backup I have different bottlecages: one time proxy, one time destination, one time source..... so for me it is impossible to understand.
Here I attach also veeam logs, syslog and kernel dump. link

Thank you for your investigation
Andrea

[PTide]

Did I get it right - the very same job shows different bottlenecks in stats? Where is CIFS share located?

Thanks

[pama]

Thanks man,
the CIFS folder is on a Windows 2008R2 server, connected in Gigabit LAN.

Two days ago the bottleneck was the target, with speed of 800KB/s, and backup failed after 3GB.
Yesterday I have seen 11MB/s, bottlenek source,but failed after 20GB of backup, freezing the source.

Currently I am disabling Veeam job.
Thank you

[nielsengelen]

Just wondering a few things to help here. Could you tell us how big the server is you are trying to backup (or give the output from df -h). I am guessing the digital signing of files by Windows could be causing an issue. Is this server part of a domain or is this a solo setup?

[pama]

Just wondering a few things to help here. Could you tell us how big the server is you are trying to backup (or give the output from df -h). I am guessing the digital signing of files by Windows could be causing an issue. Is this server part of a domain or is this a solo setup?

System is not joined to domain, and it is 58GB big:

Code: Select all

~# df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                          1.9G     0  1.9G   0% /dev
tmpfs                         382M  5.6M  377M   2% /run
/dev/mapper/nvidia_fbcfiaca1  226G   58G  157G  27% /
tmpfs                         1.9G     0  1.9G   0% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                         1.9G     0  1.9G   0% /sys/fs/cgroup
cgmfs                         100K     0  100K   0% /run/cgmanager/fs
tmpfs                         382M     0  382M   0% /run/user/0

It runs Mysql and Apache, it is an Owncloud installation over Ubuntu 14.04 LTS 64bit.

[nielsengelen]

So if I understand it correct, the Linux VM is running in owncloud and the Windows server as well? The Windows server is isolated from a domain as well so it is a standalone install?

[pama]

So if I understand it correct, the Linux VM is running in owncloud and the Windows server as well? The Windows server is isolated from a domain as well so it is a standalone install?

I use two HP Microserver, both are physical installations. One is Linux with Owncloud, the other one is a Windows 2008 R2 Domain controller.

[nielsengelen]

Could you try the following on the domain controller:
- Run GPMC.MSC
- Right click on the default domain policy and select edit
- Navigate through the Group Policy Management Editor to Computer Configuration | Policies | Windows Settings | Security Settings | Local Policies | Security Options
- Once there disable the Microsoft Network Server: Digitally Sign Communications (Always) policy. You will also need to disable the Domain Member: Digitally Encrypt or Sign Secure Channel Data (Always) policy.

Then save and close everything and try to run the backup again and see if it improves.

[pama]

Could you try the following on the domain controller:
- Run GPMC.MSC
- Right click on the default domain policy and select edit
- Navigate through the Group Policy Management Editor to Computer Configuration | Policies | Windows Settings | Security Settings | Local Policies | Security Options
- Once there disable the Microsoft Network Server: Digitally Sign Communications (Always) policy. You will also need to disable the Domain Member: Digitally Encrypt or Sign Secure Channel Data (Always) policy.

Then save and close everything and try to run the backup again and see if it improves.

Thank you, have tried and backupped 12,3GB in 2h30, then freezed again...

[PTide]

/dev/mapper/nvidia_fbcfiaca1 226G 58G 157G 27% /
<...>I use two HP Microserver, both are physical installations<...>

AFAIR you had a bulit-in RAID configured on at least one of them. Do you have a built-in RAID enabled on this machine too?

[pama]

AFAIR you had a bulit-in RAID configured on at least one of them. Do you have a built-in RAID enabled on this machine too?

Yes, I have a native RAID (onboard device) activated on the Linux system, not in the Windows one.

[nielsengelen]

I have been checking your logs and I notice a few things. How much memory does this Linux VM have? Also does it have any swap memory? If you can give us the output of 'free -m' we can try to simulate it as well to see what the minimum treshold would be.

I am seeing kills happening on the veeamagent since you are running out of memory so this seems to be the major issue.

[pama]

I have been checking your logs and I notice a few things. How much memory does this Linux VM have? Also does it have any swap memory? If you can give us the output of 'free -m' we can try to simulate it as well to see what the minimum treshold would be.

I am seeing kills happening on the veeamagent since you are running out of memory so this seems to be the major issue.

The server has 4GB ram, it is not a VM.
Here is the result of "free -m"

Code: Select all

# free -m
              total        used        free      shared  buff/cache   available
Mem:           3818         160        3092          16         565        3593
Swap:          3958         115        3843

I will look to increase memory, But I have never had problem with it...

[nielsengelen]

Can you try the following as a matter of test:
sysctl vm.overcommit_memory=2
echo "vm.overcommit_memory=2" >> /etc/sysctl.conf

And then try to run the backup to see what happens? If it would fail can you provide your syslog, dmesg -T and veeam log files?

[pama]

I have had some hope....After 25% it fails. Here are the logs

TY

[PTide]

Yes, I have a native RAID (onboard device) activated on the Linux system, not in the Windows one.

As it's been stated before - Beta is not guaranteed to be fast and stable with a built-in HW RAID.

Thanks

[pama]

Thank you, I will look forward for a stable version!

BR
Andrea

[nielsengelen]

Besides the fact that it may not be stable enough yet, I did check the logs.

I notice cgroups are enabled and keep killing your veeam backup process. I guess this is preconfigured on the system. Removing cgroups (control groups) will probably resolve the issue why the job keeps failing.

Furthermore I noticed the following Cannot store data to snapshot. Not enough space. This however could be related to a known bug.

[pama]

Besides the fact that it may not be stable enough yet, I did check the logs.

I notice cgroups are enabled and keep killing your veeam backup process. I guess this is preconfigured on the system. Removing cgroups (control groups) will probably resolve the issue why the job keeps failing.

Furthermore I noticed the following Cannot store data to snapshot. Not enough space. This however could be related to a known bug.

Thanks a lot for your deep investigation.
I don't use cgroups. I think they are part of installation.
About free space, should this be related to inodes? even if I have free ones....
Here is my disk occupation:

Code: Select all

# df -i -h
Filesystem                   Inodes IUsed IFree IUse% Mounted on
udev                           465K   486  465K    1% /dev
tmpfs                          478K   605  477K    1% /run
/dev/mapper/nvidia_fbcfiaca1    15M  190K   15M    2% /
tmpfs                          478K     1  478K    1% /dev/shm
tmpfs                          478K     8  478K    1% /run/lock
tmpfs                          478K    18  478K    1% /sys/fs/cgroup
cgmfs                          478K    14  478K    1% /run/cgmanager/fs
tmpfs                          478K     4  478K    1% /run/user/0

and

Code: Select all

# df -h
Filesystem                    Size  Used Avail Use% Mounted on
udev                          1.9G     0  1.9G   0% /dev
tmpfs                         382M   11M  372M   3% /run
/dev/mapper/nvidia_fbcfiaca1  226G   57G  158G  27% /
tmpfs                         1.9G     0  1.9G   0% /dev/shm
tmpfs                         5.0M     0  5.0M   0% /run/lock
tmpfs                         1.9G     0  1.9G   0% /sys/fs/cgroup
cgmfs                         100K     0  100K   0% /run/cgmanager/fs
tmpfs                         382M     0  382M   0% /run/user/0

I have never seen "tmpfs" before Veeam
Thank you,
Andrea

[nielsengelen]

tmpfs is nothing to worry about, this is a common name for a temporary file storage facility.

In regards to the free space, this can be related to the snapshot overflow known issue.

Cgroups is loaded in your server and this happens quite often to prevent servers from completely crashing resulting in a hard reboot/reset. However it also makes sure your cpu is throttled lowering the CPU usage and thus power usage in the long run . The reason why this is loaded is also related to systemd (default nowadays for many distro's).

You can try to disable it temporary to see if the backup continues however I guess the space error will come back up.

[pama]

You can try to disable it temporary to see if the backup continues however I guess the space error will come back up.

I think now I will wait for a second beta, or for a stable version...I bet it will released soon, isn't it?

[nielsengelen]

When it's ready . We are still collecting feedback and improving it.