-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
Terribly, terribly slow!!!
I have tried several times, but I am finding Veeam for Linux terribly slow.
I have never finished a backup.
I use CIFS folder, used before with scp file transfer from the same server without any problem. I attach here my backup HDD benchmark (I Know, it is not faster than other, but it is for backup).
Each time I run the backup I have different bottlecages: one time proxy, one time destination, one time source..... so for me it is impossible to understand.
Here I attach also veeam logs, syslog and kernel dump. link
Thank you for your investigation
Andrea
I have never finished a backup.
I use CIFS folder, used before with scp file transfer from the same server without any problem. I attach here my backup HDD benchmark (I Know, it is not faster than other, but it is for backup).
Each time I run the backup I have different bottlecages: one time proxy, one time destination, one time source..... so for me it is impossible to understand.
Here I attach also veeam logs, syslog and kernel dump. link
Thank you for your investigation
Andrea
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: Terribly, terribly slow!!!
Did I get it right - the very same job shows different bottlenecks in stats? Where is CIFS share located?
Thanks
Thanks
-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
Re: Terribly, terribly slow!!!
Thanks man,
the CIFS folder is on a Windows 2008R2 server, connected in Gigabit LAN.
Two days ago the bottleneck was the target, with speed of 800KB/s, and backup failed after 3GB.
Yesterday I have seen 11MB/s, bottlenek source,but failed after 20GB of backup, freezing the source.
Currently I am disabling Veeam job.
Thank you
the CIFS folder is on a Windows 2008R2 server, connected in Gigabit LAN.
Two days ago the bottleneck was the target, with speed of 800KB/s, and backup failed after 3GB.
Yesterday I have seen 11MB/s, bottlenek source,but failed after 20GB of backup, freezing the source.
Currently I am disabling Veeam job.
Thank you
-
- Product Manager
- Posts: 5797
- Liked: 1215 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: Terribly, terribly slow!!!
Just wondering a few things to help here. Could you tell us how big the server is you are trying to backup (or give the output from df -h). I am guessing the digital signing of files by Windows could be causing an issue. Is this server part of a domain or is this a solo setup?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
Re: Terribly, terribly slow!!!
System is not joined to domain, and it is 58GB big:vmniels wrote:Just wondering a few things to help here. Could you tell us how big the server is you are trying to backup (or give the output from df -h). I am guessing the digital signing of files by Windows could be causing an issue. Is this server part of a domain or is this a solo setup?
Code: Select all
~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 382M 5.6M 377M 2% /run
/dev/mapper/nvidia_fbcfiaca1 226G 58G 157G 27% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
cgmfs 100K 0 100K 0% /run/cgmanager/fs
tmpfs 382M 0 382M 0% /run/user/0
-
- Product Manager
- Posts: 5797
- Liked: 1215 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: Terribly, terribly slow!!!
So if I understand it correct, the Linux VM is running in owncloud and the Windows server as well? The Windows server is isolated from a domain as well so it is a standalone install?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
Re: Terribly, terribly slow!!!
I use two HP Microserver, both are physical installations. One is Linux with Owncloud, the other one is a Windows 2008 R2 Domain controller.vmniels wrote:So if I understand it correct, the Linux VM is running in owncloud and the Windows server as well? The Windows server is isolated from a domain as well so it is a standalone install?
-
- Product Manager
- Posts: 5797
- Liked: 1215 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: Terribly, terribly slow!!!
Could you try the following on the domain controller:
- Run GPMC.MSC
- Right click on the default domain policy and select edit
- Navigate through the Group Policy Management Editor to Computer Configuration | Policies | Windows Settings | Security Settings | Local Policies | Security Options
- Once there disable the Microsoft Network Server: Digitally Sign Communications (Always) policy. You will also need to disable the Domain Member: Digitally Encrypt or Sign Secure Channel Data (Always) policy.
Then save and close everything and try to run the backup again and see if it improves.
- Run GPMC.MSC
- Right click on the default domain policy and select edit
- Navigate through the Group Policy Management Editor to Computer Configuration | Policies | Windows Settings | Security Settings | Local Policies | Security Options
- Once there disable the Microsoft Network Server: Digitally Sign Communications (Always) policy. You will also need to disable the Domain Member: Digitally Encrypt or Sign Secure Channel Data (Always) policy.
Then save and close everything and try to run the backup again and see if it improves.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
Re: Terribly, terribly slow!!!
Thank you, have tried and backupped 12,3GB in 2h30, then freezed again...vmniels wrote:Could you try the following on the domain controller:
- Run GPMC.MSC
- Right click on the default domain policy and select edit
- Navigate through the Group Policy Management Editor to Computer Configuration | Policies | Windows Settings | Security Settings | Local Policies | Security Options
- Once there disable the Microsoft Network Server: Digitally Sign Communications (Always) policy. You will also need to disable the Domain Member: Digitally Encrypt or Sign Secure Channel Data (Always) policy.
Then save and close everything and try to run the backup again and see if it improves.
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: Terribly, terribly slow!!!
AFAIR you had a bulit-in RAID configured on at least one of them. Do you have a built-in RAID enabled on this machine too?/dev/mapper/nvidia_fbcfiaca1 226G 58G 157G 27% /
<...>I use two HP Microserver, both are physical installations<...>
-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
Re: Terribly, terribly slow!!!
Yes, I have a native RAID (onboard device) activated on the Linux system, not in the Windows one.PTide wrote:AFAIR you had a bulit-in RAID configured on at least one of them. Do you have a built-in RAID enabled on this machine too?
-
- Product Manager
- Posts: 5797
- Liked: 1215 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: Terribly, terribly slow!!!
I have been checking your logs and I notice a few things. How much memory does this Linux VM have? Also does it have any swap memory? If you can give us the output of 'free -m' we can try to simulate it as well to see what the minimum treshold would be.
I am seeing kills happening on the veeamagent since you are running out of memory so this seems to be the major issue.
I am seeing kills happening on the veeamagent since you are running out of memory so this seems to be the major issue.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
Re: Terribly, terribly slow!!!
The server has 4GB ram, it is not a VM.vmniels wrote:I have been checking your logs and I notice a few things. How much memory does this Linux VM have? Also does it have any swap memory? If you can give us the output of 'free -m' we can try to simulate it as well to see what the minimum treshold would be.
I am seeing kills happening on the veeamagent since you are running out of memory so this seems to be the major issue.
Here is the result of "free -m"
Code: Select all
# free -m
total used free shared buff/cache available
Mem: 3818 160 3092 16 565 3593
Swap: 3958 115 3843
-
- Product Manager
- Posts: 5797
- Liked: 1215 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: Terribly, terribly slow!!!
Can you try the following as a matter of test:
sysctl vm.overcommit_memory=2
echo "vm.overcommit_memory=2" >> /etc/sysctl.conf
And then try to run the backup to see what happens? If it would fail can you provide your syslog, dmesg -T and veeam log files?
sysctl vm.overcommit_memory=2
echo "vm.overcommit_memory=2" >> /etc/sysctl.conf
And then try to run the backup to see what happens? If it would fail can you provide your syslog, dmesg -T and veeam log files?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: Terribly, terribly slow!!!
As it's been stated before - Beta is not guaranteed to be fast and stable with a built-in HW RAID.Yes, I have a native RAID (onboard device) activated on the Linux system, not in the Windows one.
Thanks
-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
Re: Terribly, terribly slow!!!
Thank you, I will look forward for a stable version!
BR
Andrea
BR
Andrea
-
- Product Manager
- Posts: 5797
- Liked: 1215 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: Terribly, terribly slow!!!
Besides the fact that it may not be stable enough yet, I did check the logs.
I notice cgroups are enabled and keep killing your veeam backup process. I guess this is preconfigured on the system. Removing cgroups (control groups) will probably resolve the issue why the job keeps failing.
Furthermore I noticed the following Cannot store data to snapshot. Not enough space. This however could be related to a known bug.
I notice cgroups are enabled and keep killing your veeam backup process. I guess this is preconfigured on the system. Removing cgroups (control groups) will probably resolve the issue why the job keeps failing.
Furthermore I noticed the following Cannot store data to snapshot. Not enough space. This however could be related to a known bug.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
Re: Terribly, terribly slow!!!
vmniels wrote:Besides the fact that it may not be stable enough yet, I did check the logs.
I notice cgroups are enabled and keep killing your veeam backup process. I guess this is preconfigured on the system. Removing cgroups (control groups) will probably resolve the issue why the job keeps failing.
Furthermore I noticed the following Cannot store data to snapshot. Not enough space. This however could be related to a known bug.
Thanks a lot for your deep investigation.
I don't use cgroups. I think they are part of installation.
About free space, should this be related to inodes? even if I have free ones....
Here is my disk occupation:
Code: Select all
# df -i -h
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 465K 486 465K 1% /dev
tmpfs 478K 605 477K 1% /run
/dev/mapper/nvidia_fbcfiaca1 15M 190K 15M 2% /
tmpfs 478K 1 478K 1% /dev/shm
tmpfs 478K 8 478K 1% /run/lock
tmpfs 478K 18 478K 1% /sys/fs/cgroup
cgmfs 478K 14 478K 1% /run/cgmanager/fs
tmpfs 478K 4 478K 1% /run/user/0
Code: Select all
# df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 382M 11M 372M 3% /run
/dev/mapper/nvidia_fbcfiaca1 226G 57G 158G 27% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
cgmfs 100K 0 100K 0% /run/cgmanager/fs
tmpfs 382M 0 382M 0% /run/user/0
Thank you,
Andrea
-
- Product Manager
- Posts: 5797
- Liked: 1215 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: Terribly, terribly slow!!!
tmpfs is nothing to worry about, this is a common name for a temporary file storage facility.
In regards to the free space, this can be related to the snapshot overflow known issue.
Cgroups is loaded in your server and this happens quite often to prevent servers from completely crashing resulting in a hard reboot/reset. However it also makes sure your cpu is throttled lowering the CPU usage and thus power usage in the long run . The reason why this is loaded is also related to systemd (default nowadays for many distro's).
You can try to disable it temporary to see if the backup continues however I guess the space error will come back up.
In regards to the free space, this can be related to the snapshot overflow known issue.
Cgroups is loaded in your server and this happens quite often to prevent servers from completely crashing resulting in a hard reboot/reset. However it also makes sure your cpu is throttled lowering the CPU usage and thus power usage in the long run . The reason why this is loaded is also related to systemd (default nowadays for many distro's).
You can try to disable it temporary to see if the backup continues however I guess the space error will come back up.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
-
- Enthusiast
- Posts: 42
- Liked: 1 time
- Joined: Apr 20, 2011 1:02 pm
- Full Name: Andrea de Lutti
- Contact:
Re: Terribly, terribly slow!!!
I think now I will wait for a second beta, or for a stable version...I bet it will released soon, isn't it?vmniels wrote: You can try to disable it temporary to see if the backup continues however I guess the space error will come back up.
-
- Product Manager
- Posts: 5797
- Liked: 1215 times
- Joined: Jul 15, 2013 11:09 am
- Full Name: Niels Engelen
- Contact:
Re: Terribly, terribly slow!!!
When it's ready . We are still collecting feedback and improving it.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
GitHub: https://github.com/nielsengelen
Who is online
Users browsing this forum: No registered users and 13 guests