Network probe to increase transfer speeds

Backup agent for Linux servers and workstations on-premises or in the public cloud

Network probe to increase transfer speeds

Veeam Logoby togenshi » Wed Jan 18, 2017 4:13 am

Hey,

I have a remote linux box that I wish to backup to a Veeam B+R Repository. I am experiencing really low speeds but I haven't seen the processing rate increase again after on-peak periods. When I use iperf, I was able to achieve 8Mbits/sec to the Veeam B+R server however VAL is processing the job at 148 KB/s. VAL recognises the bottleneck being the network. I will note that the ping is ~60ms. I do not believe that the VAL software has probed the network to see if the transfer speeds can increase.

Currently I have a Veeam job on day 7 and only processed 72GB.

System: Ubuntu 14.04 on a Dell R630 using Broadcom Corporation NetXtreme BCM5720 Gigabit PCIe NIC.

Code: Select all
                                                                                                      Veeam Agent for Linux   [  ]


     Backup                                                                                                      16%                                                                                                     Status: Running

     

     Summary                              Data

     Duration:           21:43:26         Processed:          72.2 GB (16%)
     Processing rate:    148.3 KB/s       Read:               72.2 GB
     Bottleneck:         Network          Transferred:        14.3 GB (5.1x)

      Time            Action                                                                                                                                                                                                 Duration 
     
      18:00:02        Job Scream started at 2017-01-11 18:00:02 AEDT
      18:00:28        Preparing to backup
      18:00:53        Creating volume snapshot                                                                                                                                                                               00:00:02
      18:00:57        Starting full backup to [VC1]Onsite Linux Endpoints
      18:01:07        Backing up BIOS bootloader on /dev/sda                                                                                                                                                                 00:00:00
      18:01:18        Backed up sda 54.3 GB at 758.8 kB/s                                                                                                                                                                    20:49:48
      14:54:30 (+1)   Backing up sdb 17.9 GB at 43.1 kB/s (0%)                                                                                                                                                               00:48:43


Excerpt from iperf:

[ ID] Interval Transfer Bandwidth
[ 3] 0.0-50.3 sec 46.5 MBytes 7.76 Mbits/sec

I am also getting the following in the backup log that may or may not cause the issue:

Code: Select all
WARN|IO rate limiter sleep time is too high: [39520] milliseconds. Will be reduced.


Do you have any suggestions to speed up the backup process?
togenshi
Influencer
 
Posts: 22
Liked: 4 times
Joined: Mon Jul 27, 2015 1:16 am

Re: Network probe to increase transfer speeds

Veeam Logoby PTide » Wed Jan 18, 2017 11:43 am

Hi,

What kind of VBR repository do you use and where is it located?

Thanks
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Network probe to increase transfer speeds

Veeam Logoby togenshi » Thu Jan 19, 2017 2:27 am

B+R Verison 9.5.0.711 on a Repository that is on Windows local disks. Distance is 3500km (other side of Australia).

At this moment, we are rsyncing the data over to a local (to VBR) server and using VAL from there. I would like to get rid of the middle man (server).
togenshi
Influencer
 
Posts: 22
Liked: 4 times
Joined: Mon Jul 27, 2015 1:16 am

Re: Network probe to increase transfer speeds

Veeam Logoby togenshi » Thu Jan 19, 2017 7:15 am

Just an update, transfer is still going but the processing rate is now down to 117KB/s. It has not increased at all. This is over a MPLS network with 20MB async.
togenshi
Influencer
 
Posts: 22
Liked: 4 times
Joined: Mon Jul 27, 2015 1:16 am

Re: Network probe to increase transfer speeds

Veeam Logoby PTide » Thu Jan 19, 2017 4:42 pm

Well, your stats clearly show it's the network being a bottleneck. At the moment I'm not sure about possible ways to speed up the backup over 3500km 20 Mb link. Also what speed do you expect to get on that link? Have you tried to transfer one large file (say, 1 Gb)?

WARN|IO rate limiter sleep time is too high: [39520] milliseconds. Will be reduced.
Rate limiter does not let backup job to overload hard drives. The message that you got says that IO rate limiter has been inactive for a very long time (due to the network being the slowest component) and it is a result, not the reason for such slowdown. That's a rare situation and should not normally occur.

Thank you
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Network probe to increase transfer speeds

Veeam Logoby togenshi » Fri Jan 20, 2017 2:37 am

Well, thats the thing, I expect a transfer rate to be more than 2MB/s average but as the job progresses, the transfer speed has never increased. Iperf shows that 7MB/s is achievable. The job is currently transferring at 110KB/s. The job has been running for over a week and it has only processed 80.2GB.

This is currently the iperf results:

[ ID] Interval Transfer Bandwidth
[ 3] 0.0-50.2 sec 48.6 MBytes 8.13 Mbits/sec

Yet this is the current progress of the veeam job:
Code: Select all
Processing rate:    110.4 KB/s       Read:               80.2 GB         
     Bottleneck:         Network          Transferred:        21.7 GB (3.7x)   

      Time            Action                                                                              Duration 

      18:00:02        Job Scream started at 2017-01-11 18:00:02 AEDT
      18:00:28        Preparing to backup
      18:00:53        Creating volume snapshot                                                            00:00:02


Is there a way to record this transfer speed of a job so I can push it into a graph?
togenshi
Influencer
 
Posts: 22
Liked: 4 times
Joined: Mon Jul 27, 2015 1:16 am

Re: Network probe to increase transfer speeds

Veeam Logoby PTide » Fri Jan 20, 2017 10:56 am

After the job will be finished, please try to copy /usr/src directory from the server to the remote destination via the same link. The directory contains lots of small files and might serve as a simulation of backup transmission process.

Is there a way to record this transfer speed of a job so I can push it into a graph?

Speed numbers can be obtained from job log:

Code: Select all
cat /var/log/veeam/Backup/BackupJob/Session_YYYYMMDD_HHMMSS_\{ID}/Job.log grep | speed


Thanks
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Network probe to increase transfer speeds

Veeam Logoby simplicity » Sat Jan 28, 2017 8:03 pm

Not trying to hijack the post but we are having a similar issue doing a replication with VMware and VBR 9.5 and posting for just informational purpose.

Replicating over a L2 100MB connection from site to site.
We have worked with Veeam support and they taking a stance that it a network issue but we have not been able to find an issue with the network. We are going from coast to coast with a steady 48m sec latency from proxy to proxy.
We will get disks transferring between 15-30MB a sec and then randomly we will get a disk to copy at 3MB or less (sometimes less than 512kb) during this time other smaller replication jobs will start and will copy at the remaining bandwidth available on the connection.
It seems to have a pattern once it drops down to 3MB~ it will never increase again. Seems to be once the job exceeds 48hr~
While support says it is network issue the bottleneck says it is the source each time and normally will switch from network to source which seems to be after 24-48hr or if one of the disks starts out at a 3MB transfer speed the rest of the disks will transfer at the available bandwidth then drop down to be the source leaving the last disk transferring at 3MB~.

We have opened a case with VMware and they could not find anything wrong with the storage, vm config, vmdks and pushed it back to Veeam.

This happens on multiple vm's, esxi hosts. CBT enabled and reset many times

In short there seems to be something that will slow down the transfer/write and then will never increase again. The the two common things between your issue and our seems to be a long distance connection and VBR 9.5 (we also saw the same issue in 9.0)
simplicity
Service Provider
 
Posts: 12
Liked: 3 times
Joined: Sat Jun 06, 2015 2:46 am
Full Name: Rick Baranowski

Re: Network probe to increase transfer speeds

Veeam Logoby PTide » Mon Jan 30, 2017 11:12 am

@simplicity

Hi,

We have worked with Veeam support and they taking a stance that it a network issue but we have not been able to find an issue with the network.

Could you provide case ID for reference please?

Thank you
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Network probe to increase transfer speeds

Veeam Logoby simplicity » Mon Jan 30, 2017 9:03 pm

Could you provide case ID for reference please?

Thank you


01912147
simplicity
Service Provider
 
Posts: 12
Liked: 3 times
Joined: Sat Jun 06, 2015 2:46 am
Full Name: Rick Baranowski


Return to Veeam Agent for Linux



Who is online

Users browsing this forum: No registered users and 2 guests