Network probe to increase transfer speeds

[togenshi]

Hey,

I have a remote linux box that I wish to backup to a Veeam B+R Repository. I am experiencing really low speeds but I haven't seen the processing rate increase again after on-peak periods. When I use iperf, I was able to achieve 8Mbits/sec to the Veeam B+R server however VAL is processing the job at 148 KB/s. VAL recognises the bottleneck being the network. I will note that the ping is ~60ms. I do not believe that the VAL software has probed the network to see if the transfer speeds can increase.

Currently I have a Veeam job on day 7 and only processed 72GB.

System: Ubuntu 14.04 on a Dell R630 using Broadcom Corporation NetXtreme BCM5720 Gigabit PCIe NIC.

Code: Select all

                                                                                                      Veeam Agent for Linux   [  ]


     Backup                                                                                                      16%                                                                                                     Status: Running

     

     Summary                              Data

     Duration:           21:43:26         Processed:          72.2 GB (16%)
     Processing rate:    148.3 KB/s       Read:               72.2 GB
     Bottleneck:         Network          Transferred:        14.3 GB (5.1x)

      Time            Action                                                                                                                                                                                                 Duration  
      
      18:00:02        Job Scream started at 2017-01-11 18:00:02 AEDT
      18:00:28        Preparing to backup
      18:00:53        Creating volume snapshot                                                                                                                                                                               00:00:02
      18:00:57        Starting full backup to [VC1]Onsite Linux Endpoints
      18:01:07        Backing up BIOS bootloader on /dev/sda                                                                                                                                                                 00:00:00
      18:01:18        Backed up sda 54.3 GB at 758.8 kB/s                                                                                                                                                                    20:49:48
      14:54:30 (+1)   Backing up sdb 17.9 GB at 43.1 kB/s (0%)                                                                                                                                                               00:48:43

Excerpt from iperf:

[ ID] Interval Transfer Bandwidth
[ 3] 0.0-50.3 sec 46.5 MBytes 7.76 Mbits/sec

I am also getting the following in the backup log that may or may not cause the issue:

Code: Select all

WARN|IO rate limiter sleep time is too high: [39520] milliseconds. Will be reduced.

Do you have any suggestions to speed up the backup process?

[PTide]

Hi,

What kind of VBR repository do you use and where is it located?

Thanks

[togenshi]

B+R Verison 9.5.0.711 on a Repository that is on Windows local disks. Distance is 3500km (other side of Australia).

At this moment, we are rsyncing the data over to a local (to VBR) server and using VAL from there. I would like to get rid of the middle man (server).

[togenshi]

Just an update, transfer is still going but the processing rate is now down to 117KB/s. It has not increased at all. This is over a MPLS network with 20MB async.

[PTide]

Well, your stats clearly show it's the network being a bottleneck. At the moment I'm not sure about possible ways to speed up the backup over 3500km 20 Mb link. Also what speed do you expect to get on that link? Have you tried to transfer one large file (say, 1 Gb)?

WARN|IO rate limiter sleep time is too high: [39520] milliseconds. Will be reduced.

Rate limiter does not let backup job to overload hard drives. The message that you got says that IO rate limiter has been inactive for a very long time (due to the network being the slowest component) and it is a result, not the reason for such slowdown. That's a rare situation and should not normally occur.

Thank you

[togenshi]

Well, thats the thing, I expect a transfer rate to be more than 2MB/s average but as the job progresses, the transfer speed has never increased. Iperf shows that 7MB/s is achievable. The job is currently transferring at 110KB/s. The job has been running for over a week and it has only processed 80.2GB.

This is currently the iperf results:

[ ID] Interval Transfer Bandwidth
[ 3] 0.0-50.2 sec 48.6 MBytes 8.13 Mbits/sec

Yet this is the current progress of the veeam job:

Code: Select all

Processing rate:    110.4 KB/s       Read:               80.2 GB          
     Bottleneck:         Network          Transferred:        21.7 GB (3.7x)   

      Time            Action                                                                              Duration  

      18:00:02        Job Scream started at 2017-01-11 18:00:02 AEDT
      18:00:28        Preparing to backup
      18:00:53        Creating volume snapshot                                                            00:00:02

Is there a way to record this transfer speed of a job so I can push it into a graph?

[PTide]

After the job will be finished, please try to copy /usr/src directory from the server to the remote destination via the same link. The directory contains lots of small files and might serve as a simulation of backup transmission process.

Is there a way to record this transfer speed of a job so I can push it into a graph?

Speed numbers can be obtained from job log:

Code: Select all

cat /var/log/veeam/Backup/BackupJob/Session_YYYYMMDD_HHMMSS_\{ID}/Job.log grep | speed

Thanks

[simplicity]

Not trying to hijack the post but we are having a similar issue doing a replication with VMware and VBR 9.5 and posting for just informational purpose.

Replicating over a L2 100MB connection from site to site.
We have worked with Veeam support and they taking a stance that it a network issue but we have not been able to find an issue with the network. We are going from coast to coast with a steady 48m sec latency from proxy to proxy.
We will get disks transferring between 15-30MB a sec and then randomly we will get a disk to copy at 3MB or less (sometimes less than 512kb) during this time other smaller replication jobs will start and will copy at the remaining bandwidth available on the connection.
It seems to have a pattern once it drops down to 3MB~ it will never increase again. Seems to be once the job exceeds 48hr~
While support says it is network issue the bottleneck says it is the source each time and normally will switch from network to source which seems to be after 24-48hr or if one of the disks starts out at a 3MB transfer speed the rest of the disks will transfer at the available bandwidth then drop down to be the source leaving the last disk transferring at 3MB~.

We have opened a case with VMware and they could not find anything wrong with the storage, vm config, vmdks and pushed it back to Veeam.

This happens on multiple vm's, esxi hosts. CBT enabled and reset many times

In short there seems to be something that will slow down the transfer/write and then will never increase again. The the two common things between your issue and our seems to be a long distance connection and VBR 9.5 (we also saw the same issue in 9.0)

[PTide]

@simplicity

Hi,

We have worked with Veeam support and they taking a stance that it a network issue but we have not been able to find an issue with the network.

Could you provide case ID for reference please?

Thank you

[simplicity]

Could you provide case ID for reference please?

Thank you

01912147