Network Bottleneck

[lobo519]

I am running a backup job over the WAN (VPN). I can clearly see the available bandwidth and there is plenty available that is not being used - yet my Job shows the bottleneck as the Network.

Expected behavior?

My replication jobs usually have no problem filling the pipe.

I am not using any traffic throttling.

[foggy]

What are the whole bottleneck stats numbers for this job?

[Gostev]

Also, are you in patch #3, because it has enhancements around network load calculations.

[lobo519]

I am on patch 3. I'll post back when the job is finished. Is running a full and I am in hour 82.

[lobo519]

4/10/2012 11:32:27 PM :: Load: Source 0% > Proxy 11% > Network 98% > Target 0%

This job ran for 96 hours before I just cancelled it. I re-ran it after canceling it and it seems to be running better. It was a full backup

[Gostev]

Thank you - seeing these numbers makes troubleshooting so easy! Based on this statistics, my only guess is that there is some QoS mechanism in place on the networking equipment that limits bandwidth per connection. To verify this, try disabling multiple TCP/IP connections option in Veeam settings under Traffic Throttling settings, and see if this reduces the backup performance. I would recommend creating a very small VM for testing this, and trying full backup with and without multiple connections option enabled. And post the results (time to perform full backup) here. Thanks!

[lobo519]

hmm - well there is defiantly no QOS policies on any of our equipment.

[lobo519]

Would enabling the "Use multiple TCP/IP connections per job" setting help if indeed there was some QOS policy? Do I need to create a throttling rule for this setting to be used?

[Gostev]

Actually, this checkbox is enabled by default - are you saying you have it disabled right now?
No, this functionality is not dependent on throttling rules presence.

[lobo519]

Yes its disabled at the moment - Perhaps I should turn it back on!


Maybe a UI change is in-order for that check box..

[Gostev]

There is a good reason why this feature was added, and is enabled by default
Enabling it back will most definitely help with your WAN link utilization.

[lobo519]

Its only about 5 times faster!

Thanks!!

[Gostev]

Default is 5 TCP/IP connection per job, thus the number you are seeing. This also proves that you have some sort of QoS that limits each connection to the same amount of bandwidth. We actually have the same in our own environment on the WAN link between US and EMEA sites... no matter if I open 1 or 5 FTP sessions for upload, they all see to have the same (and slow) speed.

[lobo519]

Can you add more connections per job?!??!

I'm going to need to track down what is limiting per connection (not that its really a huge deal). It must be either on our providers end or a default setting in one of our ASAs - I set everything up!

[Gostev]

Yes, by adding this registry value in the B&R registry key on your backup management server:

Code: Select all

DownloadStreamsNumber
DWORD

[lobo519]

Excellent!

Any down side to changing this value? Do I need to restart any services/server for this to take effect?

[Gostev]

No need to restart, every newly starting job will take this setting into account.

Potential downsides:
1. Flooding your WAN link with backup traffic affecting anything else using this link.
2. Running into max connections amount. Apparently, some networking equipment seem to supports only limited amount of connections when there is VPN in play. Happens rarely indeed, but I personally got affected by this issue during our SE training last fall.
3. Some of those connections timing out and failing the job because of too many connection on too little bandwidth.

[lobo519]

is that value hex or dec?

[Gostev]

The value is REG_DWORD. You can enter it as HEX or DEC, just tell the Registry Editor what is it that you entered by selecting the corresponding radio button before clicking OK

[lobo519]

that what I figured but wanted to be sure. Thanks for your help!!

[joemagjr]

I also have some network bottleneck issues while running replication jobs. I also notice when running any of my replication jobs my network latency is increased to sometimes almost to 500ms and continues for 8-10 pings and then back down to 20ms from HQ to the DR site. We are trying to figure out what is causing this, but frankley this only occurs when running replication. Sometimes the jobs finish and sometimes they don't.


We are not using the veeam traffic throttle, but do use a network traffic throttle appliance and WAN optimiation (blue coat)that allows me to designate a certain amount of the 3mg pipe to veeam (I am also working with them and are finding nothing). I have also run these jobs without any throttle or optimization in the mix and let veeam saturate the 3meg pipe at night and I get the same results, sometimes the jobs finish and sometimes they don't. How else can I trouble shoot this network issue with veeam?
here is the config and some numbers;

At HQ site (ESXi5 host)
A VM B&R Win 2008 R2 on ESXi5 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's) also acts as repository for meta data
A VM proxy Win 2008 R2 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's)

At DR Site (also ESXi5 host)
A VM proxy Win 2008 R2 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's)

Target 0% Proxy 50% Network 99% Source 0%
Optimal compression with WAN target (have tried other compression settings same results)
Multiple TCP connection setting enabled (tried to disable (same results)
Proxy transport mode settings are set to SAN mode not automatic and datastore selection is set to automatic
real time stat show : Source proxy [san;ndb] and DR proxy [nbd]

this is the same error every single time no matter what job or data store it is replicating to;

Code: Select all

Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [12044]. Failed to replicate content of the disk [vddk://<vddkConnSpec><viConn name="VCENTER SERVER" authdPort="443" vicPort="443" /><vmxPath vmRef="vm-71" datacenterRef="datacenter-46" datacenterInventoryPath="DATACENTER NAME" snapshotRef="snapshot-5181" datastoreName="DATASTORE NAME" path="VMNAME/VMNAME.vmx" /><vmdkPath datastoreName="DATASTORENAME" path="VMNAME/VMNAME.vmdk" /><t

I am at a loss and yes I have a support ticket open and that is now getting esculated but I thought that i would reach out to the forum, since I see so much out here that could be related.

ALL HELP IS APPRECIATED

[Vitaliy S.]

I guess the easiest way to nail down this network issue is to try to run replication job without using WAN optimization tool (if possible) and compare the results. If it runs fine, then you will need to take a look at WAN optimization tool config. Thanks!

[CharlesGillanders]

Hi,

Might be stating the obvious but it's entirely possible you don't have a problem with your network. There's always a performance limit to a single TCP connection based on latency (something to do with the sliding window algorithm I forget exactly it's been a while since I needed to know this stuff). TCP connections through a VPN will have more latency than most and when there's a WAN involved as well then it may be that you've just discovered the practical limits of a single connection given your environment.

Multiple connections are a good way to resolve this issue.

Charles

[CaptainFred]

I've added this to HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication on my backup server running the replication job, should it also be added to the target proxy?

Does it take affect immediately? Or does the job need to be stopped/started or services restarted? I'm not seeing any improvement. Getting about 1MB/s!

I do have a VPN to the DR site across a 50Mbit internet link. Pings to the DR site when nothing is going on average at 14ms, thats not bad is it?

[foggy]

should it also be added to the target proxy?

No.

Does it take affect immediately? Or does the job need to be stopped/started or services restarted? I'm not seeing any improvement. Getting about 1MB/s!

It should take effect on every newly starting job, no need to restart services, as far as I know.

[CaptainFred]

Ok I changed it from 7 to 15 and that's helped, not maxed out the link but improved from 1MB/s to 4MB/s! Is there any risk or issue with having it set that high?

Btw, does it use ports 2500 and 2502 for the connections to the target proxy?

[foggy]

Ok I changed it from 7 to 15 and that's helped, not maxed out the link but improved from 1MB/s to 4MB/s! Is there any risk or issue with having it set that high?

Probably running into max connections amount.

Btw, does it use ports 2500 and 2502 for the connections to the target proxy?

Yes, those ports are used by the proxy service.

[CaptainFred]

Ok but is it safe to leave it at 15?

[CaptainFred]

And is there a "max connections amount" ?

[Gostev]

I know of a few customers using more than 20, so 15 is certainly "safe".
I don't believe there is a limit on max connection count.