Host-based backup of VMware vSphere VMs.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Network Bottleneck

Post by lobo519 »

I am running a backup job over the WAN (VPN). I can clearly see the available bandwidth and there is plenty available that is not being used - yet my Job shows the bottleneck as the Network.

Expected behavior?

My replication jobs usually have no problem filling the pipe.

I am not using any traffic throttling.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Network Bottleneck

Post by foggy »

What are the whole bottleneck stats numbers for this job?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Network Bottleneck

Post by Gostev »

Also, are you in patch #3, because it has enhancements around network load calculations.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Network Bottleneck

Post by lobo519 »

I am on patch 3. I'll post back when the job is finished. Is running a full and I am in hour 82.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Network Bottleneck

Post by lobo519 »

4/10/2012 11:32:27 PM :: Load: Source 0% > Proxy 11% > Network 98% > Target 0%

This job ran for 96 hours before I just cancelled it. I re-ran it after canceling it and it seems to be running better. It was a full backup
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Network Bottleneck

Post by Gostev »

Thank you - seeing these numbers makes troubleshooting so easy! Based on this statistics, my only guess is that there is some QoS mechanism in place on the networking equipment that limits bandwidth per connection. To verify this, try disabling multiple TCP/IP connections option in Veeam settings under Traffic Throttling settings, and see if this reduces the backup performance. I would recommend creating a very small VM for testing this, and trying full backup with and without multiple connections option enabled. And post the results (time to perform full backup) here. Thanks!
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Network Bottleneck

Post by lobo519 »

hmm - well there is defiantly no QOS policies on any of our equipment.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Network Bottleneck

Post by lobo519 »

Would enabling the "Use multiple TCP/IP connections per job" setting help if indeed there was some QOS policy? Do I need to create a throttling rule for this setting to be used?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Network Bottleneck

Post by Gostev »

Actually, this checkbox is enabled by default - are you saying you have it disabled right now?
No, this functionality is not dependent on throttling rules presence.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Network Bottleneck

Post by lobo519 »

Yes its disabled at the moment - Perhaps I should turn it back on! :oops:


Maybe a UI change is in-order for that check box..
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Network Bottleneck

Post by Gostev »

There is a good reason why this feature was added, and is enabled by default ;)
Enabling it back will most definitely help with your WAN link utilization.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Network Bottleneck

Post by lobo519 » 1 person likes this post

:D :D Its only about 5 times faster!

Thanks!!
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Network Bottleneck

Post by Gostev »

Default is 5 TCP/IP connection per job, thus the number you are seeing. This also proves that you have some sort of QoS that limits each connection to the same amount of bandwidth. We actually have the same in our own environment on the WAN link between US and EMEA sites... no matter if I open 1 or 5 FTP sessions for upload, they all see to have the same (and slow) speed.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Network Bottleneck

Post by lobo519 »

Can you add more connections per job?!??!

I'm going to need to track down what is limiting per connection (not that its really a huge deal). It must be either on our providers end or a default setting in one of our ASAs - I set everything up! :shock:
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Network Bottleneck

Post by Gostev »

Yes, by adding this registry value in the B&R registry key on your backup management server:

Code: Select all

DownloadStreamsNumber
DWORD
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Network Bottleneck

Post by lobo519 »

Excellent!

Any down side to changing this value? Do I need to restart any services/server for this to take effect?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Network Bottleneck

Post by Gostev »

No need to restart, every newly starting job will take this setting into account.

Potential downsides:
1. Flooding your WAN link with backup traffic affecting anything else using this link.
2. Running into max connections amount. Apparently, some networking equipment seem to supports only limited amount of connections when there is VPN in play. Happens rarely indeed, but I personally got affected by this issue during our SE training last fall.
3. Some of those connections timing out and failing the job because of too many connection on too little bandwidth.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Network Bottleneck

Post by lobo519 »

is that value hex or dec?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Network Bottleneck

Post by Gostev »

The value is REG_DWORD. You can enter it as HEX or DEC, just tell the Registry Editor what is it that you entered by selecting the corresponding radio button before clicking OK ;)
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Network Bottleneck

Post by lobo519 »

that what I figured but wanted to be sure. Thanks for your help!!
joemagjr
Influencer
Posts: 14
Liked: never
Joined: Jul 13, 2009 5:58 pm
Full Name: joemagjr
Contact:

Re: Network Bottleneck

Post by joemagjr »

I also have some network bottleneck issues while running replication jobs. I also notice when running any of my replication jobs my network latency is increased to sometimes almost to 500ms and continues for 8-10 pings and then back down to 20ms from HQ to the DR site. We are trying to figure out what is causing this, but frankley this only occurs when running replication. Sometimes the jobs finish and sometimes they don't.


We are not using the veeam traffic throttle, but do use a network traffic throttle appliance and WAN optimiation (blue coat)that allows me to designate a certain amount of the 3mg pipe to veeam (I am also working with them and are finding nothing). I have also run these jobs without any throttle or optimization in the mix and let veeam saturate the 3meg pipe at night and I get the same results, sometimes the jobs finish and sometimes they don't. How else can I trouble shoot this network issue with veeam?
here is the config and some numbers;

At HQ site (ESXi5 host)
A VM B&R Win 2008 R2 on ESXi5 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's) also acts as repository for meta data
A VM proxy Win 2008 R2 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's)

At DR Site (also ESXi5 host)
A VM proxy Win 2008 R2 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's)

Target 0% Proxy 50% Network 99% Source 0%
Optimal compression with WAN target (have tried other compression settings same results)
Multiple TCP connection setting enabled (tried to disable (same results)
Proxy transport mode settings are set to SAN mode not automatic and datastore selection is set to automatic
real time stat show : Source proxy [san;ndb] and DR proxy [nbd]

this is the same error every single time no matter what job or data store it is replicating to;

Code: Select all

Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [12044]. Failed to replicate content of the disk [vddk://<vddkConnSpec><viConn name="VCENTER SERVER" authdPort="443" vicPort="443" /><vmxPath vmRef="vm-71" datacenterRef="datacenter-46" datacenterInventoryPath="DATACENTER NAME" snapshotRef="snapshot-5181" datastoreName="DATASTORE NAME" path="VMNAME/VMNAME.vmx" /><vmdkPath datastoreName="DATASTORENAME" path="VMNAME/VMNAME.vmdk" /><t
I am at a loss and yes I have a support ticket open and that is now getting esculated but I thought that i would reach out to the forum, since I see so much out here that could be related.

ALL HELP IS APPRECIATED
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Network Bottleneck

Post by Vitaliy S. »

I guess the easiest way to nail down this network issue is to try to run replication job without using WAN optimization tool (if possible) and compare the results. If it runs fine, then you will need to take a look at WAN optimization tool config. Thanks!
CharlesGillanders
Lurker
Posts: 2
Liked: never
Joined: May 06, 2010 8:23 am
Full Name: Charles Gillanders

Re: Network Bottleneck

Post by CharlesGillanders »

Hi,

Might be stating the obvious but it's entirely possible you don't have a problem with your network. There's always a performance limit to a single TCP connection based on latency (something to do with the sliding window algorithm I forget exactly it's been a while since I needed to know this stuff). TCP connections through a VPN will have more latency than most and when there's a WAN involved as well then it may be that you've just discovered the practical limits of a single connection given your environment.

Multiple connections are a good way to resolve this issue.

Charles
CaptainFred
Enthusiast
Posts: 88
Liked: 2 times
Joined: Jul 31, 2013 12:05 pm
Full Name: Si
Contact:

Re: Network Bottleneck

Post by CaptainFred »

I've added this to HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication on my backup server running the replication job, should it also be added to the target proxy?

Does it take affect immediately? Or does the job need to be stopped/started or services restarted? I'm not seeing any improvement. Getting about 1MB/s!

I do have a VPN to the DR site across a 50Mbit internet link. Pings to the DR site when nothing is going on average at 14ms, thats not bad is it?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Network Bottleneck

Post by foggy »

CaptainFred wrote:should it also be added to the target proxy?
No.
CaptainFred wrote:Does it take affect immediately? Or does the job need to be stopped/started or services restarted? I'm not seeing any improvement. Getting about 1MB/s!
It should take effect on every newly starting job, no need to restart services, as far as I know.
CaptainFred
Enthusiast
Posts: 88
Liked: 2 times
Joined: Jul 31, 2013 12:05 pm
Full Name: Si
Contact:

Re: Network Bottleneck

Post by CaptainFred »

Ok I changed it from 7 to 15 and that's helped, not maxed out the link but improved from 1MB/s to 4MB/s! Is there any risk or issue with having it set that high?

Btw, does it use ports 2500 and 2502 for the connections to the target proxy?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Network Bottleneck

Post by foggy »

CaptainFred wrote:Ok I changed it from 7 to 15 and that's helped, not maxed out the link but improved from 1MB/s to 4MB/s! Is there any risk or issue with having it set that high?
Probably running into max connections amount.
CaptainFred wrote:Btw, does it use ports 2500 and 2502 for the connections to the target proxy?
Yes, those ports are used by the proxy service.
CaptainFred
Enthusiast
Posts: 88
Liked: 2 times
Joined: Jul 31, 2013 12:05 pm
Full Name: Si
Contact:

Re: Network Bottleneck

Post by CaptainFred »

Ok but is it safe to leave it at 15?
CaptainFred
Enthusiast
Posts: 88
Liked: 2 times
Joined: Jul 31, 2013 12:05 pm
Full Name: Si
Contact:

Re: Network Bottleneck

Post by CaptainFred »

And is there a "max connections amount" ?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Network Bottleneck

Post by Gostev »

I know of a few customers using more than 20, so 15 is certainly "safe".
I don't believe there is a limit on max connection count.
Post Reply

Who is online

Users browsing this forum: No registered users and 97 guests