Network Bottleneck

VMware specific discussions

Re: Network Bottleneck

Veeam Logoby lobo519 » Thu Apr 12, 2012 8:33 pm

Excellent!

Any down side to changing this value? Do I need to restart any services/server for this to take effect?
lobo519
Expert
 
Posts: 292
Liked: 34 times
Joined: Wed Sep 29, 2010 3:37 pm

Re: Network Bottleneck

Veeam Logoby Gostev » Thu Apr 12, 2012 8:46 pm

No need to restart, every newly starting job will take this setting into account.

Potential downsides:
1. Flooding your WAN link with backup traffic affecting anything else using this link.
2. Running into max connections amount. Apparently, some networking equipment seem to supports only limited amount of connections when there is VPN in play. Happens rarely indeed, but I personally got affected by this issue during our SE training last fall.
3. Some of those connections timing out and failing the job because of too many connection on too little bandwidth.
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Network Bottleneck

Veeam Logoby lobo519 » Thu Apr 12, 2012 8:52 pm

is that value hex or dec?
lobo519
Expert
 
Posts: 292
Liked: 34 times
Joined: Wed Sep 29, 2010 3:37 pm

Re: Network Bottleneck

Veeam Logoby Gostev » Thu Apr 12, 2012 9:08 pm

The value is REG_DWORD. You can enter it as HEX or DEC, just tell the Registry Editor what is it that you entered by selecting the corresponding radio button before clicking OK ;)
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Network Bottleneck

Veeam Logoby lobo519 » Thu Apr 12, 2012 9:14 pm

that what I figured but wanted to be sure. Thanks for your help!!
lobo519
Expert
 
Posts: 292
Liked: 34 times
Joined: Wed Sep 29, 2010 3:37 pm

Re: Network Bottleneck

Veeam Logoby joemagjr » Fri Apr 13, 2012 3:28 pm

I also have some network bottleneck issues while running replication jobs. I also notice when running any of my replication jobs my network latency is increased to sometimes almost to 500ms and continues for 8-10 pings and then back down to 20ms from HQ to the DR site. We are trying to figure out what is causing this, but frankley this only occurs when running replication. Sometimes the jobs finish and sometimes they don't.


We are not using the veeam traffic throttle, but do use a network traffic throttle appliance and WAN optimiation (blue coat)that allows me to designate a certain amount of the 3mg pipe to veeam (I am also working with them and are finding nothing). I have also run these jobs without any throttle or optimization in the mix and let veeam saturate the 3meg pipe at night and I get the same results, sometimes the jobs finish and sometimes they don't. How else can I trouble shoot this network issue with veeam?
here is the config and some numbers;

At HQ site (ESXi5 host)
A VM B&R Win 2008 R2 on ESXi5 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's) also acts as repository for meta data
A VM proxy Win 2008 R2 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's)

At DR Site (also ESXi5 host)
A VM proxy Win 2008 R2 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's)

Target 0% Proxy 50% Network 99% Source 0%
Optimal compression with WAN target (have tried other compression settings same results)
Multiple TCP connection setting enabled (tried to disable (same results)
Proxy transport mode settings are set to SAN mode not automatic and datastore selection is set to automatic
real time stat show : Source proxy [san;ndb] and DR proxy [nbd]

this is the same error every single time no matter what job or data store it is replicating to;

Code: Select all
Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [12044]. Failed to replicate content of the disk [vddk://<vddkConnSpec><viConn name="VCENTER SERVER" authdPort="443" vicPort="443" /><vmxPath vmRef="vm-71" datacenterRef="datacenter-46" datacenterInventoryPath="DATACENTER NAME" snapshotRef="snapshot-5181" datastoreName="DATASTORE NAME" path="VMNAME/VMNAME.vmx" /><vmdkPath datastoreName="DATASTORENAME" path="VMNAME/VMNAME.vmdk" /><t


I am at a loss and yes I have a support ticket open and that is now getting esculated but I thought that i would reach out to the forum, since I see so much out here that could be related.

ALL HELP IS APPRECIATED
joemagjr
Influencer
 
Posts: 14
Liked: never
Joined: Mon Jul 13, 2009 5:58 pm
Full Name: joemagjr

Re: Network Bottleneck

Veeam Logoby Vitaliy S. » Fri Apr 13, 2012 4:30 pm

I guess the easiest way to nail down this network issue is to try to run replication job without using WAN optimization tool (if possible) and compare the results. If it runs fine, then you will need to take a look at WAN optimization tool config. Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19536
Liked: 1097 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Network Bottleneck

Veeam Logoby CharlesGillanders » Mon Apr 16, 2012 5:46 am

Hi,

Might be stating the obvious but it's entirely possible you don't have a problem with your network. There's always a performance limit to a single TCP connection based on latency (something to do with the sliding window algorithm I forget exactly it's been a while since I needed to know this stuff). TCP connections through a VPN will have more latency than most and when there's a WAN involved as well then it may be that you've just discovered the practical limits of a single connection given your environment.

Multiple connections are a good way to resolve this issue.

Charles
CharlesGillanders
Lurker
 
Posts: 2
Liked: never
Joined: Thu May 06, 2010 8:23 am
Full Name: Charles Gillanders

Re: Network Bottleneck

Veeam Logoby CaptainFred » Fri Oct 25, 2013 2:18 pm

I've added this to HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication on my backup server running the replication job, should it also be added to the target proxy?

Does it take affect immediately? Or does the job need to be stopped/started or services restarted? I'm not seeing any improvement. Getting about 1MB/s!

I do have a VPN to the DR site across a 50Mbit internet link. Pings to the DR site when nothing is going on average at 14ms, thats not bad is it?
CaptainFred
Enthusiast
 
Posts: 88
Liked: 2 times
Joined: Wed Jul 31, 2013 12:05 pm
Full Name: Si

Re: Network Bottleneck

Veeam Logoby foggy » Fri Oct 25, 2013 2:28 pm

CaptainFred wrote:should it also be added to the target proxy?

No.

CaptainFred wrote:Does it take affect immediately? Or does the job need to be stopped/started or services restarted? I'm not seeing any improvement. Getting about 1MB/s!

It should take effect on every newly starting job, no need to restart services, as far as I know.
foggy
Veeam Software
 
Posts: 14712
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Network Bottleneck

Veeam Logoby CaptainFred » Fri Oct 25, 2013 3:38 pm

Ok I changed it from 7 to 15 and that's helped, not maxed out the link but improved from 1MB/s to 4MB/s! Is there any risk or issue with having it set that high?

Btw, does it use ports 2500 and 2502 for the connections to the target proxy?
CaptainFred
Enthusiast
 
Posts: 88
Liked: 2 times
Joined: Wed Jul 31, 2013 12:05 pm
Full Name: Si

Re: Network Bottleneck

Veeam Logoby foggy » Fri Oct 25, 2013 3:46 pm

CaptainFred wrote:Ok I changed it from 7 to 15 and that's helped, not maxed out the link but improved from 1MB/s to 4MB/s! Is there any risk or issue with having it set that high?

Probably running into max connections amount.

CaptainFred wrote:Btw, does it use ports 2500 and 2502 for the connections to the target proxy?

Yes, those ports are used by the proxy service.
foggy
Veeam Software
 
Posts: 14712
Liked: 1075 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Network Bottleneck

Veeam Logoby CaptainFred » Fri Oct 25, 2013 3:58 pm

Ok but is it safe to leave it at 15?
CaptainFred
Enthusiast
 
Posts: 88
Liked: 2 times
Joined: Wed Jul 31, 2013 12:05 pm
Full Name: Si

Re: Network Bottleneck

Veeam Logoby CaptainFred » Fri Oct 25, 2013 4:58 pm

And is there a "max connections amount" ?
CaptainFred
Enthusiast
 
Posts: 88
Liked: 2 times
Joined: Wed Jul 31, 2013 12:05 pm
Full Name: Si

Re: Network Bottleneck

Veeam Logoby Gostev » Fri Oct 25, 2013 11:51 pm

I know of a few customers using more than 20, so 15 is certainly "safe".
I don't believe there is a limit on max connection count.
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: Google [Bot], rweis and 34 guests