-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Network Bottleneck
I am running a backup job over the WAN (VPN). I can clearly see the available bandwidth and there is plenty available that is not being used - yet my Job shows the bottleneck as the Network.
Expected behavior?
My replication jobs usually have no problem filling the pipe.
I am not using any traffic throttling.
Expected behavior?
My replication jobs usually have no problem filling the pipe.
I am not using any traffic throttling.
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Network Bottleneck
What are the whole bottleneck stats numbers for this job?
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network Bottleneck
Also, are you in patch #3, because it has enhancements around network load calculations.
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: Network Bottleneck
I am on patch 3. I'll post back when the job is finished. Is running a full and I am in hour 82.
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: Network Bottleneck
4/10/2012 11:32:27 PM :: Load: Source 0% > Proxy 11% > Network 98% > Target 0%
This job ran for 96 hours before I just cancelled it. I re-ran it after canceling it and it seems to be running better. It was a full backup
This job ran for 96 hours before I just cancelled it. I re-ran it after canceling it and it seems to be running better. It was a full backup
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network Bottleneck
Thank you - seeing these numbers makes troubleshooting so easy! Based on this statistics, my only guess is that there is some QoS mechanism in place on the networking equipment that limits bandwidth per connection. To verify this, try disabling multiple TCP/IP connections option in Veeam settings under Traffic Throttling settings, and see if this reduces the backup performance. I would recommend creating a very small VM for testing this, and trying full backup with and without multiple connections option enabled. And post the results (time to perform full backup) here. Thanks!
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: Network Bottleneck
hmm - well there is defiantly no QOS policies on any of our equipment.
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: Network Bottleneck
Would enabling the "Use multiple TCP/IP connections per job" setting help if indeed there was some QOS policy? Do I need to create a throttling rule for this setting to be used?
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network Bottleneck
Actually, this checkbox is enabled by default - are you saying you have it disabled right now?
No, this functionality is not dependent on throttling rules presence.
No, this functionality is not dependent on throttling rules presence.
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: Network Bottleneck
Yes its disabled at the moment - Perhaps I should turn it back on!
Maybe a UI change is in-order for that check box..
Maybe a UI change is in-order for that check box..
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network Bottleneck
There is a good reason why this feature was added, and is enabled by default
Enabling it back will most definitely help with your WAN link utilization.
Enabling it back will most definitely help with your WAN link utilization.
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: Network Bottleneck
Its only about 5 times faster!
Thanks!!
Thanks!!
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network Bottleneck
Default is 5 TCP/IP connection per job, thus the number you are seeing. This also proves that you have some sort of QoS that limits each connection to the same amount of bandwidth. We actually have the same in our own environment on the WAN link between US and EMEA sites... no matter if I open 1 or 5 FTP sessions for upload, they all see to have the same (and slow) speed.
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: Network Bottleneck
Can you add more connections per job?!??!
I'm going to need to track down what is limiting per connection (not that its really a huge deal). It must be either on our providers end or a default setting in one of our ASAs - I set everything up!
I'm going to need to track down what is limiting per connection (not that its really a huge deal). It must be either on our providers end or a default setting in one of our ASAs - I set everything up!
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network Bottleneck
Yes, by adding this registry value in the B&R registry key on your backup management server:
Code: Select all
DownloadStreamsNumber
DWORD
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: Network Bottleneck
Excellent!
Any down side to changing this value? Do I need to restart any services/server for this to take effect?
Any down side to changing this value? Do I need to restart any services/server for this to take effect?
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network Bottleneck
No need to restart, every newly starting job will take this setting into account.
Potential downsides:
1. Flooding your WAN link with backup traffic affecting anything else using this link.
2. Running into max connections amount. Apparently, some networking equipment seem to supports only limited amount of connections when there is VPN in play. Happens rarely indeed, but I personally got affected by this issue during our SE training last fall.
3. Some of those connections timing out and failing the job because of too many connection on too little bandwidth.
Potential downsides:
1. Flooding your WAN link with backup traffic affecting anything else using this link.
2. Running into max connections amount. Apparently, some networking equipment seem to supports only limited amount of connections when there is VPN in play. Happens rarely indeed, but I personally got affected by this issue during our SE training last fall.
3. Some of those connections timing out and failing the job because of too many connection on too little bandwidth.
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: Network Bottleneck
is that value hex or dec?
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network Bottleneck
The value is REG_DWORD. You can enter it as HEX or DEC, just tell the Registry Editor what is it that you entered by selecting the corresponding radio button before clicking OK
-
- Veteran
- Posts: 315
- Liked: 38 times
- Joined: Sep 29, 2010 3:37 pm
- Contact:
Re: Network Bottleneck
that what I figured but wanted to be sure. Thanks for your help!!
-
- Influencer
- Posts: 14
- Liked: never
- Joined: Jul 13, 2009 5:58 pm
- Full Name: joemagjr
- Contact:
Re: Network Bottleneck
I also have some network bottleneck issues while running replication jobs. I also notice when running any of my replication jobs my network latency is increased to sometimes almost to 500ms and continues for 8-10 pings and then back down to 20ms from HQ to the DR site. We are trying to figure out what is causing this, but frankley this only occurs when running replication. Sometimes the jobs finish and sometimes they don't.
We are not using the veeam traffic throttle, but do use a network traffic throttle appliance and WAN optimiation (blue coat)that allows me to designate a certain amount of the 3mg pipe to veeam (I am also working with them and are finding nothing). I have also run these jobs without any throttle or optimization in the mix and let veeam saturate the 3meg pipe at night and I get the same results, sometimes the jobs finish and sometimes they don't. How else can I trouble shoot this network issue with veeam?
here is the config and some numbers;
At HQ site (ESXi5 host)
A VM B&R Win 2008 R2 on ESXi5 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's) also acts as repository for meta data
A VM proxy Win 2008 R2 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's)
At DR Site (also ESXi5 host)
A VM proxy Win 2008 R2 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's)
Target 0% Proxy 50% Network 99% Source 0%
Optimal compression with WAN target (have tried other compression settings same results)
Multiple TCP connection setting enabled (tried to disable (same results)
Proxy transport mode settings are set to SAN mode not automatic and datastore selection is set to automatic
real time stat show : Source proxy [san;ndb] and DR proxy [nbd]
this is the same error every single time no matter what job or data store it is replicating to;
I am at a loss and yes I have a support ticket open and that is now getting esculated but I thought that i would reach out to the forum, since I see so much out here that could be related.
ALL HELP IS APPRECIATED
We are not using the veeam traffic throttle, but do use a network traffic throttle appliance and WAN optimiation (blue coat)that allows me to designate a certain amount of the 3mg pipe to veeam (I am also working with them and are finding nothing). I have also run these jobs without any throttle or optimization in the mix and let veeam saturate the 3meg pipe at night and I get the same results, sometimes the jobs finish and sometimes they don't. How else can I trouble shoot this network issue with veeam?
here is the config and some numbers;
At HQ site (ESXi5 host)
A VM B&R Win 2008 R2 on ESXi5 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's) also acts as repository for meta data
A VM proxy Win 2008 R2 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's)
At DR Site (also ESXi5 host)
A VM proxy Win 2008 R2 at HQ - software iscsi connection read permissions to both HQ SAN and DR SAN (hp lefthand p4500's)
Target 0% Proxy 50% Network 99% Source 0%
Optimal compression with WAN target (have tried other compression settings same results)
Multiple TCP connection setting enabled (tried to disable (same results)
Proxy transport mode settings are set to SAN mode not automatic and datastore selection is set to automatic
real time stat show : Source proxy [san;ndb] and DR proxy [nbd]
this is the same error every single time no matter what job or data store it is replicating to;
Code: Select all
Error: Client error: End of file Unable to retrieve next block transmission command. Number of already processed blocks: [12044]. Failed to replicate content of the disk [vddk://<vddkConnSpec><viConn name="VCENTER SERVER" authdPort="443" vicPort="443" /><vmxPath vmRef="vm-71" datacenterRef="datacenter-46" datacenterInventoryPath="DATACENTER NAME" snapshotRef="snapshot-5181" datastoreName="DATASTORE NAME" path="VMNAME/VMNAME.vmx" /><vmdkPath datastoreName="DATASTORENAME" path="VMNAME/VMNAME.vmdk" /><t
ALL HELP IS APPRECIATED
-
- VP, Product Management
- Posts: 27371
- Liked: 2799 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Network Bottleneck
I guess the easiest way to nail down this network issue is to try to run replication job without using WAN optimization tool (if possible) and compare the results. If it runs fine, then you will need to take a look at WAN optimization tool config. Thanks!
-
- Lurker
- Posts: 2
- Liked: never
- Joined: May 06, 2010 8:23 am
- Full Name: Charles Gillanders
Re: Network Bottleneck
Hi,
Might be stating the obvious but it's entirely possible you don't have a problem with your network. There's always a performance limit to a single TCP connection based on latency (something to do with the sliding window algorithm I forget exactly it's been a while since I needed to know this stuff). TCP connections through a VPN will have more latency than most and when there's a WAN involved as well then it may be that you've just discovered the practical limits of a single connection given your environment.
Multiple connections are a good way to resolve this issue.
Charles
Might be stating the obvious but it's entirely possible you don't have a problem with your network. There's always a performance limit to a single TCP connection based on latency (something to do with the sliding window algorithm I forget exactly it's been a while since I needed to know this stuff). TCP connections through a VPN will have more latency than most and when there's a WAN involved as well then it may be that you've just discovered the practical limits of a single connection given your environment.
Multiple connections are a good way to resolve this issue.
Charles
-
- Enthusiast
- Posts: 88
- Liked: 2 times
- Joined: Jul 31, 2013 12:05 pm
- Full Name: Si
- Contact:
Re: Network Bottleneck
I've added this to HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication on my backup server running the replication job, should it also be added to the target proxy?
Does it take affect immediately? Or does the job need to be stopped/started or services restarted? I'm not seeing any improvement. Getting about 1MB/s!
I do have a VPN to the DR site across a 50Mbit internet link. Pings to the DR site when nothing is going on average at 14ms, thats not bad is it?
Does it take affect immediately? Or does the job need to be stopped/started or services restarted? I'm not seeing any improvement. Getting about 1MB/s!
I do have a VPN to the DR site across a 50Mbit internet link. Pings to the DR site when nothing is going on average at 14ms, thats not bad is it?
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Network Bottleneck
No.CaptainFred wrote:should it also be added to the target proxy?
It should take effect on every newly starting job, no need to restart services, as far as I know.CaptainFred wrote:Does it take affect immediately? Or does the job need to be stopped/started or services restarted? I'm not seeing any improvement. Getting about 1MB/s!
-
- Enthusiast
- Posts: 88
- Liked: 2 times
- Joined: Jul 31, 2013 12:05 pm
- Full Name: Si
- Contact:
Re: Network Bottleneck
Ok I changed it from 7 to 15 and that's helped, not maxed out the link but improved from 1MB/s to 4MB/s! Is there any risk or issue with having it set that high?
Btw, does it use ports 2500 and 2502 for the connections to the target proxy?
Btw, does it use ports 2500 and 2502 for the connections to the target proxy?
-
- Veeam Software
- Posts: 21138
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Network Bottleneck
Probably running into max connections amount.CaptainFred wrote:Ok I changed it from 7 to 15 and that's helped, not maxed out the link but improved from 1MB/s to 4MB/s! Is there any risk or issue with having it set that high?
Yes, those ports are used by the proxy service.CaptainFred wrote:Btw, does it use ports 2500 and 2502 for the connections to the target proxy?
-
- Enthusiast
- Posts: 88
- Liked: 2 times
- Joined: Jul 31, 2013 12:05 pm
- Full Name: Si
- Contact:
Re: Network Bottleneck
Ok but is it safe to leave it at 15?
-
- Enthusiast
- Posts: 88
- Liked: 2 times
- Joined: Jul 31, 2013 12:05 pm
- Full Name: Si
- Contact:
Re: Network Bottleneck
And is there a "max connections amount" ?
-
- Chief Product Officer
- Posts: 31804
- Liked: 7298 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Network Bottleneck
I know of a few customers using more than 20, so 15 is certainly "safe".
I don't believe there is a limit on max connection count.
I don't believe there is a limit on max connection count.
Who is online
Users browsing this forum: No registered users and 58 guests