Comprehensive data protection for all workloads
Post Reply
davidb1234
Expert
Posts: 162
Liked: 15 times
Joined: Nov 15, 2011 8:47 pm
Full Name: David Borden
Contact:

Why network bottleneck in SAN mode?

Post by davidb1234 »

I have a new Veeam 7 implementation. It is a physical server with direct attached SAN access backup repository as well as the vmfs datastores. When we run jobs we are getting great processing rate of 600MB/s or more during a full active for the job however the bottleneck shows NETWORK.

Why do we have a bottleneck of NETWORK like this when everything is backed up directly from and to a SAN?(confirmed from the job report).
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Why network bottleneck in SAN mode?

Post by tsightler »

Even when run on a single machine Veeam still start separate proxy and repository agents and these agents still communicate to each other via TCP, although the traffic should stay local on the machine and not likely be a true bottleneck. I have seen a handful of cases where a system that is multi-homed for some reason sends traffic to itself via the network (for example out one interface and into the other) so you'll want to make sure this is not occurring in your case.

You may also want to verify that RSS is enabled so that TCP traffic can be serviced by all CPUs. This really shouldn't make a difference since all connections are local, but I've run into a few corner cases where it still seemed to make a significant difference.

http://boerlowie.wordpress.com/2011/03/ ... -machines/

In the end though, something has to be the "bottleneck" and the algorithm simply chooses the point in the chain that it spent the most time working on. Assuming fast disk and target and plenty of CPU even the internal "network" could theoretically be the bottleneck. What specific percentages did you see?
jakeW
Novice
Posts: 6
Liked: never
Joined: Oct 15, 2014 9:46 am
Full Name: Jake W
Contact:

[MERGED] Veeam sending traffic to itself via the network

Post by jakeW »

I recently migrated an instance of Veeam B&R to a new server using the standard procedure. (I.e. Install Veeam from scratch, and then import the database and backups).

After running some test jobs on the new server I noticed that backup jobs were processing significantly slower than they used too. They used to run at >140MB/s and Veeam listed the bottleneck as "source". Now they are processing at <90MB/s and Veeam lists the bottleneck as "network".

I immediately knew something was wrong because the network should not be used for these backups. The server has direct SAN access to the target VMs (I have confirmed that the proxy mode is indeed listed as SAN) and the backup repository is local storage. There are no remote proxies involved.

Anyway, I had a look at the windows resource monitor and sure enough, the single gigabit Ethernet link was maxed out. When I inspected the network processes I could see that veeamagent.exe appeared to be sending data to itself via the network stack. (The remote IP of the connection was the Veeam server). I ran wireshark to confirm this and sure enough, there wasn't actually any network traffic traversing the network interface, it's just routing back to itself.

Does anyone have any idea what might be going on here? It's almost as if the server thinks that it needs to send the traffic to a remote proxy.

Cheers
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Why network bottleneck in SAN mode?

Post by foggy »

Jake, please review Tom's suggestions above. Could you also try to create another local repository just to check whether the behavior is similar (and this is not the consequence of configuration restore). If nothing helps, please contact technical support for closer investigation.
jakeW
Novice
Posts: 6
Liked: never
Joined: Oct 15, 2014 9:46 am
Full Name: Jake W
Contact:

Re: Why network bottleneck in SAN mode?

Post by jakeW »

Hi foggy, thanks for the reply.

Turns out there was in-fact a NIC Team on the server - I tried disabling the team, uninstalling the Broadcom teaming drivers, as well as physically removing the second Ethernet link. No luck.
I also tried creating another local repository on the C:\ of the server, but experienced the same behavior. The server is still bottle-necking at the network.

I have logged a ticket with technical support - I'll report back here when I have a resolution.

Cheers
jakeW
Novice
Posts: 6
Liked: never
Joined: Oct 15, 2014 9:46 am
Full Name: Jake W
Contact:

Re: Why network bottleneck in SAN mode?

Post by jakeW »

No luck with technical support so far. The tech I'm speaking with seems to think that this is normal behavior for Direct SAN access. And has told me that the high network traffic is just data being moved between the SAN and the local repository.

This is in spite of me showing him that the bottleneck never used to be the network. And the storage and network topology has not changed.

Historically the job load breakdown was similar to this:

Source 60% > Proxy 30% > Network 0% > Target 50%

But now the breakdown is consistently this:

Source 10% > Proxy 45% > Network 99% > Target 5%
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Why network bottleneck in SAN mode?

Post by foggy »

jakeW wrote:And has told me that the high network traffic is just data being moved between the SAN and the local repository.
There are three processing stages between the SAN and the repository, and the bottleneck stats for your last job clearly shows that the bottleneck is between the proxy and repository, so SAN is completely out of picture here. What is your support case number?
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

[MERGED] Veeam network questions

Post by b.vanhaastrecht »

Hi,

We've got a setup were a Veeam proxy & repository server is multihomed, two 10GbE NIC's for iSCSI, and two 10GbE teamed for LAN access. In our quest to troubleshooting overal sluggisch network thruput on this server we notice the Veeam services are communicating on one of the iSCSI NIC's. Is this normal behaviour, shouldn't those sevices run on the loopback interface?

Another thing we noticed is when running a local attached tape backup from local attached DAS repository, the Resource Monitor shows network traffic flowing from VeeamAgent.exe sending to another instance of VeeamAgent.exe which receives the exact flow. Is this normal behaviour? Disks are local, so is the tape unit.

Again, we are troubleshooting overal sluggish network thruput on this server, so please excuse me if I'm hunting ghosts.

Kind Regards,
Bastiaan
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Why network bottleneck in SAN mode?

Post by Gostev »

Bastiaan, you will be able to enable loopback interface for this scenario in v8. Because this capability was added last minute in the release cycle, it is disabled by default and can only be enable via a registry value. Once v8 is out and you deploy it, let me know and I will tell you how to enable this. Thanks!
Peejay62
Expert
Posts: 235
Liked: 37 times
Joined: Aug 06, 2013 10:40 am
Full Name: Peter Jansen
Contact:

Re: Why network bottleneck in SAN mode?

Post by Peejay62 »

Anton,

I will let you know too as soon as i have v8 deployed. I experience the same high localhost traffic on the interface when doing SAN mode. I already spoke to you on this matter at Vmworld Barcelona, and you told me about the registry setting and to remind you on it. So here's the reminder ;)

thanks, Peter
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Why network bottleneck in SAN mode?

Post by dellock6 »

And just to give you all a pre-feedback, on some tests we did on FC-based veeam servers (both source and target, so complete lan-less scenario), the speed was more then double once the key was enabled ;)
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
ICarter
Novice
Posts: 9
Liked: never
Joined: Apr 15, 2013 3:37 pm
Full Name: Ian Carter
Contact:

[MERGED] Curiosity on how Veeam moves data

Post by ICarter »

We run our proxy and repository roles on the same (we have a couple but they are identical) box. The proxy uses direct san access to backup VM data and sends it to directly attached disk. We've recently starting test out the tape support and have added the tape server role to one of our proxy repository boxes. This is connected directly via FC to the tape library and drives.

What we have noticed is that the throughput is capped at 1Gb, whether this is during a backup job, or more obviously, when copying local repository files off to tape. Given that everything is 4Gb FC connected this is a bit of a pain. Using third party backup software from the same server we can push the maximum LTO5 speed of 140MB/s, but see less than 100MB/s when using Veeam and I suspect network is the limiting factor (which is reported by tape backup job as the bottleneck).

Watching the performance stats I can also see the network levelling out at 1Gb as one Veeam process has around 1/2Gb being transmitted to another process (on the same box) which is receiving at the same rate. I can understand why Veeam works this way as it expects everything to be on different boxes. We have 10Gb planned for the future which will solve the issue, but I wondered if anyone had noticed the issue and had any workarounds? I suspect teaming the NICs is probably a good option (especially as Veeam doesn't use the loopback adapter). It just feels a bit ineffecient :)

Cheers, Ian
emachabert
Veeam Vanguard
Posts: 388
Liked: 168 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Curiosity on how Veeam moves data

Post by emachabert » 1 person likes this post

Hi

if you use the search function of the forum you should find another topic dealing with that behaviour.
:wink:
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
He-Man
Novice
Posts: 3
Liked: never
Joined: Jun 20, 2011 2:14 pm
Full Name: Thomas Midthjell
Contact:

Re: Bottleneck: Network in Direct SAN mode

Post by He-Man »

Hi. I experience the same issue. Can you send me the registry key?
He-Man
Novice
Posts: 3
Liked: never
Joined: Jun 20, 2011 2:14 pm
Full Name: Thomas Midthjell
Contact:

Re: Bottleneck: Network in Direct SAN mode

Post by He-Man »

Do you know the registry key they are talking about? The network utilization is near 100% when using Direct san mode with fiber channel.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Why network bottleneck in SAN mode?

Post by Vitaliy S. »

What is your destination target (repository)?
lpl
Lurker
Posts: 1
Liked: never
Joined: Dec 13, 2014 8:30 pm
Full Name: Lorenzo Platteau
Contact:

Re: Why network bottleneck in SAN mode?

Post by lpl »

Same overhere. Backup in SAN mode (iscsi SAN), to local disks... and network is the current bottleneck? Using 7.0.0.871. Don't know if it's related to patch 4 I installed last week, or to the latest HP Proliant Service Pack I installed on the backupserver, but currently doing an active full backup at the ridiculously slow speed of 14mb/s... this will take a few days I guess.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Why network bottleneck in SAN mode?

Post by Vitaliy S. »

Do you have a support case opened on this? What stage in the backup job stats takes the most part? Data transfer or something else?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Why network bottleneck in SAN mode?

Post by foggy »

Registry key improving direct communication of data movers running on the same server is included with the recently released Patch 1 for 8.0 (note that this functionality is still experimental).
Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 105 guests