Why network bottleneck in SAN mode?

[davidb1234]

I have a new Veeam 7 implementation. It is a physical server with direct attached SAN access backup repository as well as the vmfs datastores. When we run jobs we are getting great processing rate of 600MB/s or more during a full active for the job however the bottleneck shows NETWORK.

Why do we have a bottleneck of NETWORK like this when everything is backed up directly from and to a SAN?(confirmed from the job report).

[tsightler]

Even when run on a single machine Veeam still start separate proxy and repository agents and these agents still communicate to each other via TCP, although the traffic should stay local on the machine and not likely be a true bottleneck. I have seen a handful of cases where a system that is multi-homed for some reason sends traffic to itself via the network (for example out one interface and into the other) so you'll want to make sure this is not occurring in your case.

You may also want to verify that RSS is enabled so that TCP traffic can be serviced by all CPUs. This really shouldn't make a difference since all connections are local, but I've run into a few corner cases where it still seemed to make a significant difference.

http://boerlowie.wordpress.com/2011/03/ ... -machines/

In the end though, something has to be the "bottleneck" and the algorithm simply chooses the point in the chain that it spent the most time working on. Assuming fast disk and target and plenty of CPU even the internal "network" could theoretically be the bottleneck. What specific percentages did you see?

[jakeW]

I recently migrated an instance of Veeam B&R to a new server using the standard procedure. (I.e. Install Veeam from scratch, and then import the database and backups).

After running some test jobs on the new server I noticed that backup jobs were processing significantly slower than they used too. They used to run at >140MB/s and Veeam listed the bottleneck as "source". Now they are processing at <90MB/s and Veeam lists the bottleneck as "network".

I immediately knew something was wrong because the network should not be used for these backups. The server has direct SAN access to the target VMs (I have confirmed that the proxy mode is indeed listed as SAN) and the backup repository is local storage. There are no remote proxies involved.

Anyway, I had a look at the windows resource monitor and sure enough, the single gigabit Ethernet link was maxed out. When I inspected the network processes I could see that veeamagent.exe appeared to be sending data to itself via the network stack. (The remote IP of the connection was the Veeam server). I ran wireshark to confirm this and sure enough, there wasn't actually any network traffic traversing the network interface, it's just routing back to itself.

Does anyone have any idea what might be going on here? It's almost as if the server thinks that it needs to send the traffic to a remote proxy.

Cheers

[foggy]

Jake, please review Tom's suggestions above. Could you also try to create another local repository just to check whether the behavior is similar (and this is not the consequence of configuration restore). If nothing helps, please contact technical support for closer investigation.

[jakeW]

Hi foggy, thanks for the reply.

Turns out there was in-fact a NIC Team on the server - I tried disabling the team, uninstalling the Broadcom teaming drivers, as well as physically removing the second Ethernet link. No luck.
I also tried creating another local repository on the C:\ of the server, but experienced the same behavior. The server is still bottle-necking at the network.

I have logged a ticket with technical support - I'll report back here when I have a resolution.

Cheers

[jakeW]

No luck with technical support so far. The tech I'm speaking with seems to think that this is normal behavior for Direct SAN access. And has told me that the high network traffic is just data being moved between the SAN and the local repository.

This is in spite of me showing him that the bottleneck never used to be the network. And the storage and network topology has not changed.

Historically the job load breakdown was similar to this:

Source 60% > Proxy 30% > Network 0% > Target 50%

But now the breakdown is consistently this:

Source 10% > Proxy 45% > Network 99% > Target 5%

[foggy]

And has told me that the high network traffic is just data being moved between the SAN and the local repository.

There are three processing stages between the SAN and the repository, and the bottleneck stats for your last job clearly shows that the bottleneck is between the proxy and repository, so SAN is completely out of picture here. What is your support case number?

[b.vanhaastrecht]

Hi,

We've got a setup were a Veeam proxy & repository server is multihomed, two 10GbE NIC's for iSCSI, and two 10GbE teamed for LAN access. In our quest to troubleshooting overal sluggisch network thruput on this server we notice the Veeam services are communicating on one of the iSCSI NIC's. Is this normal behaviour, shouldn't those sevices run on the loopback interface?

Another thing we noticed is when running a local attached tape backup from local attached DAS repository, the Resource Monitor shows network traffic flowing from VeeamAgent.exe sending to another instance of VeeamAgent.exe which receives the exact flow. Is this normal behaviour? Disks are local, so is the tape unit.

Again, we are troubleshooting overal sluggish network thruput on this server, so please excuse me if I'm hunting ghosts.

Kind Regards,
Bastiaan

[Gostev]

Bastiaan, you will be able to enable loopback interface for this scenario in v8. Because this capability was added last minute in the release cycle, it is disabled by default and can only be enable via a registry value. Once v8 is out and you deploy it, let me know and I will tell you how to enable this. Thanks!

[Peejay62]

Anton,

I will let you know too as soon as i have v8 deployed. I experience the same high localhost traffic on the interface when doing SAN mode. I already spoke to you on this matter at Vmworld Barcelona, and you told me about the registry setting and to remind you on it. So here's the reminder

thanks, Peter

[dellock6]

And just to give you all a pre-feedback, on some tests we did on FC-based veeam servers (both source and target, so complete lan-less scenario), the speed was more then double once the key was enabled

[ICarter]

We run our proxy and repository roles on the same (we have a couple but they are identical) box. The proxy uses direct san access to backup VM data and sends it to directly attached disk. We've recently starting test out the tape support and have added the tape server role to one of our proxy repository boxes. This is connected directly via FC to the tape library and drives.

What we have noticed is that the throughput is capped at 1Gb, whether this is during a backup job, or more obviously, when copying local repository files off to tape. Given that everything is 4Gb FC connected this is a bit of a pain. Using third party backup software from the same server we can push the maximum LTO5 speed of 140MB/s, but see less than 100MB/s when using Veeam and I suspect network is the limiting factor (which is reported by tape backup job as the bottleneck).

Watching the performance stats I can also see the network levelling out at 1Gb as one Veeam process has around 1/2Gb being transmitted to another process (on the same box) which is receiving at the same rate. I can understand why Veeam works this way as it expects everything to be on different boxes. We have 10Gb planned for the future which will solve the issue, but I wondered if anyone had noticed the issue and had any workarounds? I suspect teaming the NICs is probably a good option (especially as Veeam doesn't use the loopback adapter). It just feels a bit ineffecient

Cheers, Ian

[emachabert]

Hi

if you use the search function of the forum you should find another topic dealing with that behaviour.

[He-Man]

Hi. I experience the same issue. Can you send me the registry key?

[He-Man]

Do you know the registry key they are talking about? The network utilization is near 100% when using Direct san mode with fiber channel.

[Vitaliy S.]

What is your destination target (repository)?

[lpl]

Same overhere. Backup in SAN mode (iscsi SAN), to local disks... and network is the current bottleneck? Using 7.0.0.871. Don't know if it's related to patch 4 I installed last week, or to the latest HP Proliant Service Pack I installed on the backupserver, but currently doing an active full backup at the ridiculously slow speed of 14mb/s... this will take a few days I guess.

[Vitaliy S.]

Do you have a support case opened on this? What stage in the backup job stats takes the most part? Data transfer or something else?

[foggy]

Registry key improving direct communication of data movers running on the same server is included with the recently released Patch 1 for 8.0 (note that this functionality is still experimental).