Speed performance issues copying between VLANs

[n2it-dk]

Hi, ive discovered after and still having speed issues doing full backups of my vm´s. The hardware is Dell R610 servers with latest patched vSphere and then Dell MD3000i iSCSI storage, all in a full redundant setup connected to two PowerEdge 5424 GB switches. The switches are configured with several vlans where there service console is running in a isolated vlan and so are the storage controlles etc. In the current scenario ill be talking about two vlans 51 and 52 that are both beeing backed up from a Veeam station wich is allso running as a vm. The Veeam station is located in the vlan 51 and can easily attach to both vSphere hosts and see the storage. The backup is allso running fine, BUT SLOW for a full backup that is, when its doing the block level afterwards its running 2-300MB/s so its okay, could be better.

The servers and the MD3000i arent reaching any limits in performance, leightweight use atm, and therefore I tried investigating and found out copying a 2gb file from a vm in the vlan51 to another vm in the same vlan51 ran withing few seconds, but doing the same copying from vlan51 to another host in vlan52 went with speed at 10-12MB/s and took minutes so somehow, there seems to be issues copying across vlans and that might allso tell me, that since the iSCSI storage is in vlan8 that it could easily be the reason why the speed is slow when using Veeam.

The Veeam backup station is in vlan51 and doing the backup to the Lun´s in vlan8 (iSCSI storage).

Does anyone of you out there have any good suggestions towards this? Configuration of the switches and the MD3000i is as it should be and allso fyi the vlans are all in the same vswitch sharing same nic´s but in seperate vlans accessing the same trunk.

Looking forward to see some suggestions.

[Gostev]

What do you mean by saying that you have iSCSI storage LUN in VLAN8? My understanding is that iSCSI storage device can only be sitting on physical NIC (it is not a VM)? Sorry if my question is stupid, I am still learning how shared storage works but I don't understand your description right now.

[n2it-dk]

Its a bit sophsticated setup and can be a bit hard to actually explain. But theer are several vlans for various customer sites and then there is the shared storage, the router/firewall is then configuring the access rights to the different vlans. All vlans go to the same trunk and allso, the physical esx hosts can access the iSCSI storage where several LUN´s have been mapped out to the appropriate targets... So basically the speed between the same vlans is great, its running internally from within the same vswitch but once accessing another vlan, then its going to the router/firewall unfortunately not to the switch its not a layer3 so routing has to go out and in again and thats somehow slow... Need to find out where the bottleneck is located at... But perhaps there are other similar stories that could guide me in the right way...

[Gostev]

Oh, so you have multiple iSCSI targets on your SAN which listen on IPs corresponding to the particular VLAN. Did I get it right?

Also, just wanted to confirm my understanding that you can see the performance issue with regular file copies between VMs as well, and don't blame Veeam Backup 4.1 for slow performance (because this topic's name implies otherwise)

[n2it-dk]

Yes, your allmost there How can I edit the topic, it can be misleading!

[n2it-dk]

Its not multiple iSCSI targets per say, its more like two hostbus adapters, two for each storage processor and then from there on, several LUN´s mapped to various hosts with different rights etc. The hosts (vSphere) then defines, wich vlans and vm´s should have access to what. Its all to seperate everyone from eachother...

But, im not eliminating Veeam beeing one of the issues, just trying to dissect what can be the bottleneck and could use some feedback wich im getting, thanks

[rchew]

I suspect this might be an issue with your router / firewall. If you're running stateful inspection of the traffic, the state tables must be built and checked against your return traffic. Check your router's processor history while you're backing up between vlan 51 and 52. If you're running Cisco, use the following command...

sh proc cpu his

(short for show processor cpu history)

You can easily test this theory by removing any ACL's on the 2 VLAN's before running your next backup. If the speeds improve, chances are the router / firewall is the culprit. It is also very likely that the router is undersized even if you're not running firewall rulesets on the VLAN's. Check the specs on your router and see if it is capable of routing packets at the speeds you require. Most routers are rated for Mbps transmission not Gbps.

[pentana]

If you are doing intervlan routing with a Router then you will be limited to the speed the router can run at.
You aren't getting the same speed impairment when you are copying files from the same VLAN (i.e all traffic is isolated at the switch).
You might want to check what sort of throughput your router can do.

[n2it-dk]

Thanks for the suggestions, ill make a checkup of it and see how the proc level says when making the file copy from vlan51 to vlan52.

One thing I noticed though is that once the filecopy takes place between the two vlan´s the vmkernel wich is in another vlan8 has really bad pingtimes, so im bringing up a cmd and initiates a ping session towards the iscsi data ip´s and its traveling through the vlan8 wich is in the same subnet as the iscsi data ports. Then once making the file copy, the ping session time reaches from time<1ms to up and down between 4-9 and allso there is packet drops in the same time. Once the datacopy is stopped, the ping sessions looks fine again.

Copying a file from another host, wich is connected to the same switch but running in vlan2 and is using internal storage and isnt using the iscsi datastorage, then the ping times to vlan8 is still going up and down, and as soon as the filecopy is stopped, then the ping times looks fine.

router/pix is a 515UR running 100mb and proc looks fine not at all stressed.

Im a little bit puzzled about this.

[tsightler]

router/pix is a 515UR running 100mb and proc looks fine not at all stressed.

So there's you're performance problem. If you're routing between VLAN's using a PIX with 100Mb link, then that's going to limit your throughput between VLAN's. You either need a faster device to route traffic between VLAN's, are re-architect your backups to access the SAN directly without crossing a VLAN boundary.

[n2it-dk]

Its not dependable on the pix routing features, its supposed to access direct from within the vswif itself. Im still not getting nowhere about this issue...

Routing from vlan51 to vlan52 through the pix isnt a problem, its that vlan8 and the vmkernel is affected by this and hence the degrade and lack of performance. So if only the pix would make the routing its not an issue, then direct connection between the two vlans wouldnt cause problems.

[tsightler]

Exactly how is the vswif going to "access direct" between two different VLAN's? Perhaps you didn't describe you're environment very well. What mode are you using for your backup jobs? Virtual Appliance Mode?

[n2it-dk]

backup mode is default and recomended... there are several vswifs one holding the vlans, defined in the vswifs respectively and then another vswif holding the sc and another holding the vmkernel...

Ohh and btw, even though its 100mb just imagine that, it should go way faster than 5Mb/s when doing the backup through veeam, and allso the ping packets shouldnt drop and allso the ping sessions shouldnt rise from <1 to 4-8ms

[rchew]

This is pretty simple to determine. If your 2 end devices are on different network segments (subnets) then you need a router in between to route the packets. As far as I know, there is no way for VMWare to circumvent this basic rule of networking. You could build a software router in a VM but it is still a router.

Ray

[n2it-dk]

yes, vyetta could do this or if the 5224 were l3 switches it could have been solved with intervlan routing. But the problem now is the higher ping times and allso the packetloss, and howcome the vlan8 (vmkernel) is affected by this, the vlan8 shouldnt be affected at all.

[tsightler]

So you're saying backup is "SAN mode with NBD/failover" right. So, since the Veeam VM doesn't have direct access to the storage, it will be falling back to NBD mode. NBD mode requires the VM to connect the to SC and you stated that the Veeam is a VM in VLAN 51 so something has to route that traffic from VLAN51 to the SC VLAN, it won't just magically cross the vswitch because it's a layer 2 switch and otherwise it would be bypassing your ACL's.

I think you might be confusing MB/sec with Mb/sec. Those are drastically different terms. Veeam generally reports performance in MB/sec or "Megabytes per second". Networks are generally referred to in Mb/sec or "Megabits per second".

100Mb/sec = ~12MB/sec

So on your best day you could get 12MB/sec, however, since it's a "one-armed" configuration where traffic in and traffic out has to go over the same link, you're looking at a theoretical max of 6MB/sec. Add in the fact that the port is saturated and yes, you'd see higher pings and dropped packets and 5MB/sec is pretty close to what I'd expect you to see.

Options include:

1. Use "Virtual Appliance" mode for your backups, which would access the storage directly via the ESX host
2. Configure your Veeam VM to have at least one interface on the same VLAN as the service console and continue to use NBD mode since it will not longer need to be routed
3. Configure your Veeam VM to have at least one interface on the same VLAN with the storage and do SAN backups.

Since you're running Veeam in a VM, virtual appliance mode would likely be the best option, and the simplest.

[n2it-dk]

ur absolutely right about your writeup of the various setups, ive tried them all, and still there are bottleneck somewhere... Even though ive made a local iscsi connection to the iscsi box from the Veeam appliance there are still indications of slow speed. Yea ur right about the theoretical speed limits etc when we talk 100Mb or Gb speed and yes the speed is around 5MB wich is as it would be, or I would have wanted it to be highter since no network traffic is going on excepts for the backup window. Ill try and see if I can tweak it and rewiew the setup. Perhaps run a sniffer on the network, cause it seems some of the traffic is running places I dont want it to.

Even though ive tried Virtual Appliance mode, the speed were still the same, read: slow.. and allso the problem with consolidated helper file, that werent released afterwards... Therefore ill stick with SAN mode with NBD/failover.

Im not confused about the Mb and MB terms, my head were just elsewhere and focus werent on terms, but actual events.

Cheers.

[rchew]

Maybe I don't understand what you're trying to convey. Nothing you've told me so far refutes the premise that if your backups are traversing a router that is not capable of keeping up with the data flow, your backups will be slow. Your ping times will be affected because it also has to traverse the same router that can't keep up. If it can't process your backups fast enough, it can't process your pings either.

Am I missing something here? As indicated by tsightler, you are probably running on the "Network" failover so you must be routed since your SC and SAN networks are on different subnets.

Ray

[n2it-dk]

One thing though, what would be an idea to try out would be the E1000 or the VMXNET (2) Enhanced or the VMXNET (3)

What would be the suggested solution and what are peeps experiences with the various nics?

[tsightler]

OK, so you're focused on "events" rather than "terms", I get that. But you're asking for help on a forum. If you're going to take time from troubleshooting your "events" to ask for help on a forum, and then expect others to take their time to assist with troubleshooting your "events" then you need to slow down and actually communicate the problem. Communicating the problem involves using the correct terms so that the people attempting to assist you aren't wasting their time and yours because of misunderstanding of the setup.

I'd suggest, if you actually want to get to the bottom of your situation with assistance from people on this forum, that you start over again, explain your environment, and then do your best to respond to the best of your ability to the questions people ask and the suggestions they make. So far people have offered many helpful suggestions, yet your answers are always "I've tried them all" or you turn around and just ask another question. This implies that you either don't understand your environment enough to explain it, don't understand enough about the suggestions to know why they might help, or both.

[rchew]

One thing though, what would be an idea to try out would be the E1000 or the VMXNET (2) Enhanced or the VMXNET (3)

What would be the suggested solution and what are peeps experiences with the various nics?

If you had a 10 lane highway with a toll-booth in each lane, the traffic would probably flow fairly well. Now imagine if you had that same 10 lane highway but only 2 toll-booths. The infrastructure would be overloaded and cars would have to merge into the 2 toll-booths causing traffic jams. What do you think would happen if you expanded your highway to 100 lanes but did not increase the number of toll-booths? You are applying the same logic with your network cards. Until you resolve your router (aka toll-booth) problem, you will continue to have issues.

Ray

[n2it-dk]

a workaround here is that ive added several nics to the veeam backup station and then added those nics to individual vlans and its progressively better in performance. Ive allso installed iscsi sw on the veeam backup station aswell and added the Luns to the backup station to try and make SAN backups.

So far results are like below.

Total size Processed size Processing rate Duration
20,00 GB 20,00 GB 45 MB/s 0:07:39

Im getting there, just not there yet

[n2it-dk]

btw rchew and tsightler your suggestions are more than welcome... guess we all troubleshoot differently.

[Gostev]

Multiple NICs cannot make difference with Veeam Backup working in network mode. I believe what helped here is the fact that the product switch to SAN mode backups after you installed iSCSI. Backup traffic now goes direct from Veeam Backup VM to your storage, instead of going through the service console.

But I don't think you can get more speed via software iSCSI initiator... 45-50MB/s this is pretty much best you can get for full backups in ideal situation, with Veeam Backup running in VM and using software iSCSI initiator.

[n2it-dk]

Yes Gostev, your might right, but if I choose SAN mode only the backup job gives an error, so I have to go for the default settings to achieve the results above. Ill try rebooting the Veeam backup station and see if the SAN mode will behave afterwards.

Thanks

[n2it-dk]

Somehow this issue is solved, there are nomore unusual traffic coming in through the iscsi vlan, ping session times is absolutely fine. Its a combination between a semi intervlan routing setup and then the sw based iscsi locally installed on the veeam backup station beeing able to create san backup.

Cheers and have a nice friday.