Poor vStorage iSCSI SAN speed after Windows 2008 Upgrade

[Felix]

Hi,

I upgraded a clients shared vCenter / Veeam Backup server from Windows 2003 Std x86, vCenter 4.0 U2, Veeam Backup 4.1.2 x86 to Windows 2008 Std x64, vCenter 4.1, Veeam Backup 4.1.2 x64.

The Server is a ProLiant DL380 G5 2x Quad-Core Xeon 5420 2.5GHz w/6GB RAM.
The SAN is a HP MSA 2012i Dual-Controller iSCSI SAN with 8x 15k spindles in RAID6. The two portals per controller are distributed to two iSCSI VLANs on different switches.

Both the old Windows 2003 install and the new Windows 2008 install were configured with MS iSCSI Initiator in Multipath Round-Robin mode.
NIC used for iSCSI is an HP NC364T Intel Quad-Port Gigabit Adapter with two 2GB trunks, one for each portal. Teaming was configured using the HP networking utility with IP based load balancing.

Testing using HD Tune Pro show that load balancing is working with traffic being distributed equally among the two paths and sequential throughput of about 60 MByte/s during normal workload.

After restoring the Veeam Backup DB and updating all passwords, I noticed that the vStorage SAN backups throughput was down to about 6-10 MByte/s, while switching to vStorage NBD (Network) mode, I'm seeing around 35-40 MByte/s throughput. Both monitored during full backup of same vm using task manager network monitor and backup status.

Anyone has an idea what cause this?

Under Windows 2003 vStorage SAN speed was considerably better than NBD speed.
All iSCSI ports are set to 1Gbps FD with Flow Control enabled and Jumbo Frames disabled.
All offloading functions on the intel NIC are enabled.

Regards,
Felix Buenemann

[joergr]

Hi,

did you change sth on your san, too? if yes, tell us, please. if no, let´s have a little check by disabling the trunks and re-configuring sw iscsi initiator to NOT use mpio at all (no round robin, no other mpio). let´s see what it does with a single 1gb link without any trunking/mpio. enabling flow control is fine, disabling jumbo frames is also fine. please make also sure that any unicast anti spoofing/hammering/flooding mechanism on your switch AND on your NIC is turned OFF. Disable ANY hardware offloading mechanism on your nic. Please also uninstall any specially crafted iscsi initiator by the vendor of your san other than the ms iscsi initiator. please also uninstall any san specialized client tool (like for example tools that provide synchronous snapshot operations guest-san). after that please make a backup with veeam b+r but to be sure the dedup engine isn´t eating your bandwith because it has hard work to do and the cpu is what is causing the delay please do a FULL backup to test the speed. please post your results. If it is faster now we have to find the cause by slowly going back to your initial config by adding a certain part at each new test cycle.

best regards,
Joerg

[Felix]

Hi Joerg, thx for the tips. I'll check it out once the current backups have finished.

Rate limiting is disabled on the switches (ProCurve 1800-24G, I know not the ideal choice for iSCSI). Only MS iSCSI Initiator installed, no MSA specific stuff.

The windows firewall is enabled on the server, but I don't think that's an issue.

I noticed that some of the offloading options were not available in the Windows 2003 drivers for the Intel NIC. Currently enabled: IPv4 Checksum Offload Rx/Tx, TCP+UDP Checksum Offload Rx/Tx and Large Send Offload Version 2 enabled.

[joergr]

Hi Felix,

ok, please keep us updated. please also disable Windows Firewall (not certain rules in the adv firewall - please disable it completely) and all hw offload options on your nic. we have to start from a very clean state to find out what is causing your speed problem. you may also want to disable tcp autotuning (it´s a sticky post right on top of this forum).

best regards,
Joerg

[Felix]

OK, problem seems to be related to load balancing. If I switch the Multi-Path policy from Round-Robin to Failover-Only, I get normal performance with peak throughput of > 90 MByte/s.

All other load balancing policies like least queue depth excibit the same problem as Round-Robin: throughput drops to 2x 4 MByte/s.

I'll the wether automatic tcp windows scaling of win 2008 conflicts with round robin. Or maybe the windows 2008 initiator does more context switches, thereby overloading the controller in the san.

[joergr]

Great to hear you found the causer of the speed problem

To be honest, I never ever used round robin MPIO - from the performance point of view no hit. for failover purposes you really don´t need it (especially when using esx/i you have other good options). If it is for performance and we speak of 1:1 connections nothing compares to 10 GB iSCSI

best regards
Joerg

[Felix]

After doing the netsh and registry tweaks suggested by StarWind on http://www.starwindsoftware.com/forums/ ... 92-15.html, Round-Robind and Least-Queue-Depth balancing are back to normal.

I'm currently filling an 8GB VMDK with random data to make a conclusive comparison between performance in failover-only vs. round-robin vs. least-queue-depth.

Btw. 10GB iSCSI: wish I had the budget. I've seen some pretty nice performance from an OpenSolaris/ZFS iSCSI target with multiple gigabit links and round-robin. But I guess the MSA 2012i lacks cpu power to even saturate two gigabit links, given it uses a lowly 600 MHz Celeron CPU.

[Felix]

After some more network tweaking I found that load-balancing performance was bad again.

I searched what had changed and found out that the Tcp1323Opts option had been reset to it's default value 0 instead of the recommended 3.

I went into HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters and set it back to 3 and after a reboot performance was OK again.

If someone suffers the same porblem, he might also choose to set it as a DWORD value in HKLM\...\Services\Tcpip\Parameters\Interfaces\{NIC-GUID} so it only applies to interfaces used for iSCSI.

Btw. performance with and without load-balancing were about equal with unbalanced mode having a slight edge, though I hope that by using Least-Queue-Depth balancing, the backup server might better balance traffic to the right portal, if there is traffic from ESX hosts and thereby cause less clogging of a single portal. I also noticed that during idle load, NBD performance was about equal, though when the SAN was more loaded, vStorage SAN performance was better than NBD.

I also found out that none of NIC's the TCP/IP offloading algorithms or Windows Firewall were affecting performance negatively.