Improving direct-from-SAN backup speed with iSCSI SAN

[Gostev]

joergr has just shared some recommendations (copied below) on tuning the Windows TCP settings which allowed him to achieve almost 2x faster backup speed on DELL Equallogic iSCSI SAN. Since I know EQL is arguably most common SAN among our customers nowadays, I thought this information might be useful for many of you. Besides, this may also help with to other iSCSI SAN makes and models, since this comes down to Windows settings.

Please try these tweaks if you want to, and let us know your results below.

It depends and differs from system to system. what you need to know is that microsoft (i guess since vista and server 2008) adjusts the tcp window scaling dynamically with tcp autotuning feature. where this is mostly a good one it can become a bad one on some systems. ive explicitly seen very bad experience with especially high-performance access and rdp access. to get rid of this autotuning feature you have to manually disable it:

netsh, interface, tcp, set global autotuninglevel=disable

but then again, sometimes you may want to only set it to highlyrestricted. and again sometimes you may want to set it even more aggressive than microsoft does per default. as i said before, it depends on your system, your nic, your driver, your switches & cabling. You have to find an ideal solution for your special environment.

...and, for heavens sake, if you use Equallogic storage, PLEASE enable flow control on your switches and PLEASE use good ones (for example the dell 6248 is a very good switch for doing great stuff with equallogic).

See the screenshot below for results with tweaks applied. Please keep in mind that with such storage throughtput, you will need to have a very modern 4-8 core CPU on your backup server, otherwise CPU will be your primary performance bottleneck with the default compression level. Check your CPU load during backup!

[larry]

I tested this is my production environment. Just 3 days using a 500 gig VM at 1am. I have other backups and jobs running then so this is not a controlled test by any means. I the other jobs are the same each night with about the same amount of data.

My Veeam servers are Win2008R2, 4 gig ram, 1 Xeon my SAN is the HP lefthand 4300 ver 8.
I did the autotuninglevel=disable

After 53 MB/s 520gb
Before 46 MB/s 520 gb

Same data I just deleted the backups.

PS how do you get the iscsi to show in task mgr ?

[Gostev]

Larry, do you perhaps use Software iSCSI Initiator via your single production LAN NIC?
You may want to use dedicated NIC for your iSCSI network for better performance.

Basically, the Task Manager will show the number of networking graphs equal to the number of your network connections listed here:
Control Panel\Network and Internet\Network Connections

Joerg is actually using dedicated hardware iSCSI HBA, which is why there are 2 graphs on the screenshot.

[larry]

Yes I use the software initiator. I have 2 NICs one for the isolated iscsi network and one to the main network.

[Gostev]

If you have 2 NICs, then you should see 2 adaptors in the Task Manager for sure. Anyhow, while you probably won't get 90MB/s via software iSCSI initiator, still the 10% improvement over what you had before is pretty nice!

[larry]

I am keeping the improvment.

[joergr]

well i can add a surprise. my screenshot is a software initiator dedicated to a physical intel server nic

I wanted to point that out all the time but had no time: if you got good dedicated nics (intel server nics are really good for iscsi) and a strong machine (lets say two quad core cpu´s, like nehalem 5x) you won´t see that much difference compared to hardware hba (i compared to qlogic 4062c dual port iscsi hba). important is only to explicitly tell ms iscsi initiator for EACH connection to use this dedicated ip (which is the ip assigned to the dedicated iscsi interface). all other network traffic sould use another interface (as you see in my screenshot).

best regards
Joerg

[tsightler]

Right, we've been doing iSCSI for about 6 years, starting with iSCSI to FC gateways for our old EMC Clarriion systems, but moving to 100% iSCSI in 2007 with Equallogic arrays. Back then there were many advantages to HBA's, both in raw throughput and CPU overhead, however, we're seeing virtually no difference with modern servers/1Gb NIC's. HBA's do still seem to offer some latency win, especially for high IOP environments, but I'm not sure how much the overall impact really is, probably not a lot, unless you have very low latency storage (large cache or SSD's for example). In those environments, the over the wire latency can be enough to have a very measurable impact. Also, for older systems the CPU utilization was generally much lower with an iSCSI HBA, something that's important for running Veeam on marginal hardware, but this is virtually a non-issue for today's powerful CPU's. Software iSCSI on modern CPU's can saturate a 1Gb NIC, and in most cases even 2Gb NIC, while only using single digit percentages of CPU time.

In the world of 10Gb there's still very little throughput difference, but CPU overhead can be significant different. 10Gb iSCSI HBA's are generally using <5% CPU (in most cases ~2%) while the software initiators can use as much as 25% and average 10-20%. This difference is especially noticeable during large block reads and high IOP operations. Still probably not a huge difference in all but the most demanding implementations, but there's is a difference. Latency also seems to favor the HBA's in this environment as well, although the gap seems to be much less.

[Gostev]

This was very useful information for me. Thank you both!!

[larry]

Joerg thanks for the info ,I now have a nic called iscsi of my own.

[wycheadm]

I realize this post is about a month old but I wanted to add that I have switched to direct from SAN backups and wow, what an improvement.

I was originally using network backups, but now pulling from a LeftHand SAN to local storage on my backup server.

File Server: ~500 GB
- Network Backup: 400 MB/s
- SAN Backup: 1 GB/s at least. On slower office days it runs at 2 GB/s

Exchange Server: 375 Gb
- Network Backup: 40 MB/s
- SAN Backup: 120 MB/s - 180 MB/s

I have some jobs where the starting and stopping process of Veeam takes longer than the actual backups now. I love checking my Veeam backup emails each morning and seeing just how fast the backups ran the night before. I had one File Server job when I was testing the SAN backups where I had run the job about 4 hours before the scheduled job ran that night and it processed at 4 GB/s, it made me laugh. Just incredible.

[Vitaliy S.]

Tom, thanks for sharing! That's right, SAN backup mode is the best choice to use with shared storage.

[TheVirtualDude]

Would this apply if your VBR server was a virtual machine?

[TrevorBell]

hi if your backup machine is in a VM then VA mode is slightly faster of course if you have vmware enterprise so you have the HOT ADD feature as it uses this... check your vmware licensing in vcentre it will tell you

thanks

trev

[Vitaliy S.]

Trev,

Actually, you do not need to have an Enterprise license to use the Virtual Appliance mode, it will even work with VMware Essentials

[Gostev]

Yes, it no longer requires Enterprise license since ESX 4.0 Update 1
ESX4 U1: Hot Add backups for everyone!

Faisal, yes this does fully apply to virtual Veeam Backup server.

[TheVirtualDude]

Do we know if this would apply to windows 2003 server as well?

[Gostev]

Read the first post carefully OP believes it does not.

[joergr]

right, i think/i believe the autotuning feature was introduced with vista kernel, so w2003 is not affected.

best regards,
Joerg

[dekkar]

Hi all.... I just made a few positive changes that might help someone....

Im backing up a 500gb VM, and my average processing rate was about 10MB/s
I changed the Storage Optimized setting for LAN target (in the job properties), and also ran that netsh script at the top of the post....

I just re-ran the job and its now runing at 70MB/s
Equallogic SAN with dedicated iSCSI NIC in the veeam server. (2008 server)

Thx for the tips!

[m.novelli]

Hi guys, would like to resurrect this old topic talking about netsh interface tcp set global autotuninglevel=disable


I'm using this command on every Windows 2008 R2 backup/proxy server and so far I've never has issues or speed complains

Do you think I should use the same command on Windows 2012 R2 backup servers? I'm using only this OS for new installations or re-installations and I'm wondering if the old guidelines apply also to the newer version of Windows OS

Cheers

Marco

[foggy]

I think yes, since the corresponding library article leads to the same command reference.

[m.novelli]

Well, best practice differs from mere existence of the command on the new version of the OS...

[lobo519]

Also Interested - Setting up a new 2012 R2 Veeam server right now planning to use Direct SAN access.

[Gostev]

We collect all best practices from our end users, so someone's gotta be the first to try and report what works best

[jbush]

A little confused here. If the Veeam server is a VM, and setup as direct SAN access to a Dell Equallogic iSCSI SAN, how would modifying the TCP autotuninglevel make any change when Veeam mounts the VMDK's directly to the VM.

When is it best to disable the global autotuninglevel on the TCP stack? When using network mode in Veeam or direct SAN?

[dellock6]

Jason, the idea here is to use directSAN even if the Veeam proxy is a VM, but having an ethernet connection from the VM to the iscsi network. This is the only scenario where directSAN can also be used when Veeam is virtual.
Honestly, I'm not a fan of this solution, there are too many "loops" of ip storage flowing in and out of the hypervisor, the ideal scenario for DirectSAN is when Veeam proxy is a physical machine.

Luca.

[jbush]

Hi Luca, thank you for your explanation, makes complete sense.

I'm in the process of setting up MPIO to the Equallogic in the Veeam VM. Will post results for before and after.
Also I will test direct SAN access using a physical server, again configured with MPIO to the iSCSI fabric and post results. This wont be for a few weeks as the project is yet to be completed.

[dellock6]

I'm always interested in these kind of tests, I'll be waiting for the results

Luca.

[jbush]

I got a slight difference by disabling autotuninglevel. This is with the VM directly connected to the iSCSI production fabric using MS iSCSI initiator and MPIO.

Before: Average speed 84MB/s
After: Average speed 86MB/s

I will run more tests as the bottle neck is the iSCSI fabric of the Equallogic. MPIO is not configured correctly so I will have to fix this after hours and run more tests.

Will keep you posted.