Improving direct-from-SAN backup speed with iSCSI SAN

Availability for the Always-On Enterprise

Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby Gostev » Wed Jun 09, 2010 11:54 am

joergr has just shared some recommendations (copied below) on tuning the Windows TCP settings which allowed him to achieve almost 2x faster backup speed on DELL Equallogic iSCSI SAN. Since I know EQL is arguably most common SAN among our customers nowadays, I thought this information might be useful for many of you. Besides, this may also help with to other iSCSI SAN makes and models, since this comes down to Windows settings.

Please try these tweaks if you want to, and let us know your results below.

joergr wrote:It depends and differs from system to system. what you need to know is that microsoft (i guess since vista and server 2008) adjusts the tcp window scaling dynamically with tcp autotuning feature. where this is mostly a good one it can become a bad one on some systems. ive explicitly seen very bad experience with especially high-performance access and rdp access. to get rid of this autotuning feature you have to manually disable it:

netsh, interface, tcp, set global autotuninglevel=disable

but then again, sometimes you may want to only set it to highlyrestricted. and again sometimes you may want to set it even more aggressive than microsoft does per default. as i said before, it depends on your system, your nic, your driver, your switches & cabling. You have to find an ideal solution for your special environment.


joergr wrote:...and, for heavens sake, if you use Equallogic storage, PLEASE enable flow control on your switches and PLEASE use good ones (for example the dell 6248 is a very good switch for doing great stuff with equallogic).


See the screenshot below for results with tweaks applied. Please keep in mind that with such storage throughtput, you will need to have a very modern 4-8 core CPU on your backup server, otherwise CPU will be your primary performance bottleneck with the default compression level. Check your CPU load during backup!

Image
Gostev
Veeam Software
 
Posts: 21364
Liked: 2336 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby larry » Thu Jun 17, 2010 1:06 pm

I tested this is my production environment. Just 3 days using a 500 gig VM at 1am. I have other backups and jobs running then so this is not a controlled test by any means. I the other jobs are the same each night with about the same amount of data.

My Veeam servers are Win2008R2, 4 gig ram, 1 Xeon my SAN is the HP lefthand 4300 ver 8.
I did the autotuninglevel=disable

After 53 MB/s 520gb
Before 46 MB/s 520 gb

Same data I just deleted the backups.

PS how do you get the iscsi to show in task mgr ?
larry
Expert
 
Posts: 371
Liked: 88 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby Gostev » Thu Jun 17, 2010 1:37 pm

Larry, do you perhaps use Software iSCSI Initiator via your single production LAN NIC?
You may want to use dedicated NIC for your iSCSI network for better performance.

Basically, the Task Manager will show the number of networking graphs equal to the number of your network connections listed here:
Control Panel\Network and Internet\Network Connections

Joerg is actually using dedicated hardware iSCSI HBA, which is why there are 2 graphs on the screenshot.
Gostev
Veeam Software
 
Posts: 21364
Liked: 2336 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby larry » Thu Jun 17, 2010 1:58 pm

Yes I use the software initiator. I have 2 NICs one for the isolated iscsi network and one to the main network.
larry
Expert
 
Posts: 371
Liked: 88 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby Gostev » Thu Jun 17, 2010 2:46 pm

If you have 2 NICs, then you should see 2 adaptors in the Task Manager for sure. Anyhow, while you probably won't get 90MB/s via software iSCSI initiator, still the 10% improvement over what you had before is pretty nice!
Gostev
Veeam Software
 
Posts: 21364
Liked: 2336 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby larry » Thu Jun 17, 2010 2:50 pm

I am keeping the improvment.
larry
Expert
 
Posts: 371
Liked: 88 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby joergr » Thu Jun 17, 2010 8:43 pm

well i can add a surprise. my screenshot is a software initiator dedicated to a physical intel server nic ;-)

I wanted to point that out all the time but had no time: if you got good dedicated nics (intel server nics are really good for iscsi) and a strong machine (lets say two quad core cpu´s, like nehalem 5x) you won´t see that much difference compared to hardware hba (i compared to qlogic 4062c dual port iscsi hba). important is only to explicitly tell ms iscsi initiator for EACH connection to use this dedicated ip (which is the ip assigned to the dedicated iscsi interface). all other network traffic sould use another interface (as you see in my screenshot).

best regards
Joerg
joergr
Expert
 
Posts: 377
Liked: 39 times
Joined: Tue Jun 08, 2010 2:01 pm
Full Name: Joerg Riether

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby tsightler » Thu Jun 17, 2010 9:12 pm

Right, we've been doing iSCSI for about 6 years, starting with iSCSI to FC gateways for our old EMC Clarriion systems, but moving to 100% iSCSI in 2007 with Equallogic arrays. Back then there were many advantages to HBA's, both in raw throughput and CPU overhead, however, we're seeing virtually no difference with modern servers/1Gb NIC's. HBA's do still seem to offer some latency win, especially for high IOP environments, but I'm not sure how much the overall impact really is, probably not a lot, unless you have very low latency storage (large cache or SSD's for example). In those environments, the over the wire latency can be enough to have a very measurable impact. Also, for older systems the CPU utilization was generally much lower with an iSCSI HBA, something that's important for running Veeam on marginal hardware, but this is virtually a non-issue for today's powerful CPU's. Software iSCSI on modern CPU's can saturate a 1Gb NIC, and in most cases even 2Gb NIC, while only using single digit percentages of CPU time.

In the world of 10Gb there's still very little throughput difference, but CPU overhead can be significant different. 10Gb iSCSI HBA's are generally using <5% CPU (in most cases ~2%) while the software initiators can use as much as 25% and average 10-20%. This difference is especially noticeable during large block reads and high IOP operations. Still probably not a huge difference in all but the most demanding implementations, but there's is a difference. Latency also seems to favor the HBA's in this environment as well, although the gap seems to be much less.
tsightler
Veeam Software
 
Posts: 4745
Liked: 1729 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby Gostev » Thu Jun 17, 2010 9:50 pm

This was very useful information for me. Thank you both!!
Gostev
Veeam Software
 
Posts: 21364
Liked: 2336 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby larry » Mon Jun 21, 2010 8:03 pm

Joerg thanks for the info ,I now have a nic called iscsi of my own. :D
larry
Expert
 
Posts: 371
Liked: 88 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby wycheadm » Tue Jul 20, 2010 4:00 pm

I realize this post is about a month old but I wanted to add that I have switched to direct from SAN backups and wow, what an improvement.

I was originally using network backups, but now pulling from a LeftHand SAN to local storage on my backup server.

File Server: ~500 GB
- Network Backup: 400 MB/s
- SAN Backup: 1 GB/s at least. On slower office days it runs at 2 GB/s

Exchange Server: 375 Gb
- Network Backup: 40 MB/s
- SAN Backup: 120 MB/s - 180 MB/s

I have some jobs where the starting and stopping process of Veeam takes longer than the actual backups now. I love checking my Veeam backup emails each morning and seeing just how fast the backups ran the night before. I had one File Server job when I was testing the SAN backups where I had run the job about 4 hours before the scheduled job ran that night and it processed at 4 GB/s, it made me laugh. Just incredible.
wycheadm
Lurker
 
Posts: 1
Liked: never
Joined: Mon Dec 28, 2009 4:13 pm
Full Name: Tom

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby Vitaliy S. » Tue Jul 20, 2010 4:05 pm

Tom, thanks for sharing! That's right, SAN backup mode is the best choice to use with shared storage.
Vitaliy S.
Veeam Software
 
Posts: 19466
Liked: 1092 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby TheVirtualDude » Fri Aug 20, 2010 3:21 am

Would this apply if your VBR server was a virtual machine?
TheVirtualDude
Influencer
 
Posts: 12
Liked: never
Joined: Thu Apr 30, 2009 6:52 pm
Full Name: Faisal Farooqui

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby TrevorBell » Fri Aug 20, 2010 6:58 am

hi if your backup machine is in a VM then VA mode is slightly faster :) of course if you have vmware enterprise so you have the HOT ADD feature as it uses this... check your vmware licensing in vcentre it will tell you

thanks

trev
TrevorBell
Expert
 
Posts: 331
Liked: 2 times
Joined: Fri Feb 13, 2009 10:13 am
Location: Redditch UK
Full Name: Trevor Bell

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Veeam Logoby Vitaliy S. » Fri Aug 20, 2010 7:32 am

Trev,

Actually, you do not need to have an Enterprise license to use the Virtual Appliance mode, it will even work with VMware Essentials :)
Vitaliy S.
Veeam Software
 
Posts: 19466
Liked: 1092 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot] and 25 guests