Comprehensive data protection for all workloads
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Improving direct-from-SAN backup speed with iSCSI SAN

Post by Gostev »

joergr has just shared some recommendations (copied below) on tuning the Windows TCP settings which allowed him to achieve almost 2x faster backup speed on DELL Equallogic iSCSI SAN. Since I know EQL is arguably most common SAN among our customers nowadays, I thought this information might be useful for many of you. Besides, this may also help with to other iSCSI SAN makes and models, since this comes down to Windows settings.

Please try these tweaks if you want to, and let us know your results below.
joergr wrote:It depends and differs from system to system. what you need to know is that microsoft (i guess since vista and server 2008) adjusts the tcp window scaling dynamically with tcp autotuning feature. where this is mostly a good one it can become a bad one on some systems. ive explicitly seen very bad experience with especially high-performance access and rdp access. to get rid of this autotuning feature you have to manually disable it:

netsh, interface, tcp, set global autotuninglevel=disable

but then again, sometimes you may want to only set it to highlyrestricted. and again sometimes you may want to set it even more aggressive than microsoft does per default. as i said before, it depends on your system, your nic, your driver, your switches & cabling. You have to find an ideal solution for your special environment.
joergr wrote:...and, for heavens sake, if you use Equallogic storage, PLEASE enable flow control on your switches and PLEASE use good ones (for example the dell 6248 is a very good switch for doing great stuff with equallogic).
See the screenshot below for results with tweaks applied. Please keep in mind that with such storage throughtput, you will need to have a very modern 4-8 core CPU on your backup server, otherwise CPU will be your primary performance bottleneck with the default compression level. Check your CPU load during backup!

Image
larry
Veteran
Posts: 387
Liked: 97 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by larry »

I tested this is my production environment. Just 3 days using a 500 gig VM at 1am. I have other backups and jobs running then so this is not a controlled test by any means. I the other jobs are the same each night with about the same amount of data.

My Veeam servers are Win2008R2, 4 gig ram, 1 Xeon my SAN is the HP lefthand 4300 ver 8.
I did the autotuninglevel=disable

After 53 MB/s 520gb
Before 46 MB/s 520 gb

Same data I just deleted the backups.

PS how do you get the iscsi to show in task mgr ?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by Gostev »

Larry, do you perhaps use Software iSCSI Initiator via your single production LAN NIC?
You may want to use dedicated NIC for your iSCSI network for better performance.

Basically, the Task Manager will show the number of networking graphs equal to the number of your network connections listed here:
Control Panel\Network and Internet\Network Connections

Joerg is actually using dedicated hardware iSCSI HBA, which is why there are 2 graphs on the screenshot.
larry
Veteran
Posts: 387
Liked: 97 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by larry »

Yes I use the software initiator. I have 2 NICs one for the isolated iscsi network and one to the main network.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by Gostev »

If you have 2 NICs, then you should see 2 adaptors in the Task Manager for sure. Anyhow, while you probably won't get 90MB/s via software iSCSI initiator, still the 10% improvement over what you had before is pretty nice!
larry
Veteran
Posts: 387
Liked: 97 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by larry »

I am keeping the improvment.
joergr
Veteran
Posts: 391
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by joergr »

well i can add a surprise. my screenshot is a software initiator dedicated to a physical intel server nic ;-)

I wanted to point that out all the time but had no time: if you got good dedicated nics (intel server nics are really good for iscsi) and a strong machine (lets say two quad core cpu´s, like nehalem 5x) you won´t see that much difference compared to hardware hba (i compared to qlogic 4062c dual port iscsi hba). important is only to explicitly tell ms iscsi initiator for EACH connection to use this dedicated ip (which is the ip assigned to the dedicated iscsi interface). all other network traffic sould use another interface (as you see in my screenshot).

best regards
Joerg
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by tsightler »

Right, we've been doing iSCSI for about 6 years, starting with iSCSI to FC gateways for our old EMC Clarriion systems, but moving to 100% iSCSI in 2007 with Equallogic arrays. Back then there were many advantages to HBA's, both in raw throughput and CPU overhead, however, we're seeing virtually no difference with modern servers/1Gb NIC's. HBA's do still seem to offer some latency win, especially for high IOP environments, but I'm not sure how much the overall impact really is, probably not a lot, unless you have very low latency storage (large cache or SSD's for example). In those environments, the over the wire latency can be enough to have a very measurable impact. Also, for older systems the CPU utilization was generally much lower with an iSCSI HBA, something that's important for running Veeam on marginal hardware, but this is virtually a non-issue for today's powerful CPU's. Software iSCSI on modern CPU's can saturate a 1Gb NIC, and in most cases even 2Gb NIC, while only using single digit percentages of CPU time.

In the world of 10Gb there's still very little throughput difference, but CPU overhead can be significant different. 10Gb iSCSI HBA's are generally using <5% CPU (in most cases ~2%) while the software initiators can use as much as 25% and average 10-20%. This difference is especially noticeable during large block reads and high IOP operations. Still probably not a huge difference in all but the most demanding implementations, but there's is a difference. Latency also seems to favor the HBA's in this environment as well, although the gap seems to be much less.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by Gostev »

This was very useful information for me. Thank you both!!
larry
Veteran
Posts: 387
Liked: 97 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by larry »

Joerg thanks for the info ,I now have a nic called iscsi of my own. :D
wycheadm
Lurker
Posts: 1
Liked: never
Joined: Dec 28, 2009 4:13 pm
Full Name: Tom

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by wycheadm »

I realize this post is about a month old but I wanted to add that I have switched to direct from SAN backups and wow, what an improvement.

I was originally using network backups, but now pulling from a LeftHand SAN to local storage on my backup server.

File Server: ~500 GB
- Network Backup: 400 MB/s
- SAN Backup: 1 GB/s at least. On slower office days it runs at 2 GB/s

Exchange Server: 375 Gb
- Network Backup: 40 MB/s
- SAN Backup: 120 MB/s - 180 MB/s

I have some jobs where the starting and stopping process of Veeam takes longer than the actual backups now. I love checking my Veeam backup emails each morning and seeing just how fast the backups ran the night before. I had one File Server job when I was testing the SAN backups where I had run the job about 4 hours before the scheduled job ran that night and it processed at 4 GB/s, it made me laugh. Just incredible.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by Vitaliy S. »

Tom, thanks for sharing! That's right, SAN backup mode is the best choice to use with shared storage.
TheVirtualDude
Influencer
Posts: 12
Liked: never
Joined: Apr 30, 2009 6:52 pm
Full Name: Faisal Farooqui
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by TheVirtualDude »

Would this apply if your VBR server was a virtual machine?
TrevorBell
Veteran
Posts: 364
Liked: 17 times
Joined: Feb 13, 2009 10:13 am
Full Name: Trevor Bell
Location: Worcester UK
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by TrevorBell »

hi if your backup machine is in a VM then VA mode is slightly faster :) of course if you have vmware enterprise so you have the HOT ADD feature as it uses this... check your vmware licensing in vcentre it will tell you

thanks

trev
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by Vitaliy S. »

Trev,

Actually, you do not need to have an Enterprise license to use the Virtual Appliance mode, it will even work with VMware Essentials :)
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by Gostev »

Yes, it no longer requires Enterprise license since ESX 4.0 Update 1
ESX4 U1: Hot Add backups for everyone!

Faisal, yes this does fully apply to virtual Veeam Backup server.
TheVirtualDude
Influencer
Posts: 12
Liked: never
Joined: Apr 30, 2009 6:52 pm
Full Name: Faisal Farooqui
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by TheVirtualDude »

Do we know if this would apply to windows 2003 server as well?
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by Gostev »

Read the first post carefully ;) OP believes it does not.
joergr
Veteran
Posts: 391
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by joergr »

right, i think/i believe the autotuning feature was introduced with vista kernel, so w2003 is not affected.

best regards,
Joerg
dekkar
Enthusiast
Posts: 38
Liked: 1 time
Joined: Jun 17, 2010 12:21 am
Full Name: Nathan Tong
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by dekkar »

Hi all.... I just made a few positive changes that might help someone....

Im backing up a 500gb VM, and my average processing rate was about 10MB/s
I changed the Storage Optimized setting for LAN target (in the job properties), and also ran that netsh script at the top of the post....

I just re-ran the job and its now runing at 70MB/s
Equallogic SAN with dedicated iSCSI NIC in the veeam server. (2008 server)

Thx for the tips!
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by m.novelli »

Hi guys, would like to resurrect this old topic talking about netsh interface tcp set global autotuninglevel=disable


I'm using this command on every Windows 2008 R2 backup/proxy server and so far I've never has issues or speed complains

Do you think I should use the same command on Windows 2012 R2 backup servers? I'm using only this OS for new installations or re-installations and I'm wondering if the old guidelines apply also to the newer version of Windows OS

Cheers :)

Marco
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by foggy »

I think yes, since the corresponding library article leads to the same command reference.
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by m.novelli »

Well, best practice differs from mere existence of the command on the new version of the OS...
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by lobo519 »

Also Interested - Setting up a new 2012 R2 Veeam server right now planning to use Direct SAN access.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by Gostev » 1 person likes this post

We collect all best practices from our end users, so someone's gotta be the first to try and report what works best ;)
jbush
Novice
Posts: 4
Liked: 3 times
Joined: Jan 23, 2014 2:09 pm
Full Name: Jason Bush
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by jbush »

A little confused here. If the Veeam server is a VM, and setup as direct SAN access to a Dell Equallogic iSCSI SAN, how would modifying the TCP autotuninglevel make any change when Veeam mounts the VMDK's directly to the VM.

When is it best to disable the global autotuninglevel on the TCP stack? When using network mode in Veeam or direct SAN?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by dellock6 » 1 person likes this post

Jason, the idea here is to use directSAN even if the Veeam proxy is a VM, but having an ethernet connection from the VM to the iscsi network. This is the only scenario where directSAN can also be used when Veeam is virtual.
Honestly, I'm not a fan of this solution, there are too many "loops" of ip storage flowing in and out of the hypervisor, the ideal scenario for DirectSAN is when Veeam proxy is a physical machine.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
jbush
Novice
Posts: 4
Liked: 3 times
Joined: Jan 23, 2014 2:09 pm
Full Name: Jason Bush
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by jbush »

Hi Luca, thank you for your explanation, makes complete sense.

I'm in the process of setting up MPIO to the Equallogic in the Veeam VM. Will post results for before and after.
Also I will test direct SAN access using a physical server, again configured with MPIO to the iSCSI fabric and post results. This wont be for a few weeks as the project is yet to be completed.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by dellock6 »

I'm always interested in these kind of tests, I'll be waiting for the results :)

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
jbush
Novice
Posts: 4
Liked: 3 times
Joined: Jan 23, 2014 2:09 pm
Full Name: Jason Bush
Contact:

Re: Improving direct-from-SAN backup speed with iSCSI SAN

Post by jbush »

I got a slight difference by disabling autotuninglevel. This is with the VM directly connected to the iSCSI production fabric using MS iSCSI initiator and MPIO.

Before: Average speed 84MB/s
After: Average speed 86MB/s

I will run more tests as the bottle neck is the iSCSI fabric of the Equallogic. MPIO is not configured correctly so I will have to fix this after hours and run more tests.

Will keep you posted.
Post Reply

Who is online

Users browsing this forum: Amazon [Bot], bct44, Google [Bot], Ivan239, Max93 and 196 guests