Equallogic iSCSI and Broadcom = slow backup speed

VMware specific discussions

Equallogic iSCSI and Broadcom = slow backup speed

Veeam Logoby PedroVDP » Thu Jul 19, 2012 8:37 am

I've been breaking my head over this and so far I've not been able to find a reason why my backups are running this slow.

I'm running Veeam B&R 6.1 on a physical dedicated host with local storage as backup target. The server has iSCISI access to the datastores, obviously, to make a direct SAN backup. It is connected to the SAN network with 4 GB NICs with jumbo frames enabled.

My bottleneck values are Source 98% > Proxy 14% > Network 1% > Target 0% for pretty much all of my jobs and somehow I still can't squeeze more out of my SAN backups than 25MB/s even if there is only 1 job running. When I present a volume to the backup server itself and pump data onto it, it's easily 4 times faster.

Does anyone have any insights to offer?
PedroVDP
Service Provider
 
Posts: 18
Liked: 2 times
Joined: Wed Jan 18, 2012 9:10 am

Re: Equallogic isCSI - Slow backup speed ?

Veeam Logoby foggy » Thu Jul 19, 2012 9:34 am

First of all, could you please check whether the direct SAN mode is effectively used to get VMs' data from the storage? This can be verified in the job's realtime statistics window by selecting the particular VM to the left. For direct SAN mode you should see the [san] label after the name of the selected proxy server.
foggy
Veeam Software
 
Posts: 15272
Liked: 1131 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Equallogic isCSI - Slow backup speed ?

Veeam Logoby Vitaliy S. » Thu Jul 19, 2012 11:00 am

PedroVDP wrote:When I present a volume to the backup server itself and pump data onto it, it's easily 4 times faster.

Could you please tell us exactly what you do here? Do you present the volume to the backup server and run the backup job which becomes 4 times faster? Or you're writing data to the NTFS formatted volume on the same SAN via iSCSI?
Vitaliy S.
Veeam Software
 
Posts: 19942
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Equallogic isCSI - Slow backup speed ?

Veeam Logoby PedroVDP » Thu Jul 19, 2012 12:05 pm

The job/proxy is definitely running in san mode, not nbt.

When I say I get 4 times better result I meant i presented a volume on the san to the backup server, formatted in ntfs and did a file-copy to it of large 10GB files. Aftwards I did a copy in the other direction, san to local. Doing that my speeds shoot up into triple digits easily... while Veeam backup data delivery when the job is running barely makes it past 25MB/s, even if I'm running a first-time backup to not be reading all over the place with cbt.
PedroVDP
Service Provider
 
Posts: 18
Liked: 2 times
Joined: Wed Jan 18, 2012 9:10 am

Re: Equallogic isCSI - Slow backup speed ?

Veeam Logoby Cokovic » Thu Jul 19, 2012 1:25 pm

This isn't really comparable as Veeam will hit your storage much harder in reverse incremental mode with random reads and writes then what you did.

You should maybe use some benchmarking tools like Iometer, Atto or CrystalDiskMark to test your storage performance as they will test with different block sizes and queue depths.
Cokovic
Expert
 
Posts: 295
Liked: 59 times
Joined: Tue Sep 06, 2011 8:45 am
Full Name: Haris Cokovic

Re: Equallogic isCSI - Slow backup speed ?

Veeam Logoby PedroVDP » Thu Jul 19, 2012 2:30 pm

I think some wires crossed there. Allow me to try to clarify.

Veeam server: physical host
Backup target: internal disks inside the Veeam server, raid set of 22 disks.
Source: VMFS datastores on an Equallogic SAN
Additional SAN speed test: using an NTFS volume on the san to do read / write operations.
job metrics: Source 98% > Proxy 14% > Network 1% > Target 0%

I can understand that random blocks being read can be less performant than sequential ones but my SAN is not the bottleneck, there is no latency on the volumes, there is no queing, ... if that were the issue, I'd already seen why the backup was slow. I've been looking at performance counters at both the SAN and system side and nothing indicates any problems there. It just doesn't go above 25MB/s
PedroVDP
Service Provider
 
Posts: 18
Liked: 2 times
Joined: Wed Jan 18, 2012 9:10 am

Re: Equallogic isCSI - Slow backup speed ?

Veeam Logoby tsightler » Thu Jul 19, 2012 2:43 pm

Do you have the Equallogic HIT kit installed or not?
tsightler
Veeam Software
 
Posts: 4872
Liked: 1819 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Equallogic isCSI - Slow backup speed ?

Veeam Logoby PedroVDP » Thu Jul 19, 2012 3:14 pm

tsightler wrote:Do you have the Equallogic HIT kit installed or not?


yes, it is installed and working fine: creating managed connections according to the settings as expected, configured to use the recommended 'Least Queue Depth'
PedroVDP
Service Provider
 
Posts: 18
Liked: 2 times
Joined: Wed Jan 18, 2012 9:10 am

Re: Equallogic isCSI - Slow backup speed ?

Veeam Logoby dwilliam62 » Sat Jul 21, 2012 3:42 pm

It sounds like you are running SANHQ? Is that correct?

I didn't see where you stated what version of ESX you are using. The problem could actually be on the ESX host. There are a number of tweaks there that could improve performance. I.e. If you are using VMware Round Robin, adjusting the IOs per path from 1000 to 3, will make a big difference. As will installing MEM 1.1.0, if you have Enterprise or better licensing. Disabling Delayed ACK is another key configuration that will improve read spead. Along with disabling Large Recieve Offload. There are KB articles on the Equallogic Support Site that covers how to do this for ESX v4.x->5.x. Including a script to set the IO threshold. If your switch can handle both Jumbo Frames and Flowcontrol then enabling Jumbo Frames can offer an incremental improvement.

Do you have a case open with Dell? They can help you with these various settings and check the health of the SAN group. Running current firmware is also strongly recommended. As is upgrading ESX to most current build. Often an update to a network driver can improve performance.

Regards,

Don
dwilliam62
Lurker
 
Posts: 2
Liked: never
Joined: Sat Jul 21, 2012 3:35 pm
Full Name: Don Williams

Re: Equallogic isCSI - Slow backup speed ?

Veeam Logoby daddygrant » Tue Jul 24, 2012 4:37 am 2 people like this post

I'm always on the forums but rarely comment. I refuse to let you suffer as I did. Pick up your sword spartan.

I was battling this same issue for months on a similar setup. I tried multiple versions on Equallogic H.I.T and poured over documentation. The max I could get on the CBT was 17mb/s. Double... no triple checked everything and performance was as expected everywhere but on my Veeam server (DL360 G5 with a dual port PCE-E Broadcom 5709c based card for my ISCSI Connections). Then once day I decide to install the latest and greatest Broadcom drivers and it wrecked my box. No matter what I did I couldn't get the on-board Broadcom NICS, the PCI-E broadcom card and Broadcom direct drivers to function together. I ended up reverting back to the MIcrosoft drivers Windows 2008 R2 provide to get the NICS to function. This event haunted me all night so in the morning I went through my space parts and threw in an old dual port PCI-E Intel Pro 1000 MT card. The latest drivers from Intel installed with no issue. I turned on Jumbo Frames and ensured flow control was good. Then I went in to microsoft iscsi management to ensure the H.I.T tools saw my LUNS.

I held my breath and ran a job. Bam!!!!!!!!!!!! 50MB/s CBT all day every day.

Try a card from another vendor for your ISCSI connection. It might work for you or might not but if you spent as much time as I trying to solve the speed issue it's worth the hour to test.

FYI - Veeam 6.1 -> (2) Equallogic PS6000X -> (2) Stacked Cisco 3760X - (3) DL360 G7
daddygrant
Novice
 
Posts: 4
Liked: 2 times
Joined: Tue Sep 13, 2011 5:16 am
Full Name: UGRANT

Re: Equallogic isCSI - Slow backup speed ?

Veeam Logoby chimera » Thu Jul 26, 2012 8:50 am

I concur with the above. Broadcom are *@#$# when it comes to performance. Had the same issue with Broadcom 57711 10GbE NIC's in my VMWare ESXi hosts to an EqualLogic PS6010XV. Exceptionally slow - latency through the roof. Eventually downloaded a VMWare driver update for the Broadcom's that helped significantly. Had similar issues from the physical Veeam server, upgraded the NIC drivers and it helped. The only other thing I can think of it EqualLogic firmware - I had an issue with a specific revision of EqualLogic firmware (can't remember the exact revision sorry, 5.0.7?) that was really really bad with Veeam, upgraded the Firmware to 5.2.2 and it was signficantly faster.
chimera
Enthusiast
 
Posts: 57
Liked: 3 times
Joined: Thu Apr 09, 2009 1:00 am
Full Name: J I

Direct SAN Backup Speed & Best Practices Dell EqualLogic SAN

Veeam Logoby paps40 » Thu Aug 09, 2012 9:55 pm

[merged]

After talking with some veeam employees and other users at the Veeam User Group Meeting I decided to overcome my fears about direct san backups and swith to Direct SAN backups. I put a ticket in with veeam support for some offical instrustions and they gave me links to a couple different articles by Rick Vanover on how to set that up. It was pretty easy.

I wanted to post these questions to the veeam forum moderators or other users and see what an acceptable speed is via Direct SAN Backups. I'm not getting the speeds that I thought I would get and I'm still seeing high CPU usage during backup jobs.

Environment Notes
Veeam Server - Physical, Dell PE 2950, 2 Quad Xeon CPUs and 30GB RAM, Dedicated to only run veeam.
Veeam Software - running 6.1.0.181
SAN - Dell EqualLogic PS6100XV
Backups going to old SAN - Dell MD3000i, 4 iSCSI connections.
VMware - running version 4.1

My reverse incrementals are very fast. It is my active fulls that I thought would be much faster. We are doing active fulls every weekend.

Test Job Notes
Test Job - 2 VM's, Server 2003 File Servers, job settings are at defaults.

Network Mode Backup Notes (Running Veeam 6.0)
Duration = 4:55
Processing Rate = 20 MB/s
Bottleneck = Source (Source 99%, Proxy 67%, Network 9%, Target 10%)
Processed = 341 GB
Transferred = 246.6 GB

Direct SAN Backup Notes (Running Veeam 6.1)
Duration = 3:05
Processing Rate = 30 MB/s
Bottleneck = Source (Source 99%, Proxy 90%, Network 14%, Target 14%)
Processed = 322.9 GB
Transferred = 235.2 GB

Questions
1. If I am using direct SAN backups now why is my bottleneck still the source? I assumed that direct san would be at least be 2x as fast as network mode. Is this normal for direct san speeds to only be approximately 10MB/s faster than network mode? I don't think so.

2. Why does veeam only use approximately 36 to 40 % of the NIC hardwired into the SAN? Is there something in the source code that limits nic usage? The support tech I worked with seemed to think so but wasn't 100 % sure. If I use a NIC Team and can get the iSCI initiator to talk to EQL SAN will veeam use more of that NIC than 36 to 40 % or would that be all for nothing if there is something in the code to limit that.

3. There are times when I run this test job and veeam consumes 100% of my CPU. I thought that this would be fixed when upgrading to veeam 6.1 but it is not. Is anyone else still seeing that issue? If I lower compression from Optimal to low then it drops by 30 to 35 % but it then increase my test job size by 70GB which won't work as I'm constantly fighting the space battle. Veeam support wasn't sure and I got tired of the back and forth so I closed the ticket (Ticket # 5206035). They did recommend an uninstall and reinstall of the software. Haven't tried that yet.

Thanks in advance for any suggestions, tips, best practices or answers.
paps40
Influencer
 
Posts: 24
Liked: 10 times
Joined: Mon Dec 12, 2011 4:10 pm
Full Name: Peter Pappas

Re: Equallogic iSCSI and Broadcom = slow backup speed

Veeam Logoby Vitaliy S. » Fri Aug 10, 2012 9:42 am 1 person likes this post

Hi Peter,

1. Since bottleneck just represents the "weakest" link in your environment, it is ok to have it on source side even if you're running direct SAN Mode. And yes, I would also expect the better performance rates for your backup job. Please review this topic for some tips and tricks that might help you.

2. No, we do not have any hard caps in the source code. Could you please tell me what Windows Server version do you have deployed on your proxy machine? If it is Windows Server 2008, then please take a look a this post that might give you a couple of ideas on how to improve the backup job performance.

3. 100% of CPU utilization is not an issue, it is absolutely expected (for the on the fly data compression an deduplication) to have high CPU utilization during backup/replication job runs.

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19942
Liked: 1144 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Equallogic iSCSI and Broadcom = slow backup speed

Veeam Logoby nmace » Fri Aug 10, 2012 1:17 pm 4 people like this post

Just a FYI, I successfully backed up a test VMFS volume last night using direct SAN access @ 200+MB/sec. This was an Equal Logic SAN with 10 gig interfaces and my Veeam physical server has a Broadcom 57711 10 gig card in a Dell R710. I'm using Dell drivers for the NIC dated 2/4/2011 driver version 6.2.9.0. I did use Dell's lifecycle management tool to update the firmware of the 10 gig card and raid controller to the newest version before I started. I don't know if that had any effect or not as I hadn't tried it before I updated the firmware. I'm not using Dell's MPIO software, just what ships with Windows Server 2008 R2. I'm also writing to local storage on the physical box, which is a raid 10 volume.

Just a thought to people experiencing poor performance, it might be worth checking the firmware of your iSCSI NICs and firmware for your Raid controller.
nmace
Enthusiast
 
Posts: 93
Liked: 12 times
Joined: Mon Jul 23, 2012 3:48 pm

Re: Equallogic iSCSI and Broadcom = slow backup speed

Veeam Logoby paps40 » Wed Aug 15, 2012 9:10 pm 3 people like this post

Vitaliy,
Thanks for the response. I was able to improve my Veeam Direct SAN Backup speed to 62 MB/s. Big improvement from 30 MB/s.

I did the following things:
1. Used the link you gave me to disabled auto tunning: netsh interface tcp set global autotuninglevel=disable (This only increased my speeds with my Dell EQL SAN by about 27 MB/s)
2. Updated the Intel Quad Nics with the latest drivers / PROset software.
3. Turned off file and print sharing on that nic connection.
4. Set Receive Side Scaling Queues from default value of 1 to 8 queues. (All those changes increased my speed by about 5MB/s more)

I tweaked pretty much every NIC setting and then re-ran my job after every change and this is what gave me the best performance. After 6 re-runs I settled on just leaving the nic at default settings but just adjusting the RSS Queues to 8. If anyone has any other NIC tweaks that have increased performance I would love to hear about them. I also spoke with Intel and Dell EQL but they didn't want to provide any insight and gave me the runaround. They just referred me to some support articles to read. Below are links to the intel site that go over what settings can be tweaked on the nic.

http://www.intel.com/support/network/ad ... 029402.htm
http://www.intel.com/support/network/sb/CS-025841.htm
paps40
Influencer
 
Posts: 24
Liked: 10 times
Joined: Mon Dec 12, 2011 4:10 pm
Full Name: Peter Pappas

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 2 guests