Equallogic iSCSI and Broadcom = slow backup speed

[PedroVDP]

I've been breaking my head over this and so far I've not been able to find a reason why my backups are running this slow.

I'm running Veeam B&R 6.1 on a physical dedicated host with local storage as backup target. The server has iSCISI access to the datastores, obviously, to make a direct SAN backup. It is connected to the SAN network with 4 GB NICs with jumbo frames enabled.

My bottleneck values are Source 98% > Proxy 14% > Network 1% > Target 0% for pretty much all of my jobs and somehow I still can't squeeze more out of my SAN backups than 25MB/s even if there is only 1 job running. When I present a volume to the backup server itself and pump data onto it, it's easily 4 times faster.

Does anyone have any insights to offer?

[foggy]

First of all, could you please check whether the direct SAN mode is effectively used to get VMs' data from the storage? This can be verified in the job's realtime statistics window by selecting the particular VM to the left. For direct SAN mode you should see the [san] label after the name of the selected proxy server.

[Vitaliy S.]

When I present a volume to the backup server itself and pump data onto it, it's easily 4 times faster.

Could you please tell us exactly what you do here? Do you present the volume to the backup server and run the backup job which becomes 4 times faster? Or you're writing data to the NTFS formatted volume on the same SAN via iSCSI?

[PedroVDP]

The job/proxy is definitely running in san mode, not nbt.

When I say I get 4 times better result I meant i presented a volume on the san to the backup server, formatted in ntfs and did a file-copy to it of large 10GB files. Aftwards I did a copy in the other direction, san to local. Doing that my speeds shoot up into triple digits easily... while Veeam backup data delivery when the job is running barely makes it past 25MB/s, even if I'm running a first-time backup to not be reading all over the place with cbt.

[Cokovic]

This isn't really comparable as Veeam will hit your storage much harder in reverse incremental mode with random reads and writes then what you did.

You should maybe use some benchmarking tools like Iometer, Atto or CrystalDiskMark to test your storage performance as they will test with different block sizes and queue depths.

[PedroVDP]

I think some wires crossed there. Allow me to try to clarify.

Veeam server: physical host
Backup target: internal disks inside the Veeam server, raid set of 22 disks.
Source: VMFS datastores on an Equallogic SAN
Additional SAN speed test: using an NTFS volume on the san to do read / write operations.
job metrics: Source 98% > Proxy 14% > Network 1% > Target 0%

I can understand that random blocks being read can be less performant than sequential ones but my SAN is not the bottleneck, there is no latency on the volumes, there is no queing, ... if that were the issue, I'd already seen why the backup was slow. I've been looking at performance counters at both the SAN and system side and nothing indicates any problems there. It just doesn't go above 25MB/s

[tsightler]

Do you have the Equallogic HIT kit installed or not?

[PedroVDP]

Do you have the Equallogic HIT kit installed or not?

yes, it is installed and working fine: creating managed connections according to the settings as expected, configured to use the recommended 'Least Queue Depth'

[dwilliam62]

It sounds like you are running SANHQ? Is that correct?

I didn't see where you stated what version of ESX you are using. The problem could actually be on the ESX host. There are a number of tweaks there that could improve performance. I.e. If you are using VMware Round Robin, adjusting the IOs per path from 1000 to 3, will make a big difference. As will installing MEM 1.1.0, if you have Enterprise or better licensing. Disabling Delayed ACK is another key configuration that will improve read spead. Along with disabling Large Recieve Offload. There are KB articles on the Equallogic Support Site that covers how to do this for ESX v4.x->5.x. Including a script to set the IO threshold. If your switch can handle both Jumbo Frames and Flowcontrol then enabling Jumbo Frames can offer an incremental improvement.

Do you have a case open with Dell? They can help you with these various settings and check the health of the SAN group. Running current firmware is also strongly recommended. As is upgrading ESX to most current build. Often an update to a network driver can improve performance.

Regards,

Don

[daddygrant]

I'm always on the forums but rarely comment. I refuse to let you suffer as I did. Pick up your sword spartan.

I was battling this same issue for months on a similar setup. I tried multiple versions on Equallogic H.I.T and poured over documentation. The max I could get on the CBT was 17mb/s. Double... no triple checked everything and performance was as expected everywhere but on my Veeam server (DL360 G5 with a dual port PCE-E Broadcom 5709c based card for my ISCSI Connections). Then once day I decide to install the latest and greatest Broadcom drivers and it wrecked my box. No matter what I did I couldn't get the on-board Broadcom NICS, the PCI-E broadcom card and Broadcom direct drivers to function together. I ended up reverting back to the MIcrosoft drivers Windows 2008 R2 provide to get the NICS to function. This event haunted me all night so in the morning I went through my space parts and threw in an old dual port PCI-E Intel Pro 1000 MT card. The latest drivers from Intel installed with no issue. I turned on Jumbo Frames and ensured flow control was good. Then I went in to microsoft iscsi management to ensure the H.I.T tools saw my LUNS.

I held my breath and ran a job. Bam!!!!!!!!!!!! 50MB/s CBT all day every day.

Try a card from another vendor for your ISCSI connection. It might work for you or might not but if you spent as much time as I trying to solve the speed issue it's worth the hour to test.

FYI - Veeam 6.1 -> (2) Equallogic PS6000X -> (2) Stacked Cisco 3760X - (3) DL360 G7

[chimera]

I concur with the above. Broadcom are *@#$# when it comes to performance. Had the same issue with Broadcom 57711 10GbE NIC's in my VMWare ESXi hosts to an EqualLogic PS6010XV. Exceptionally slow - latency through the roof. Eventually downloaded a VMWare driver update for the Broadcom's that helped significantly. Had similar issues from the physical Veeam server, upgraded the NIC drivers and it helped. The only other thing I can think of it EqualLogic firmware - I had an issue with a specific revision of EqualLogic firmware (can't remember the exact revision sorry, 5.0.7?) that was really really bad with Veeam, upgraded the Firmware to 5.2.2 and it was signficantly faster.

[paps40]

[merged]

After talking with some veeam employees and other users at the Veeam User Group Meeting I decided to overcome my fears about direct san backups and swith to Direct SAN backups. I put a ticket in with veeam support for some offical instrustions and they gave me links to a couple different articles by Rick Vanover on how to set that up. It was pretty easy.

I wanted to post these questions to the veeam forum moderators or other users and see what an acceptable speed is via Direct SAN Backups. I'm not getting the speeds that I thought I would get and I'm still seeing high CPU usage during backup jobs.

Environment Notes
Veeam Server - Physical, Dell PE 2950, 2 Quad Xeon CPUs and 30GB RAM, Dedicated to only run veeam.
Veeam Software - running 6.1.0.181
SAN - Dell EqualLogic PS6100XV
Backups going to old SAN - Dell MD3000i, 4 iSCSI connections.
VMware - running version 4.1

My reverse incrementals are very fast. It is my active fulls that I thought would be much faster. We are doing active fulls every weekend.

Test Job Notes
Test Job - 2 VM's, Server 2003 File Servers, job settings are at defaults.

Network Mode Backup Notes (Running Veeam 6.0)
Duration = 4:55
Processing Rate = 20 MB/s
Bottleneck = Source (Source 99%, Proxy 67%, Network 9%, Target 10%)
Processed = 341 GB
Transferred = 246.6 GB

Direct SAN Backup Notes (Running Veeam 6.1)
Duration = 3:05
Processing Rate = 30 MB/s
Bottleneck = Source (Source 99%, Proxy 90%, Network 14%, Target 14%)
Processed = 322.9 GB
Transferred = 235.2 GB

Questions
1. If I am using direct SAN backups now why is my bottleneck still the source? I assumed that direct san would be at least be 2x as fast as network mode. Is this normal for direct san speeds to only be approximately 10MB/s faster than network mode? I don't think so.

2. Why does veeam only use approximately 36 to 40 % of the NIC hardwired into the SAN? Is there something in the source code that limits nic usage? The support tech I worked with seemed to think so but wasn't 100 % sure. If I use a NIC Team and can get the iSCI initiator to talk to EQL SAN will veeam use more of that NIC than 36 to 40 % or would that be all for nothing if there is something in the code to limit that.

3. There are times when I run this test job and veeam consumes 100% of my CPU. I thought that this would be fixed when upgrading to veeam 6.1 but it is not. Is anyone else still seeing that issue? If I lower compression from Optimal to low then it drops by 30 to 35 % but it then increase my test job size by 70GB which won't work as I'm constantly fighting the space battle. Veeam support wasn't sure and I got tired of the back and forth so I closed the ticket (Ticket # 5206035). They did recommend an uninstall and reinstall of the software. Haven't tried that yet.

Thanks in advance for any suggestions, tips, best practices or answers.

[Vitaliy S.]

Hi Peter,

1. Since bottleneck just represents the "weakest" link in your environment, it is ok to have it on source side even if you're running direct SAN Mode. And yes, I would also expect the better performance rates for your backup job. Please review this topic for some tips and tricks that might help you.

2. No, we do not have any hard caps in the source code. Could you please tell me what Windows Server version do you have deployed on your proxy machine? If it is Windows Server 2008, then please take a look a this post that might give you a couple of ideas on how to improve the backup job performance.

3. 100% of CPU utilization is not an issue, it is absolutely expected (for the on the fly data compression an deduplication) to have high CPU utilization during backup/replication job runs.

Thanks!

[nmace]

Just a FYI, I successfully backed up a test VMFS volume last night using direct SAN access @ 200+MB/sec. This was an Equal Logic SAN with 10 gig interfaces and my Veeam physical server has a Broadcom 57711 10 gig card in a Dell R710. I'm using Dell drivers for the NIC dated 2/4/2011 driver version 6.2.9.0. I did use Dell's lifecycle management tool to update the firmware of the 10 gig card and raid controller to the newest version before I started. I don't know if that had any effect or not as I hadn't tried it before I updated the firmware. I'm not using Dell's MPIO software, just what ships with Windows Server 2008 R2. I'm also writing to local storage on the physical box, which is a raid 10 volume.

Just a thought to people experiencing poor performance, it might be worth checking the firmware of your iSCSI NICs and firmware for your Raid controller.

[paps40]

Vitaliy,
Thanks for the response. I was able to improve my Veeam Direct SAN Backup speed to 62 MB/s. Big improvement from 30 MB/s.

I did the following things:
1. Used the link you gave me to disabled auto tunning: netsh interface tcp set global autotuninglevel=disable (This only increased my speeds with my Dell EQL SAN by about 27 MB/s)
2. Updated the Intel Quad Nics with the latest drivers / PROset software.
3. Turned off file and print sharing on that nic connection.
4. Set Receive Side Scaling Queues from default value of 1 to 8 queues. (All those changes increased my speed by about 5MB/s more)

I tweaked pretty much every NIC setting and then re-ran my job after every change and this is what gave me the best performance. After 6 re-runs I settled on just leaving the nic at default settings but just adjusting the RSS Queues to 8. If anyone has any other NIC tweaks that have increased performance I would love to hear about them. I also spoke with Intel and Dell EQL but they didn't want to provide any insight and gave me the runaround. They just referred me to some support articles to read. Below are links to the intel site that go over what settings can be tweaked on the nic.

http://www.intel.com/support/network/ad ... 029402.htm
http://www.intel.com/support/network/sb/CS-025841.htm

[Andreas Neufert]

My 2 cents to this:

You can also try enabling disk alignment (Google) if you using Linux or Windows 2003 R2 (or older) as Repository.

Also check your NTFS formated block size vs. your Disk System block size.
If you have very big NTFS block size and very small disk system blocks => many drives are used to store one block => big Raid performance overhead
If you have very small NTFS block size and very big disk system blocks => only one drive will be used for multible blocke => Write performance are reduced to the performance of a single disk
A good size is NTFS 1/2 the size of a disk system block (1:1 isn´t best because you normally didn´t start to write things on the first bit of a disk storage block).

[J1mbo]

(1:1 isn´t best because you normally didn´t start to write things on the first bit of a disk storage block).

On 2k3 use diskpart to create the partition; create partition primary align=1024, then 64K NTFS block size could indeed be aligned with 64K RAID stripe size.

Interesting that Broadcom are singled out here. My own experience is similar; I don't know why Dell and everyone else don't just use Intel NICs for everything since they in my experience they always work well and rarely fail as well as being cheap and having near universal driver support, to get you going at least. Just MHO of course

[Andreas Neufert]

Hi,

NTFS blocks will not be filled 100% each time.
Storage Blocks will be filled 100%

So it happens than you write one NTFS block to more then one Storage block.
1:2 means that you have more chance to store one NTFS block on one storage block => Which ends up in better performance.

[J1mbo]

Why not; the minimum NTFS transfer unit is the cluster size?

[jkeegan]

if I'm doing VEEAM backups directly from the SAN (Dell Equallogic) where:

Veeam is a VM in Vsphere 5
VM has (2) dedicated Nics in the storage segment (separate physical segment)
The target for VM backups is a Buffalo NAS

Should the NAS be on the LAN segment or the STORAGE segment? Currently I have it on the LAN segment and I'm getting rates in the 60mb/sec with a Cisco Catalyst 6509 as the LAN switch, and pair of dell PowerConnect 5424 switches as the storage segment.

Thoughts?

[Vitaliy S.]

Hi Joseph, If you want to separate your backup traffic from the production network, then placing your NAS box on the storage segment will be a good idea. Anyway, if you're happy with your current job performance, you can continue using your current configuration. Just my 2cents.

[Abdullah.Amer]

Hello Everyone,

We are evaluating Veeam 9.5 Backup and Recovery to backup all our virtual environment, simply the infrastructure: 2 EXSi 6.0u3 hosts, Vcenter 6.0 and Equalogic SAN PS6210. Using BCM57810 10Gb Nic, Netgear XS70T 10Gb switch.

Using the 10Gb network, I am trying to achieve a kind of a high backup speed in Veeam B+R. all the recommendation around suggesting going to the direct SAN access will get the best performance in terms of backup speed.
Thanks for the guide under [FAQ] Frequently Asked Questions vmware-vsphere-f24/vmware-frequently-as ... tml#p39948 and to many urls like the justins IT blog http://www.jpaul.me/2010/08/the-missing ... n-backups/ ...

I configured a physical windows 2012 r2 server with 2 Nics (10Gb for Backup , second 1Gb for production), iSCSI initiator, access to the Equalogic SAN, "diskpart, automount disable", connect to a LUN ... I can see the disk at the windows management disk, run the backup and getting max speed of 200MB/s . followed this link veeam-backup-replication-f2/improving-d ... t4093.html "netsh, interface, tcp, set global autotuninglevel=disable" no change , some recommendation under this link http://datacore.custhelp.com/app/answer ... -practices ... no success neither. Therefore I have a 10Gb network, but just getting a 2Gb speed

Another test I mad is creating a VM Veeam B+R, getting 500MB/s. then created another VM Veeam on the other ESXi horst to use another Nic and resources and run 2 backup in parallel to the same target... I get 800MB/s at the target. this test is just to make sure there is no bottleneck at the source, destination, switch ...

So I think I am missing something in the physical SAN direct access method to get at least 800MB/s

much appreciated you efforts.

Kind regards,

[PTide]

Hi and welcome to the community!

Another test I mad is creating a VM Veeam B+R, getting 500MB/s.

First of all, could you please provide the bottleneck stats for the job?

Thanks

[Abdullah.Amer]

Hi PTide,

Many thanks for your quick reply,
for the physical SAN direct access 200MB/s test its "Load: Source 99% > Proxy 54% > Network 21% > Target 0%" ,

for the first VM Veeam 500MB/s:

3/17/2017 4:02:44 AM :: Load: Source 27% > Proxy 35% > Network 98% > Target 5%
3/17/2017 3:47:38 AM :: Load: Source 32% > Proxy 35% > Network 54% > Target 32%
3/17/2017 3:41:40 AM :: Load: Source 53% > Proxy 53% > Network 80% > Target 34%
3/16/2017 4:45:25 PM :: Load: Source 16% > Proxy 44% > Network 72% > Target 51%
3/16/2017 4:47:16 AM :: Load: Source 14% > Proxy 45% > Network 77% > Target 43%
3/15/2017 4:38:35 AM :: Load: Source 15% > Proxy 45% > Network 66% > Target 53%
3/12/2017 11:42:25 PM :: Load: Source 21% > Proxy 76% > Network 76% > Target 34%

Cheers,

[PTide]

The problem that you experience resembles another problem with Equallogic and Broadcom (exactly the hardware that you have). Please carefully review this thread.

If none of the mentioned fixes (including firmware updates) help, then please contact our support team directly for a deeper investigation and post your case ID.

Thank you