iSCSI (Equallogic) performance

[dbrett]

Like most I was greatly anticpating the release of v4, but, I'm after testing last night it seems I'm having issues with the speed of the full backups. I suspect this is down to setup rather than Veeam but I know a couple of EQL users are having good results so thought I'd post up.

I did a couple of test backups of complete VM's, one via network, and one via SAN. The network backup returned 15MB/sec, the SAN backup returned 17MB/sec without jumbo frames enabled and 19MB/sec with jumbo frames. Based on previous forum posts I'd have thought the SAN backup should've returned two or three times more throughput.

Network connectivity is via 1Gbps connections to a Cisco 2960G, SAN connectivity is via 1Gbps connections to a Cisco 3750.

At the moment I have only cabled up one connection into the SAN to avoid any potential multipathing issues. I can't think of any particular reason why the performance should be so slow, the only thing I can think of is that since the Equallogic group consists of two arrays then I should possibly cable up two connections so the array can load balance.

Can any EQL users offer some tips?

Thanks.

[McClane]

What compression level was used? I have problems on my rather old and slow backup server with compression set to "Optimal".

[dbrett]

Compression was set to 'optimal' too. - I'll change that and see what happens.

[tsightler]

I'm an EQL user. We just installed Veeam 4 on an old Dell 2650 so not much CPU horsepower. We're using vStorage API mode and yesterday, around 5PM, I started two backups that contain a total of about 40 VM's that are almost 4TB of total storage. We're using optimal compressions and seeing 25-30MB/sec per backup, so a total of about 60MB/sec. We're backing up to a Linux target at a remote site (connected via 1Gb MetroE) and we're seeing about 70-90% CPU utilization on the Veeam server, and about 50% CPU usage on the Linux host, a Dell 2950 with 2Ghz quad-core processor.

This is obviously a little slow, but we think it will be plenty fast for the incremental runs so we're going to wait and see. With Veeam 4 "Optimal" is much more of a CPU hog than with Veeam 3. My understanding is because, with Veeam 3, "Optimal" meant optimal performance when used in network backup mode as this placed minimal load on the ESX host. With Veeam 4, "Optimal" seems to have a new definition of being "Optimal" for vStorage API mode with backup server that has a modern processor. In our test this means that "Optimal" compression in Veeam 4 preforms similar, although not quite as good, as "Best" compression on Veeam 3, but needs a powerful CPU.

[dbrett]

Hmmm, just tried a quick lunchtime run using 'low' or 'best' compression. 'Low' gave us 10MB/sec, 'best' was around the 15MB/sec mark.

Tom, thanks for the information. Are you using a single NIC into your storage? And do you have more than one EQL array in the group? The backup server is a DL380 G4, dual Xeon 3Ghz. Not blindingly fast but similar to a 2650.

I noticed that the network performance was spiking between 25% and 50% almost continuously, that may be indicative of CPU issues (ie drag down some data, process the data and repeat) but equally could be something else. I'll have to dig out perfmon and see if the issue is local or elsewhere.

[mealan]

We are getting 50-55MB/s on our ps5000e,

the veeam backup server is a VM living on local storage....


do you have jumbo frames on? are your vm's or vol thin pro?

[tsightler]

I'm actually using a dual port Qlogic iSCSI HBA and Equallogic's host integration tools with load balancing. We have multiple arrays, but LUN's are not spread across them. The two backups I mentioned above were both running from the same PS400E (I guess that's now a PS5000E with 750GB SATA drives). If I just run a single backup we generally see 45-50MB/sec, so I think we're probably topping out the backup hardware.

Our Veeam 3 server was running on an IBM HS20, which has a little more horsepower than the Dell 2650, but not a lot. Using VCB we would typically see 30-35MB/sec, but with optimal compression we could run 6 jobs in parallel and get reasonably close to topping out the 2 HBA ports at about 180MB/sec on that single host.

[dbrett]

Thanks guys - I think I've identified the issue but am just about to run another test. Looks like the disks on the backup server are flat out so am going to switch to different storage and re-run the tests.

Jumbo frames are on, as is flow control, all vm's are converted from v4 hardware and thick provisioned.

I'll run some more tests and see how we get on, but I think the backup server is the cause.

[dbrett]

Changed the storage and now seeing 30MB/sec for full backups with a single nic.

Incrementals running at 170MB/sec.