Backup Speeds on iSCSI with SATA

[jreid]

Hi,

we have a HP MSA2012i iSCSI SAN with two SATA arrays (11 disks per array).
The drives are all SATA as this is for a DR platform.
They are connected to a series of esx3.5 servers via two dedicated Cisco 2950G switches with 4 x 1000TX links to the MSA and 2 x 1000TX links per esx host.
The hosts are used only as service console gateways so run no live VM's.

When replicating over a Gigabit link I am getting initial pass speeds of 12MB/s.
On the incremental/synthetic pass it increases to 20MB/s.

Both of these readings seem alot lower than I would expect based on other posts, etc.
Does anyone have any experience to confirm that this is low?? Am not sure if the HP MSA2012 is the issue, or SATA drives.

Any help or advice on this is most welcome....

[Gostev]

The fact that both initial and incremental pass speed is slow points at issue with source storage access speed. What mode are you replicating in, VCB or Network? I may have some suggestions depending on that.

[pizang]

It is not a SATA isue. We have iSCSI SAS source and iSCSI SATA target (QNAP 639 with 6x1,5TB in RAID5). Yesterday our incremental fileserver backup showed 115MB/s!

[jreid]

We are using the network method for back/replication.

The servers are connected by 1Gbit/s NIC's which in turn runs over a 10Gbit/s LAN.

Thanks

Jason

[Gostev]

Jason, what speed are you getting when downloading VMDK file from you source ESX storage using Vmware Infrastructure Client's datastore browser? Also, do you have service console connection enabled in all ESX hosts' properties to allow service console agent to install (this provides increased performance comparing to agentless mode).

[Evgeny]

Hi,

I have simmilar situation. We have two iSCSI SANs with SAS and SATA drives, obviously SATA for DR.
VCB proxy/Veeam backup is virtual server.
I've tried to do backup on the same SATA SAN from one folder to another and maximum I could get is 20MB/s.
Any thoughts?

Evgeny

[Gostev]

Evgeny, this would be different situation - if you are using shared iSCSI storage and VCB SAN backup mode, the backup is done directly from storage, and backup traffic does not go through ESX host as it is the case with network backup mode Jason uses.

I assume that you are using VCB SAN mode via software iSCSI initiator bundled with Windows, is that correct? What is the Windows version and service pack level for Veeam server? And what is the compression level in the job settings.

[Evgeny]

Hi Anton,

Thank you for your reply.
You correct, Data shouldn't leave MSA. But for some reason speed is not good.

We are using MS iSCSI which we download from MS website.
VCB Server:
Windows 2003 R2 SP2
Veeam 3.1 is on the same box as VCB.
Compression level: Optimal
=== Job VCB SAN mode
Name: Backup Job 2
Type: Backup (VCB)
Target host: sol-esx02
Destination: "/vmfs/volumes/restore/Backup/SAN test.vbk"
Command line: "C:\Program Files\Veeam\Backup and FastSCP\VeeamManager.exe" backup 30fac3ee-7d3f-48bb-9338-b68ccb87a30e
===

Evgeny

[Gostev]

Evgeny, if you launch the vcbmounter.exe command from Veeam logs (Help | Support Information) manually on Veeam server, what performance do you get? Just change the destination path in command to the real destination you are backing up to (instead of folder with GUID), and provide real username/passwords instead of ******.

[Evgeny]

Anton, not sure what would be my destination path. Is there is any way to find it out?
in ESX it is /vmfs/volumes/49b1c1f7-42ed60e6-c392-001cc4e2c2b2/Backup

[Gostev]

Evgeny, you can just use the local folder on you Veeam server for simplicity here - we want to verify VCB data retrieval speed with this test.

[Evgeny]

I create aditional harddisk on same LUN storage as backup VM for VCB/Veeam vm. Direct backup with vcbmounter to that hdd.
run veeam and vcb on it's own and compared speed
with direct vcbmounter speed increased by 10-15%

[Gostev]

So the speed is still 22-23 MB/s?

[Evgeny]

Yes, about this.
that what is showing in Veeam software. Not sure if this is correct. Is there is any software you can recomend to monitor bandwidth during backup? Or it matter of Calculator?

[Gostev]

Yes, and this is exactly why I asked you to perform the separate test with Vmware vcbmounter utility - to completely exclude our software from equation.

[Evgeny]

Ok,
when I run vcbmounter I run utility which showing me aproximate bandwidth usage on the nic.
So by comparing both VCBmounter and Veeam, Bandwidth usage with vcbmounter was greater by 10-15% while speed shows in Veeam was 8MB/s

[Gostev]

10-15% difference is expected, because with vcbmounter testing you are backing up locally and without any backup file processing done.

However, the speed you are getting with vcbmounter is very low - you should look into your storage infrastructure and troubleshoot this. Consider using physical server with hardware iSCSI HBA, since software iSCSI initiator could potentially be a limiting factor here.

[tsightler]

You should definitely be able to do better that 20MB/sec but I do think performance on iSCSI is somehow limited compared to other setups, perhaps because of a limited queue depth and the higher latency (relatively) of iSCSI of fiber channel solutions (just a guess). We are a 100% iSCSI shop and run Veeam on a server that's about 5 years old, so certainly not ideal. We average about 40MB/sec for most backups, sometimes a little higher, and a good bit lower for systems with a high change rate. The only systems that show very high speed (100MB/sec or more) are system with a large percentage of free space. We can run multiple backups simultaneously on our host and get over 200MB/sec pretty much saturating the two hardware iSCSI HBA's in our setup. Software iSCSI might have a little more overhead, but I don't think that's a major limiting factor. In initial testing we ran Veeam in a VM, using the Microsoft iSCSI client, and were actually seeing speeds in the 50-55MB/sec range, slightly faster than native hardware, but our VMware servers are a much newer hardware and were much faster than the physical servers we were testing with, so those speed differences are probably to be expected.

I know nothing I said above really helps you, but hopefully it gives you a least a little idea of what speeds could be and lets you know there are setups that are seeing better that 20MB/sec.