Comprehensive data protection for all workloads
Post Reply
jreid
Influencer
Posts: 10
Liked: never
Joined: Aug 27, 2009 3:33 pm
Full Name: Jason Reid
Contact:

Backup Speeds on iSCSI with SATA

Post by jreid »

Hi,

we have a HP MSA2012i iSCSI SAN with two SATA arrays (11 disks per array).
The drives are all SATA as this is for a DR platform.
They are connected to a series of esx3.5 servers via two dedicated Cisco 2950G switches with 4 x 1000TX links to the MSA and 2 x 1000TX links per esx host.
The hosts are used only as service console gateways so run no live VM's.

When replicating over a Gigabit link I am getting initial pass speeds of 12MB/s.
On the incremental/synthetic pass it increases to 20MB/s.

Both of these readings seem alot lower than I would expect based on other posts, etc.
Does anyone have any experience to confirm that this is low?? Am not sure if the HP MSA2012 is the issue, or SATA drives.

Any help or advice on this is most welcome....

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Gostev »

The fact that both initial and incremental pass speed is slow points at issue with source storage access speed. What mode are you replicating in, VCB or Network? I may have some suggestions depending on that.

pizang
Enthusiast
Posts: 61
Liked: never
Joined: Aug 02, 2009 7:33 pm
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by pizang »

It is not a SATA isue. We have iSCSI SAS source and iSCSI SATA target (QNAP 639 with 6x1,5TB in RAID5). Yesterday our incremental fileserver backup showed 115MB/s!

jreid
Influencer
Posts: 10
Liked: never
Joined: Aug 27, 2009 3:33 pm
Full Name: Jason Reid
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by jreid »

We are using the network method for back/replication.

The servers are connected by 1Gbit/s NIC's which in turn runs over a 10Gbit/s LAN.

Thanks

Jason

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Gostev »

Jason, what speed are you getting when downloading VMDK file from you source ESX storage using Vmware Infrastructure Client's datastore browser? Also, do you have service console connection enabled in all ESX hosts' properties to allow service console agent to install (this provides increased performance comparing to agentless mode).

Evgeny
Influencer
Posts: 10
Liked: never
Joined: Sep 09, 2009 9:40 am
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Evgeny »

Hi,

I have simmilar situation. We have two iSCSI SANs with SAS and SATA drives, obviously SATA for DR.
VCB proxy/Veeam backup is virtual server.
I've tried to do backup on the same SATA SAN from one folder to another and maximum I could get is 20MB/s.
Any thoughts?

Evgeny

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Gostev »

Evgeny, this would be different situation - if you are using shared iSCSI storage and VCB SAN backup mode, the backup is done directly from storage, and backup traffic does not go through ESX host as it is the case with network backup mode Jason uses.

I assume that you are using VCB SAN mode via software iSCSI initiator bundled with Windows, is that correct? What is the Windows version and service pack level for Veeam server? And what is the compression level in the job settings.

Evgeny
Influencer
Posts: 10
Liked: never
Joined: Sep 09, 2009 9:40 am
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Evgeny »

Hi Anton,

Thank you for your reply.
You correct, Data shouldn't leave MSA. But for some reason speed is not good.

We are using MS iSCSI which we download from MS website.
VCB Server:
Windows 2003 R2 SP2
Veeam 3.1 is on the same box as VCB.
Compression level: Optimal
=== Job VCB SAN mode
Name: Backup Job 2
Type: Backup (VCB)
Target host: sol-esx02
Destination: "/vmfs/volumes/restore/Backup/SAN test.vbk"
Command line: "C:\Program Files\Veeam\Backup and FastSCP\VeeamManager.exe" backup 30fac3ee-7d3f-48bb-9338-b68ccb87a30e
===

Evgeny

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Gostev »

Evgeny, if you launch the vcbmounter.exe command from Veeam logs (Help | Support Information) manually on Veeam server, what performance do you get? Just change the destination path in command to the real destination you are backing up to (instead of folder with GUID), and provide real username/passwords instead of ******.

Evgeny
Influencer
Posts: 10
Liked: never
Joined: Sep 09, 2009 9:40 am
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Evgeny »

Anton, not sure what would be my destination path. Is there is any way to find it out?
in ESX it is /vmfs/volumes/49b1c1f7-42ed60e6-c392-001cc4e2c2b2/Backup

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Gostev »

Evgeny, you can just use the local folder on you Veeam server for simplicity here - we want to verify VCB data retrieval speed with this test.

Evgeny
Influencer
Posts: 10
Liked: never
Joined: Sep 09, 2009 9:40 am
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Evgeny »

I create aditional harddisk on same LUN storage as backup VM for VCB/Veeam vm. Direct backup with vcbmounter to that hdd.
run veeam and vcb on it's own and compared speed
with direct vcbmounter speed increased by 10-15%

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Gostev »

So the speed is still 22-23 MB/s?

Evgeny
Influencer
Posts: 10
Liked: never
Joined: Sep 09, 2009 9:40 am
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Evgeny »

Yes, about this.
that what is showing in Veeam software. Not sure if this is correct. Is there is any software you can recomend to monitor bandwidth during backup? Or it matter of Calculator? :)

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Gostev »

Yes, and this is exactly why I asked you to perform the separate test with Vmware vcbmounter utility - to completely exclude our software from equation.

Evgeny
Influencer
Posts: 10
Liked: never
Joined: Sep 09, 2009 9:40 am
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Evgeny »

Ok,
when I run vcbmounter I run utility which showing me aproximate bandwidth usage on the nic.
So by comparing both VCBmounter and Veeam, Bandwidth usage with vcbmounter was greater by 10-15% while speed shows in Veeam was 8MB/s

Gostev
SVP, Product Management
Posts: 26679
Liked: 4268 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by Gostev »

10-15% difference is expected, because with vcbmounter testing you are backing up locally and without any backup file processing done.

However, the speed you are getting with vcbmounter is very low - you should look into your storage infrastructure and troubleshoot this. Consider using physical server with hardware iSCSI HBA, since software iSCSI initiator could potentially be a limiting factor here.

tsightler
VP, Product Management
Posts: 5675
Liked: 2486 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Backup Speeds on iSCSI with SATA

Post by tsightler »

You should definitely be able to do better that 20MB/sec but I do think performance on iSCSI is somehow limited compared to other setups, perhaps because of a limited queue depth and the higher latency (relatively) of iSCSI of fiber channel solutions (just a guess). We are a 100% iSCSI shop and run Veeam on a server that's about 5 years old, so certainly not ideal. We average about 40MB/sec for most backups, sometimes a little higher, and a good bit lower for systems with a high change rate. The only systems that show very high speed (100MB/sec or more) are system with a large percentage of free space. We can run multiple backups simultaneously on our host and get over 200MB/sec pretty much saturating the two hardware iSCSI HBA's in our setup. Software iSCSI might have a little more overhead, but I don't think that's a major limiting factor. In initial testing we ran Veeam in a VM, using the Microsoft iSCSI client, and were actually seeing speeds in the 50-55MB/sec range, slightly faster than native hardware, but our VMware servers are a much newer hardware and were much faster than the physical servers we were testing with, so those speed differences are probably to be expected.

I know nothing I said above really helps you, but hopefully it gives you a least a little idea of what speeds could be and lets you know there are setups that are seeing better that 20MB/sec.

Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Google [Bot] and 32 guests