Slower SAN speed than expected

[mblackmore]

Hey Guys

Got a FULL ESX System, all 10GBe with DELL Equalogic SAS San Arrays, all connected via 10GBe also into DELL 8024f switches, which are stacked together using 4x 10gbe links

Installed Veeam onto a Physical dell r710 (Dual Quad core, 16gb ram, Raid 5 array) and configured a local backup job to grab snapshots and save them on a local drives (raid 5 array)

The backup server is connected via 10gbe to the SAN network and ISCSI iniator has been setup, all the volumes from the san now show in windows disk manager, however i did not give them drive letters

The backup job is setup to grab all our important prod VMs (Around 20), VSS is on and quiescince is OFF, This job goes at around 40MB/s which is under half of what you would expect on a 1GB/s link, definetly no where near a 10gb/s link

IF i create another backup job, just to test, using same config, but only selecting 1x Prod VM in the job to grab it goes at around 900 - 1000 MB/s (Read: Megabytes) which is definetly what i would expect from a 10gbe network

Can anyone maybe see why im getting such a slow speed on my initial backup job

[Vitaliy S.]

Hello Michael,

I would expect seeing 900-1000 MB/s figures during the incremental pass using vStorage API CBT, but not for a full run. By the way, have you had a chance reviewing a sticked thread that has some recommendations on improving backup speeds for DELL Equallogic iSCSI SAN?
Improving direct-from-SAN backup speed with iSCSI SAN

Thanks!

[Gostev]

Processing rate may vary heavily from VM to VM dependins on the amount of white space in VMDK, so jobs with different VMs should not be compared directly. But I agree, the speed should be much faster than 40MB/s even in worst case scenario with no white space in VMDK. As shown in the thread referenced above, in properly tuned environment a single job can fully saturate iSCSI adaptor.

This could be target storage performance issue as well, see here for more information:
How to improve backup speed: backup storage performance

[mblackmore]

Thank you for your replies, yes i have read those and ensured those modifications are set.

Intersting last nights backup report as follows,



I had to blank them out, but ill call them a number based on their order

1 - P2V'd Windows 2003 SP2, single volume, server went at 120mb/s (terminal server)
2 - P2V'd Windows 2003 SP2 server, 3 volumes, only at 77mb/s (app server, sql, corporate shared drive)
3 - P2V'd Windows 2003 Server SP2, 2 volumes, went at 31mb/s (app server, sql)
4 - V2V'd Windows 2003 Server SP2, 1 Volume, went at 579mb/s (app server, sql)
5 - V2V'd Windows 2003 Server, 1 Volume, went at 182mb/s (internal web)
6 - P2V'd Windows 2003 Server, 2 Volumes, went at 61mb/s (Backup domain controller, profile server, print server)
7 - P2V'd WIndows 2003 Server, 1 volume, Failed (web server)
8 - Fresh Server created from Template, Server 2008 R2, 1 volume, went at 350mb/s (domain controller)
9 - P2V'd Windows 2003 SP2 server, 2 volumes, went at 60mb/s (app server, sql)
10 - P2V'd Windows 2003 SP2 server, 2 volumes, went at 36mb/s (app server, sql)
11 - P2V'd Windows 2003 SP2 server, 2 volumes, went at a whopping 4mb/s (exchange)

[Gostev]

Interesting, it looks like every VM created via P2Ved has awful performance, while "normal" VMs get expected performance. Differences are pretty dramatic... I am starting to suspect some P2V issue around how it creates disks...

[mblackmore]

Thanks

Do you have anything i could try? because we just purchased the product and would like it to work as it should

[nielselfrink]

Hi mblackmore,

Did you by any change find out what caused the slower speeds with p2v-ed machines?
we are also suffering from the same problem in our test environment.

We use direct SAN access (1Gb link) and get speeds varying from approx 16MB/s on most P2v-ed machines and up to 90MB/s on newly installed vm's, which is an expected speed with software initiators i would guess .this is all for NEW/full backups, imcrementals run at higher speeds as expected.
would love to hear if you or anyone ever found out a solution?

Regards,
Niels

[jlehtinen]

Out of curiosity, what is the SCSI/HDD adapter on your P2V machines? Depending on what software you used for the P2V, I have seen cases where a P2V'd VM uses IDE-based HDD adapters. Might be performance loss there, or poor emulation? Do the non-p2v'd VM's have a different adapter in use?

Other thing - Server 2003 is almost never aligned properly in a vmware environment. NTFS uses the starting block at 32kb, which staggers all your data off across the 64kb vmfs blocks. Depending on your physical block size, OS block size, and RAID/LUNs setup you can see huge performance drags due to lack of alignment, because the SAN has to perform seeks across extra blocks. This should really only be an issue on random read/write and not sequential read/write like you would get in a backup operation, but it might be worth checking.

[jnalldr]

I'm seeing similar results here. Freshly built servers (were never physical) seem to back up and a substantially faster pace than P2V boxes. I have a 2k3 backup job running right now and almost all of the VMs were physical at one point. The speeds I'm seeing are 9 MB/s, 5 MB/s and 18 MB/s for the first 3 servers in the job. I was thinking there was something wrong with my setup somewhere but none of my backup environment is being taxed. CPU is at 50%, network is at 10 - 20% and the target disk isn't seeing high I/O. I think jlehtinen is on to something though with the virtual storage adapter that the P2V'd machines use.

[jlehtinen]

I went through and checked my VM's. Ones that I have P2V'd with VizionCore's Vconverter had the IDE HDD adapter.

However, when checking speeds, I could not locate any issues related to the adapter. Backup jobs, with NAS as target, ran between 40mb/s and 90mb/s with no real difference between P2V or non-P2V VM's. Replica jobs, with 2nd ESX host as target, ran between 60mb/s and 400mb/s with P2V VM's actually getting the best speeds.

Based off this, I kind of doubt the IDE adapter is the culprit here. One thing I will say is that my backup/replication jobs are all set up separately, they're staggered to not run at the same time as any other network-intensive task (i.e. a cloud sync we run) and they all run afterhours (midnight-4am or so). The jobs in the screenshot above are running right through the middle of the day. I don't know about his infrastructure, but I know that if I ran jobs during those times I would see very poor/sporadic performance from the normal traffic our network has during business hours.

[derekf]

To test it out, could you clone the VM and change the disk adapter (or try copying the data to a different sort)>

[tsightler]

Any possibility that this is related to partition alignment? Perhaps VM's that were migrated with P2V tools are poorly aligned since most people didn't pay much attention to alignment issues before SAN storage.