How could NBD possibly be faster than SAN?

[cparker4486]

Hi there,

tl;dr Running the same job twice, I got better performance from a worse configuration.

Source:

• Dell Equallogic PS4000X
• RAID 50 (14 x 450GB 10K SAS)

Target on run 1:

• Dell Precision T7400
  • RAID 1 (2 x 1TB Samsung 850 EVO SSD)

Proxy on run 1:

• VM on ESXi host
  • Has direct SAN access
  • 8GB RAM
  • 4 cores @ 2.6GHz
    • All cores utilized

Target on run 2:

• Same as above but reconfigured to RAID 0 because RAID 1 was too small

Proxy on run 2:

• Dell Precision T7400 (same machine that hosts storage)
  • 4 cores @ 2.0 GHz
    • All cores utilized
  • 4 GB RAM
  • Windows 8.1 Enterprise
  • Network access

The story (in bulleted format for clarity):

• I did not change the job's setup between runs except for which proxy was to be used.
• The Source was the bottleneck in both cases.
• I deleted the data on disk before each run so that it would start from scratch.
• I started each run at about the same time in the morning (yesterday and today.)

First run:

• Processing rate: 47 MB/s
• Duration (stopping due to disk space): 10 hours 17 minutes
• All but one VM ran as [san]. The other ran as [hotadd].
  • The reason for this is because that one VM has its own datastores and I forgot to give the proxy iSCSI access to the volume.
• The one VM that ran under [hotadd] had a significantly higher average processing rate than any other VM (24.5MB/s vs ~11MB/s)
• Load: Source 86% > Proxy 42% > Network 10% > Target 17%

Second run:

• Processing rate: 80 MB/s
• Duration: 6 hours 28 minutes
• All VMs ran as [NBD]
• Load: Source 93% > Proxy 83% > Network 27% > Target 19%

On paper the 2nd run should have been significantly slower.

[emachabert]

Without having more information regarding your setup, we can't give useful information.
You should explain:
- how is setup the iscsi network Inside and outside the ESX server (cabling, port config, initiator type, jumbo config etc.)
- how is the VM connected to the iSCSI network (vmnic type etc, ,dedicated vnic ?, vswitch etc.)
- how is setup the MPIO in the iSCSI proxy vs the selected PSP in the ESX
- how is setup the management network Inside and outside the ESX server, through which you did the NBD run