SAN read buffer size

[Nils]

When testing our SMB storage with large I/Os I found that vDisk read performance could be increased quite a bit (1 Gbit/s -> 3 Gbit/s with 32+ MB buffers, measured simply with dd inside a guest).

In order to decrease our backup window I was looking for increasing SAN read buffers for the backup proxy and all I could find was an ancient thread. The mentioned registry value VddkPreReadBufferSize doesn't seem to do anything.

Listening in to VeeamAgent.exe with ProcessMonitor revealed the value VmfsSanPreReadBufSize which does change VMFS SAN PREREAD BUFFER SIZE according to the log. Is this what I'm looking for? Is there any documentation for those registry options?

(In case you're interested: we're using a Dell MD3220i iSCSI array with 4x Gigabit to each host.)

Nils

[Vitaliy S.]

Nils, if you want to decrease your backup window, can you please tell us your current bottleneck stats for the backup job?

[Nils]

In order to get a meaningful figure I deactivated CBT and ran an incremental job (VMfsSanPreReadBufSize=32MiB) - 3.8 TB read, 80 GB written. Load: Source 98% > Proxy 50% > Network 16% > Target 0%. Before improving the setup, the proxy used to be the bottleneck (the backup server's Nehalem QC). I'm now using a virtual proxy with 8 K10 cores and 4x1 Gb SAN access (backup server has 1xGb). With an active full backup the new setup promptly overwhelmed the Gb downlink to the repository but that may change in a few months. Generally, I'd like to know what our scenario can scale up to and wondered why these parameters are nowhere to be found.

I'll repeat with default VMfsSanPreReadBufSize (4 GiB) tonight and post the result.

[rreed]

Nils, out of curiosity, what is your 1Gb switch that you overwhelmed please? Were you killing the output buffers?

[Nils]

The switch isn't the problem - if you read 4 Gbit/s and funnel the output through 1 Gbit/s the load is bound to overrun any amount of buffering you can have. I used "overwhelmed" in terms of our PCM throwing warnings for exceeding 98% thresholds. The connection itself runs fine and without problems. Note that's it's not even an iSCSI connection but the SAN proxy to repository link.

The backup server/repository is located ~2 km from the SAN and currently we've only got a single 1000BASE-LX link running for everything. (By the way, VoIP w/ QoS on its own VLAN worked 100% fine while running at 99.5%+ during the backup; flow control is deliberately turned off on the link.)

[rreed]

Exactly, and you should try funneling 10Gb through a single 1Gb connection. Very nicely done, Nils.

[Nils]

You've got to take what you get...

Running the incremental job with CBT turned off and 4 Mib default prebuffer: Load: Source 98% > Proxy 51% > Network 15% > Target 0% (3.8 TB read, 60.6 GB written) - apparently the preread buffer setting has no impact on the actual speed; I'll stick to the default size then.