Poor Performance with Direct SAN Access

[a_vila]

Hi guys,

I'm testing backup with Direct SAN Access but I find the performance really poor compared to what I get with Virtual Appliance.
Let me describe the infrastructure:
We have a VMware environment with MS Windows VMs mainly, everything sits on two storage devices, one is Pure and one is PowerStore. We have Veeam 12 (with all the patches, soon to be upgraded to 13). Backup Server is on a VM and also many proxies, transport mode is majoritary Virtual Appliance (there are a few agents). There are some big VMs, one for example has 3 vdisks, one of them is 3TB, the other vdisks are around 400 MB. Full backup of this VM usually takes around 20min with virtual appliance, the processing of the largest disk reaches 2GB/s.
On the other hand there's a physical server we're implementing, MS Windows, 32-cores, 64GB RAM, 16GB 2-port FC card, around 40TB on SSD disks. It has vmware proxy and repository roles. SAN access has been configured properly, MPIO configuration is according to the storage devices' best practices and backups using this proxy show [san] transport mode. We are testing the previously described VM, it starts processing all three disks but total throughput only reaches ~600 MB/s. As soon as smaller disks finish processing, the biggest disk continues with ~200 to ~300 MB/s and backup job takes around 3 hours to finish.
We already tried changing block size in the backup job, throughput could reach around 350MB/s and backup duration was around 2.5 hours.
We tried changing queue depth to the maximum value but no improvement was seen.
We opened a case with veeam, they told us to test the server-storage speed with diskspd, and default values (as suggested) also reach around 300MB/s, so they reported that there is no problem with veeam.
We opened a case with Cisco (MDS), they said there is no congestion. We opened a case with Pure, they said MPIO configuration is correct. We opened a case with Microsoft, they said there's no problem with windows, no disk failures, no SAN conectivity failures detected and actually pointed out that apparently veeam uses the queue depth value of 1 which limits performance to 160MB/s so there's nothing to be done...
So I wanted to ask for your experience on Direct SAN backups, is it expected for performance to be lower than virtual appliance? did you have some similar performance issue? how did you solve it? is there anything you suggest?

Thanks in advance,
A

[Andreas Neufert]

Direct SAN is expected to be 2-4x slower than our own HotAdd processing at backup and a magnitude slower on restore.

The reason for this is that for Direct SAN we as backup vendor do not understand the disk format and so we need to use the VMware VDDK kit for reading. It performs the task in a syncronous way with a queue depth of 1. So there is not much the storage controller can optimize. With HotAdd mode we completely go around the VMware libraries and read natively from the storage with a higher queue depth. Which allows the storage to optimize the backup speed (specifically when under more load or random IO processing is needed for incremental backup).

For VMware the backup the "Local 1MB" block size is the optimal one in case of performance vs. storage space for incrementals.