Optimal speed - vmware vs storage?

[colohost]

Hi all, we have a vsphere 7 environment with Nimble iSCSI storage. We have VB&R 11 that has a Windows-based proxy VM only able to do vmware-based backups, and have recently added a linux proxy VM which is able to also do storage-based backups. We altered one of our backup policies to allow storage-based backup, added in the new proxy as an option, and gave the linux proxy VM interfaces on the storage vlans. We've not tuned the maximum backups per storage snapshot parameter yet; default of zero, which perhaps means do the entire environment on one storage snapshot if there's not a different internal default. The linux VM is using vmxnet3 and jumbo frames on its iscsi vlan interfaces, along with kernel tuning designed for optimal throughput on 25gig interfaces.

We're seeing about 15% reduction in backup performance when using the linux proxy and storage-based backups.

My possibly incorrect understanding of what's going on behind the scenes is with vmware backups, the VB&R master will instruct vcenter to take snaps, then a proxy is attached to the relevant disk, it's reading by way of raw file I/O and vsphere is obviously doing the real i/o to the storage, in what could possibly be an optimal manner if there are storage-specific plugins in play, such as Nimble connection manager. Upon completion of backup, the VM snap is deleted. So, during backup, performance for that VM could be degraded given it's running with a snapshot.

Then, with storage-based backups, VB&R master instructs vcenter to snapshot the relevant number of VM's (perhaps all if set to zero?), upon completion, takes a storage snapshot, deletes all the VM snaps returning VM performance to normal, and the proxy is presented the storage snap over iSCSI to back all VM's up.

If all the above is correct, I'm just curious if I should spend any time hunting down the performance difference or if there could be other factors involved and this is expected. Obviously a vsphere host with storage client is probably heavily optimized compared to a random linux VM talking through the host to the same storage; perhaps that's the initial difference. But, if that's the case, is there optimization to be had at Veeam proxy level short of deploying it as a bare metal proxy?

[PetrM]

Hello,

Do I understand correctly that you observe a performance degradation after switching to the backup from storage snapshots from HotAdd mode? If yes, are you talking about backup speed or VMs performance reduction? If it's about backup speed, I would check iSCSI channel performance, most probably the Source will be a "bottleneck" but it would be best to check through the backup job statistics.

If VMs become slower, then I'd compare the source storage latency during backups in both (HotAdd and storage snapshots) modes and ESXTOP output.

Thanks!

[colohost]

Hi, yes, 15% reduction in total throughput of the backup job compared to prior full backups. The backup is about 50 TB for ~700 VM's, and the total time taken is about 15% greater after having switched to backup from storage snapshots, from what had been the VM-based snaps and hot add on the proxy.

I could tell from the array and the proxy VM side that two iscsi paths were connected, but I do not know if both were being utilized. Our hosts 2x25 Gbit LAGs for storage and 2x25 Gbit LAGs for "data", and the proxy VM would be utilizing the data interfaces since it's a VM. The links are very under utilized so it isn't a physical capacity issue.

Does the Veeam proxy agent just use native Linux iscsi client software to establish the connections to the arrays for the storage snapshot access? Would there potentially be opportunity to tune those settings somehow? Like a config or mount options that Veeam specifies, or does it use the defaults the operating system would have in place for iscsi client connections? Just trying to figure out where to start looking on what I can tune.

[MaartenA]

You should test it out with a Windows VM and install the nimble connection manager on it and see how it goes. If you like the performance then switch to a physical box as a proxy/reprository so it will even be better.

[colohost]

Ah okay, so if NCM is present, the proxy software is establishing iscsi connections in a manner that would utilize it, vs just calling OS level commands?

[MaartenA]

You can do it without the NCM but i do recommend to use it because it would optimize your settings and multipathing to your Nimble (you just need to enable MPIO and add your Nimble group IP's)
The proxy will just mount the storage snapshot based on the VM snapshots it took before so it knows what snapshot it needs to mount for your VM's to backup. Just spin up a Windows VM and try it out.

We just switched from virtual proxy to physical proxy and repository on one box and the performance is just great. Virtual proxy performance was OK but physical is even better because you will bypass the VMware part.

and a leasons learned make sure your ACL's are setup right for your ESXi hosts on the nimble side (Volume only instead of volume & snapshot). In earlier versions from the vCenter nimble plugin it would create new volumes with the ACL Volume & snapshot in the latest version they changed it to the right ACL Volume only for new volumes.

[colohost]

Thanks; will give it a try. We've only used ncm on the vsphere side so far but makes sense.