VStorage API: Network vs. Virtual Appliance mode

[sbrumlwy]

Hi I am in the process of evaluating Veeam B&R 4.1 and am very impressed. I will most definitely be a customer in the next week or so. I have a question about the Vstorage API mode. My set up is that I am running Vsphere 4 w/ ESX 4.0 U1 hosts (3 hosts) using an EqualLogic PS4000 for shared storage. All my VM's reside on the EQL. I am backing up to an Iomega StorCenter ix4-200d NAS box. I have Veeam installed in a VM.

I use two networks in my VSphere set up - the LAN (which also has the service consoles, and to which all my VMS are connected) is on 172.30.0.0 subnet. The SAN, to which the EQL and the Iomega are connected, is a 192.168.20.0 subnet.

I have tested multiple configurations for performance. I will not go into all the combinations I have tested, but a few examples are, backing up to a CIFS share on the NAS, creating an NFS data store on the NAS and backing up to the vmfs directly or to a .vmdk in the data store, doing the same with an iSCSI target on the NAS, etc.

The best combination I have found is backing up to a .vmdk added to the Veeam server from an NFS data store on the NAS.

I have only tested using the VStorage API. The real curiosity I have is that all other things being as equal as I can make them, I always see better backup throughput if I use the VStorage API in Network (NBD) mode as opposed to Virtual Appliance (VA) mode.

For example, I just did a full backup of a 24GB VM. In VA mode, I saw 54 MB/s. I deleted the backup, modifed the job to use NBD mode, and re-ran it and saw 64 MB/s. This has been fairly common in all the tests I have done.

So, since everything I've read here has indicated I should expect better performance using VA mode, I was wondering if there is anything eccentric in my set up that makes the NBD mode faster for me?

I should note that I have done similar tests running 32-bit and 64-bit Veeam, as well as with 2 and 4 vCPU's in the VM, and memory from 2 to 4 GB, and the results are roughly the same each time, always backing up the same 24GB VM.

I am very satisfied with the performance and I'm thrilled with the product, but was just curious as to why I would see better performance in NBD mode vs. VA mode.

Any insights would be greatly appreciated.

Thanks,

Steven

[Gostev]

So, since everything I've read here has indicated I should expect better performance using VA mode, I was wondering if there is anything eccentric in my set up that makes the NBD mode faster for me?

Hi Steven, welcome to our forums. The references about better performance with VA mode compared to Network mode may no longer be valid, as they were all posted before VMware has released the patch fixing an issue with Network mode performance on ESX4 (a few weeks ago). Looking at your results you definitely has this patch applied on your ESX. So please don't pay attention to previous speed comparisons, it is "new era" now. From your results it looks like NBD mode is simply more effective for the given ESX hardware and SAN.

[sbrumlwy]

Thank you Gostev. One more thing I've searched for, but can't seem to find is maybe a "best practice" for configuring the VM for running Veeam? I hope I didn't overlook it.

I am backing up around a dozen VM's, and I run the Veeam SQL database locally with the VM. I currently have it on 64-bit w/ 4 vCPU and 4GB RAM. The VM serves no other purpose than to run Veeam. Overkill?

Thanks,
Steven

[Gostev]

Yes, you could reduce that to 2 vCPU and 2GB of RAM.

[glennsantacruz]

With regard to the question about VM-specific setup to run Veeam B&R inside a VM: we, too, are evaluating the product and have decided to virtualize Veeam. So far, so good -- however, I have noticed (using virtual appliance mode) that the Windows VM ( 2003 R2, 32-bit, 2GB RAM, local SQLExpress, Veeam 4.1.1 ) throws errors in the eventlog when attaching the hot-add scsi disks of the VM being backed up.

For example, we see:

The device 'VMware Virtual disk SCSI Disk Device' (SCSI\Disk&Ven_VMware&Prod_Virtual_disk&Rev_1.0\4&1588251b&0&000400) disappeared from the system without first being prepared for removal.

These errors are consistent ; are we perhaps missing some setup information specific to running Veeam as a virtual appliance? Something along the lines, maybe, of automount settings via diskpart (similar to VCB mode) ?

Anyone else notice this?

[Gostev]

Yes, automount settings need to be disabled via diskpart for Virtual Appliance mode, similar to VCB.

[glennsantacruz]

Thank you for confirming that ; the User guide does mention those settings, but only in context of a VCB deployment ( p.35, Dec.2009 ). I'll adjust our notes to indicate that a VM-based Veeam in appliance mode should also be configured like a VCB.

[glennsantacruz]

Bad news - after disabling automount via diskpart (and subsequently rebooting the VM), I still get the errors. Can evaluation licenses open support cases with Veeam, or should we use these forums?

[Vitaliy S.]

Glenn, yes we do provide technical assistance during your official evalution period.

[Gostev]

Glenn, sorry - I have merely confirmed that you do need to disable automount, but I did not mean to say that this will fix this error. As for the error itself, it will not cause any issues for Veeam Backup server, and can be ignored. This error is important for devices mounted for write access, but with our product, VM disks are mounted as read-only. Read-only storage does not need to be prepared for removal, especially given that Windows itself will not interact with these volumes - only our application accesses these volumes.

[glennsantacruz]

Gostev - thank you for confirming that this error message is not an indication of something wrong, and that we can effectively ignore the error. I do understand that the appliance is attaching the "pre-snapshot" disk via scsi hot-add and reading through the disk contents (this is effectively the same technique we've seen in VMware Data Recovery and esxpress), and this certainly is understandable. It's unfortunate, though, that the error message implies there's some method of possibly preparing the scsi device for removal. Is Veeam unaware of any available Windows API to provide for this "preparation for removal"?

We typically use an enterprise event monitoring system for our servers, and these errors are tripping the event monitor, telling us there's problems with the Veeam server. Although we can flag these to be ignored, I think it's not good practice.

[Gostev]

Glenn, I see you point, but our product operates on higher abstraction level and it does not do actual mount/unmount of these volumes, these operations are handled by VMware vStorage API. So the only way to achieve this is injecting some new code into vStorage API functions... which is obviously not the greatest thing to do...