NFS datastores, horrible replication performance

[schaffeb]

Hello,

We currently have a vSphere 4.0 cluster which uses NFS datastores exclusively. We have three production datastores and one backup datastore. Each datastore is a dell 1950 server (running ubuntu 9.04 server) connected to a MD1000 disk array(s). We have a private gigabit lan dedicated to these nfs datastores.

Performance in general is not an issue, however when backing up with Veeam 3.1 (32bit installed on a physical server running win2003 r2 32) performance is miserable.

Replication tasks using either VCB (nbdssl) or the veeam agent run at 8MB/s.

Is there a fundamental issue with backing up NFS datastores? Or is there something I'm missing?

edit:

I should add that I can scp a file from a production NAS to a backup NAS at 45MB/s

[Gostev]

Ben, what speed are you getting when downloading a file from NFS storage using datastore browser in VMware Infrastructure Client on Veeam Backup server?

[schaffeb]

It looks like I'm getting roughly 11 MB/S.

Edit:

I am now attempting to scp a file from a VM in our cluster (running off an NFS datastore) to another physical machine on the same gigabit switch. I am getting a transfer rate of 16-17MB/S.

[schaffeb]

I've performed to additional tests:

I've logged in to the service console of one of our hosts and performed two copies.

First copy was from one nfs datastore to another. This resulted in 11 MB/s copies.

Second copy was from one nfs datastore to the local disk on the host. This resulted in 14 MB/s copies.

It looks like my main problem is VMware nfs performance. I believe I should be getting more that 11MB/s when copying between datastores.

[Gostev]

Yes, NFS can be quite slow depending on implementation, but most likely this issue makes more effect in your case.

[schaffeb]

I don't believe I'm experiencing the throttling issue. I've tried VCB and I get the same speeds. We're going to look further into optimizing our NFS servers.

Veeam itself is working great however!

[Gostev]

Since you can be using VCB in NBD mode only, you are probably still subject to the same limitations because all I/O still goes through ESX host (although I have not tested this mode myself, I believe this is the case because you get 45MB/s with SCP copy - looks good for software-based NFS server). Only VCB SAN mode provides direct storage access, but this is impossible to do with NFS storage.

Actually, I think you can quickly test this by running VCB NBD job against a VM located on local ESX storage.