Comprehensive data protection for all workloads
Post Reply
schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

NFS datastores, horrible replication performance

Post by schaffeb » Jun 12, 2009 5:47 pm

Hello,

We currently have a vSphere 4.0 cluster which uses NFS datastores exclusively. We have three production datastores and one backup datastore. Each datastore is a dell 1950 server (running ubuntu 9.04 server) connected to a MD1000 disk array(s). We have a private gigabit lan dedicated to these nfs datastores.

Performance in general is not an issue, however when backing up with Veeam 3.1 (32bit installed on a physical server running win2003 r2 32) performance is miserable.

Replication tasks using either VCB (nbdssl) or the veeam agent run at 8MB/s.

Is there a fundamental issue with backing up NFS datastores? Or is there something I'm missing?

edit:

I should add that I can scp a file from a production NAS to a backup NAS at 45MB/s

Gostev
SVP, Product Management
Posts: 25285
Liked: 3756 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: NFS datastores, horrible replication performance

Post by Gostev » Jun 12, 2009 6:53 pm

Ben, what speed are you getting when downloading a file from NFS storage using datastore browser in VMware Infrastructure Client on Veeam Backup server?

schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

Re: NFS datastores, horrible replication performance

Post by schaffeb » Jun 12, 2009 7:15 pm

It looks like I'm getting roughly 11 MB/S.

Edit:

I am now attempting to scp a file from a VM in our cluster (running off an NFS datastore) to another physical machine on the same gigabit switch. I am getting a transfer rate of 16-17MB/S.

schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

Re: NFS datastores, horrible replication performance

Post by schaffeb » Jun 12, 2009 8:07 pm

I've performed to additional tests:

I've logged in to the service console of one of our hosts and performed two copies.

First copy was from one nfs datastore to another. This resulted in 11 MB/s copies.

Second copy was from one nfs datastore to the local disk on the host. This resulted in 14 MB/s copies.

It looks like my main problem is VMware nfs performance. I believe I should be getting more that 11MB/s when copying between datastores.

Gostev
SVP, Product Management
Posts: 25285
Liked: 3756 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: NFS datastores, horrible replication performance

Post by Gostev » Jun 15, 2009 10:01 am

Yes, NFS can be quite slow depending on implementation, but most likely this issue makes more effect in your case.

schaffeb
Enthusiast
Posts: 50
Liked: never
Joined: Apr 14, 2009 10:44 pm
Full Name: Ben S
Contact:

Re: NFS datastores, horrible replication performance

Post by schaffeb » Jun 15, 2009 3:28 pm

I don't believe I'm experiencing the throttling issue. I've tried VCB and I get the same speeds. We're going to look further into optimizing our NFS servers.

Veeam itself is working great however!

Gostev
SVP, Product Management
Posts: 25285
Liked: 3756 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: NFS datastores, horrible replication performance

Post by Gostev » Jun 15, 2009 3:46 pm

Since you can be using VCB in NBD mode only, you are probably still subject to the same limitations because all I/O still goes through ESX host (although I have not tested this mode myself, I believe this is the case because you get 45MB/s with SCP copy - looks good for software-based NFS server). Only VCB SAN mode provides direct storage access, but this is impossible to do with NFS storage.

Actually, I think you can quickly test this by running VCB NBD job against a VM located on local ESX storage.

Post Reply

Who is online

Users browsing this forum: edirschedl, Google [Bot] and 61 guests