for a test- and development environment I have an extra ESX-host running with vSphere 4.1i. The free (unlicened) version.
Because of an agreement between VMware and Veeam it is not possible to use Veeam B&R to make backup's or copies from such host. Or any other vendor's product for making backups. Ok, understandable...
Because I do want to make a backup copy of the VM's now and then I make use of ghettoVCB. On the ESX host I have a NFS datastore mounted that is provided by the Veeam-server. It is part of the "vPower Instant Restore feature". Very nice; thank you Veeam
This went very well and quite speedy when I was using Veeam V5, but since I upgraded to V6 the NFS performance is terribly slow.
Did anybody notice any difference for NFS between the V5 and V6 version. And maybe any suggestions how to improve the lack of performance?
Ed, actually, the whole vPower functionality has not been changed since v5. Most likely this is a hardware issue. Do you have the vPower NFS server on the same server it used to be with v5? Anyway, you can specify another NFS server in the backup repository properties and check whether it acts faster than the current one. Thanks.
It cannot be a hardware issue because nothing has changed. The server I use is a dedicated HP Proliant DL360 G6 with dual quad-cores and 24 GB memory. It has local storage for Veeam backups. The servers sole purpose is running Veeam. Did an in-place upgrade from V5 to V6 and am quite happy with the improved performance of the (4) replication jobs, but not so happy with NFS at the moment.
Nothing changed on the ESX-host either.
Our environment is not that large that I can just choose another NFS-server
Maybe I can try to install MS services for NFS on another machine...
That is btw how I discovered originally that there is a NFS-service in Veeam: the port was already in use
Ed, could you please provide some details on how do you use Veeam NFS datastore? We need to know on what operations you see NFS server performance decrease to be able to make some performance tests to investigate. Thanks.
The Veeam-server's D:-drive has a folder named NFS.
During the install of Veeam there is a question about the path to use for "vPower Instant Restore feature". I'm not sure about the correct text for this; then I shoud have to reinstall to see this. Anyway, pointed this to D:\NFS.
Then with vCenter select an ESX-host:
- Add Storage
- Network File System
- Server = VeeamServer
- Folder = /VeeamBackup_VeeamServer
- DataStore Name = NFS
And then the host sees the new DataStore. This was done still running V5 and worked without any problems. The ghettoVCB script copied three VM's with a total of 200GB in about 4 hours. Once a week. Great, never one hickup.
About a week ago I upgraded to V6 by running "Veeam_Backup_Setup_x64.exe" directly on the VeeamServer. Answered yes when asked to upgrade.
From that moment on the NFS DataStore became slow. I have a test VM of 8 GB which takes 2.5 hours to complete now.
To make the story complete; in V5 the path originally pointed to a folder under the C:-drive instead of D:
When I found out about Veeam's integrated NFS service and wanted to make better use of that I changed the path in the registry and rebooted. As stated above this was no problem in V5; just worked great.
Please ask for any detail you need; happy to oblige.
To troubleshoot myself I'm thinking about:
1. built another NFS-server
Just for elimination purposes, would not be a great solution.
2. remove Veeam completely and reinstall V5; test NFS; upgrade to V6. Or, completely start over with V6.
Not to keen on doing that because it takes a lot of time and we do have several replication-jobs running to a remote site. And these jobs gave quite some trouble in the 'conversion' from V5 to V6. But thats another story and water under the bridge. Also, can I export my current job definition?
Thank you both for taking this serious.
Just for testing and satisfying my own curiosity I quickly installed W2K8r2-standard on a already retired server. A HP Proliant with 2 Xeons and 7 GB Ram. Local storage; raid-V UW-SCSI. All about 6 or 7 years old and nothing fancy.
Installed Services for NFS and mounted the new share on the same ESX host and ran the same ghettoVCB job. The result was 45 minutes for the 8 GB VM compared to 2.5 hours earlier on the Veeam server.
All servers (ESX, Veeam and test) are directly connected to the same switch, 1 GB. No fancy stuff either and in the same subnet/vlan.
So to me it is clear the bottleneck is the Veeam server because nothing changed exept for the V6 upgrade.
If needed I can supply the ghettoVCB logfiles of both jobs.
I agree with Ed. Although I don't have any historical performance numbers to back it up when comparing the v5 to v6 upgrade, v6 NFS does "feel" slower. I have also tested vPower NFS (v6) against a different server running W2K8r2 and services for NFS. Using one of our ESXi hosts to copy a iso file from the vPower NFS takes twice as long as copying the same iso file from the Windows NFS share. I would love to see the vPower NFS performanced tune. I think it would drastically improve our Surebackup and Instant Recovery jobs, and really elevate vPower NFS as a feature.
tsightler wrote:Are you guys running the vPower NFS services on the same server that is your repository?
Hi,
I'm not sure what you mean by that but my aswer is yes because I only have one server for Veeam. So all the Veeam services are running on the same physical server. And please don't tell me I have to split up services over several machines because I work for a small company and then Veeam will be no longer suitable for our needs. And I realy like(d?) Veeam.
Having vPower NFS server and backup repository on the same server is good for performance, Tom was wondering if you have them on separate servers by any chance (which would then explain lower performance).
Of course, you do not need to split anything and go for distributed architecture if the single server with all components on it works fine for your small deployment.
Right, I wasn't suggesting that they need to be split, just asking if they happened to be split. I worked with one user that was experiencing lower than expected vPower NFS performance because he installed the vPower NFS services on the proxy, not the repository thus adding a second hop and additional latency into the mix. If it's all on one host then this is obviously not the issue.
Ok, that I can understand.
My physical, dedicated server is pure for Veeam. Just did an upgrade from V5 to V6. This gave me two differences:
1. Much better performance for the repliation jobs
2. Bad NFS performance.
Is there a way to save the job-configuration so I can completely reinstall the server and import the jobs after that? That way it will be a fresh install of V6 instead of an upgrade from V5. I feel I have to try something to get things up-and-running again. Or do I just wait for suggestions from the guys at Veeam?
Importing backup/replication jobs is only possible when you migrate/re-use Veeam SQL configuration database, please search these forums for a proper procedure if needed.
Just a little update, so far our QC is unable to see any differences in vPower NFS performance between v5 and v6 in their testing. One thing they asked to verify: in the beginning of VeeamNFS log around where the service starts, there should be log entry reading something like "starting service in synchronous mode". Verify that it says synchronous, and not asynchronous mode.
Also, it would really speed up the process significantly if you post your support case ID for this issue, so that devs could lookup the logs directly.
One of the labs did show worse performance. But v6 in another lab is a few times faster with the exact same test. We are looking for the differences...
I just reconfigured NFS-settings via Backup Infrastucture-->Backup Repositories-->Default Backup Repository-->Properties.
Just acknowledged all settings and it reinstalled NFS from msi. Then tested NFS performance again, but slow as #&*#
If I backup the SQL-database with "Microsoft SQL Server Management Studio Express" and then completely reinstall my server from the ground up, bare metal, and install V6, then restore the database, will I then also have my jobs back and will replication start without seeding?
I really have to get this fixed because I have been unable to backup several servers for weeks now.
E.Kok wrote:If I backup the SQL-database with "Microsoft SQL Server Management Studio Express" and then completely reinstall my server from the ground up, bare metal, and install V6, then restore the database, will I then also have my jobs back and will replication start without seeding?
Yes.
E.Kok wrote:I really have to get this fixed because I have been unable to backup several servers for weeks now.
Not sure I understand how the NFS performance affects your ability to backup?
Gostev wrote:Not sure I understand how the NFS performance affects your ability to backup?
That I explained in the opening of this topic; I use a script (ghettoVCB) on the ESX-host to backup VM's to a NFS-datastore. The NFS-datastore is provided by Veeam and is now unusable slow.
To prove (to myself) that the script and the ESX-host perform as usual I installed MS NFS under Windows server 2008 on a different server and that runs fine. So my conclusion is that something changed on the Veeam-server during the upgrade from V5 to V6 because that is the only thing that changed.
That I also explained in the opening of this topic; there seems to be an agreement between VMware and third-party software suppliers (like Veeam) to not allow making backups of 'free' ESX hosts.
When I run a Veeam backup-job against that host the job-log shows:
09-Jan-12 8:06:16 AM :: Error: Your Veeam Backup license does not allow processing of free ESXi.
Because this host is only for testing/development-purposes it is not part of the VMware-cluster and does not have a license installed.
Previously, with Veeam V5, I was able to backup the VM's on this host just fine to the NFS-datastore provided by the Veeam-server. Since the upgrade to V6 it is VERY sluggish.
This morning I installed the 6.0.0.158 update but that didn't help either.
Last week I got a call from Veeam-support. We talked and they had a look at my Veeam-server. Then they concluded they could not support third-party scripts like ghettoVCB. Ok, I can understand that. But it was just a simple way to prove that the NFS-service was very slow.
The call ended with that and I started using a MS NFS-server to make my backups and was somewhat disappointed.
But then, last friday, I received an email from Veeam-support with a 'test'-version of the NFS-service. I tried that and it is great! The performance is like it was with V5 and I'm very happy with that. Support promissed me this fix will be in future releases.
So I'm happily running V6 now (although with some minor other things but that is for another topic).
