Network backup performance issue after upgrade to ESX4

[SystemFuchs]

Hi there,

since my upgrade to vSphere I'm fighting with same issue (DAS->NAS Backup)
With ESX 3.5 and Veeam Backup & Replication i had speeds about 60-70MB to my nas.

Since my upgrade to vSphere 4 I have 20MB maximum speed. Seems to be some kind
of incompatibility somewhere. My local storage adapter is a 3ware 9690 SAS/SATA Controller.

Regards

Stephan

[Gostev]

Stephan, no - actually in your specific case this is a known issue with ESX4, see 2nd post in this topic for link to resolution.

[SystemFuchs]

Hi Gostev,

I've installed the patch already last weekend, but this didn't help any way

Code: Select all

[root@esx01 ~]# esxupdate query
----Bulletin ID---- -----Installed----- ----------------Summary----------------- 
ESX400-Update01a    2010-04-24T15:32:49 VMware ESX 4.0 Complete Update 1         
ESX400-200912401-BG 2010-04-24T15:40:27 Updates vmx, vmkernel etc                
ESX400-200912402-SG 2010-04-24T15:40:27 Updates OpenSSL                          
ESX400-200912403-SG 2010-04-24T15:40:28 Updates NSS and NSPR                     
ESX400-200912404-SG 2010-04-24T15:40:28 Updates DHCP                             
ESX400-201002401-BG 2010-04-24T15:43:54 Updates vmkernel64,vmx,hostd etc         
ESX400-201002402-BG 2010-04-24T15:43:54 Updates initscripts                      
ESX400-201002404-SG 2010-04-24T15:43:54 Updates glib2                            
ESX400-201002406-SG 2010-04-24T15:43:54 Updates newt                             
ESX400-201002407-SG 2010-04-24T15:43:54 Updates nfs-utils                        
ESX400-201002408-BG 2010-04-24T15:43:54 Updates Enic driver                      
ESX400-201003401-BG 2010-04-24T15:46:50 Updates vmkernel64,vmx,hostd and scripts 
ESX400-201003402-BG 2010-04-24T15:46:50 Updates hpsa driver                      
ESX400-201003403-SG 2010-04-24T15:46:50 Updates acpid                            
ESX400-201003404-BG 2010-04-24T15:46:50 Updates lpfc820 driver                   
ESX400-201003405-SG 2010-04-24T15:46:50 Updates samba-client and samba-common    
ESX400-201003406-BG 2010-04-24T15:46:50 Updates megaraid-sas driver              
ESX400-201003407-BG 2010-04-24T15:46:50 Updates ehci-ehci-hcd driver

[SystemFuchs]

Hi there,

question for me at this point is where I can get support?
Is it a Veeam issue or do I need to request support from VMware?

Regards

Stephan

[Gostev]

Please check if you have service console settings provided in the corresponding ESX settings in Veeam Backup Servers tree. After that, try to copy a vmdk of shutdown VM from that host to Veeam Backup server (using copy/paste in Veeam Backup tree), and see what the transfer speed is. This should help to understand where exactly the issue is.

[SystemFuchs]

Hi,

when I copy a vmdk to my local backup machine the performance isn't better.
Perhaps it's important that Veeam is running of a virtual machine residing on my
ESX host?!

PS: Only the full backup takes such a long time, when ckecking changed block tracking data everything is fast.
But this does not solve the problem with my full backup

Regards

Stephan

[Gostev]

when I copy a vmdk to my local backup machine the performance isn't better.

If you get low performance with local copy, I would say there is some problem with that ESX patch resolving network backup performance... It helps everyone else but in your case it did not resolve the issue, which is unexpected. I recommend that you open a case with VMware on this and explain that performance dropped after ESX 3.5 upgrade but did not improve as expected after applying the said patch.

Also if you are running Veeam in virtual machine, you should try giving it 2-4 vCPU, and also try to leverage the vstorage API Virtual Appliance mode, which is generally faster than Network option.

Also, since full backup is performed only once with Veeam, you can as well leave everything as is if incremental backup speed is fine for you.

[SystemFuchs]

Hi,

I've investigated a little bit further and found the following KB Article 1012159 at VMware site.

When I/O operations are done between storage that is managed by the VMkernel (such as a VMFS or NFS datastore) and storage that is managed by the service console (such as an EXT3, CIFS, or NFS mount point), a high performance data mover is not utilized and performance is degraded.

To resolve this issue, try these available options:

- Utilize a data protection model that does not interact with the service console.
- If you are backing up to a network share, do not mount the network share in the service console, but configure it as a datastore in ESX, then use vcbMounter to export to that NFS share. This share appears inside the service console under /vmfs/volumes, just like any other datastore.
- If you export to a local staging directory, such as /tmp inside the service console, consider using a VMFS datastore as a staging directory.
- If you are using VCB on the Proxy, VMware recommends to use SAN or hot-add mode instead of nbd for data transport.

Seems that there is no resolution for the problem at this time. Think I've to live with the situation as it is in the moment.

[Gostev]

Yes, this is original KB article about this problem VMware published very long ago. The ESX patch mentioned above fixes this issue. I have a blog post where I follow this issue for almost a year now (I was the first to discover and confirm the issue), here is it: http://www.vnotion.com/?p=38

[SystemFuchs]

When you look at the publishing date it was updated in march.
So it seems that although there is a patch available the problem still exists.

Problem is that I've only vSphere Essential and I have to pay for a support incident.

[tsightler]

We also didn't see much improvement from the patch in our environment. We're lucky to get 15-20MB/sec when using NBD mode even with the patch, while we get much faster performance (40-70MB/sec) with SAN mode. I'd strongly suggest trying virtual appliance mode if possible.