How to diagnose Backup performance

[Deox]

Hi guys,

I was hoping you may be able to help me diagnose a potential bottle neck in our backup throughput.

Currently we are averaging around 8MB/s differential backups using Veeam Backup v3.0. We are looking to migrate the ESX and Veeam storage to a new EVA 4400 SAN however I am keen to get the most out of the current setup prior to moving the backend storage. We have updates the Qlogic HBA drivers to the latest available, and I am also using defrag/zero wipe scripts to optimise any white space that is present in the VMs.

What I would like to know, is what metrics or measurements can I monitor to help identify where the bottleneck may be? You may just say that the disk subsystem needs to be upgraded to improve the performance. However I am keen to know what performance I should expect to be seeing. Can anyone offer real world stats of systems on HP EVA or MSA SANs?

What if anything else can be done to tune throughput?

Good product, I have thoroughly enjoyed working with it so far!

Veeam Server
OS Name: Microsoft(R) Windows(R) Server 2003, Standard Edition
OS Version: 5.2.3790 Service Pack 2 Build 3790
System Model: HP ProLiant BL25p G1
Processor(s): Dual core 1.8Ghz Mhz
Memory: 2GB
HBA: QLogic v9.1.3.18
SAN storage MSA 1500cs 2Gbps 7 x 146GB RAID5 Array

4 x VMware Server Cluster
OS Name: ESX 3.5
System Model: HP ProLiant DL380 G5
Processor(s): 2x Quad Core 2.7Ghz
Memory: 22GB
HBA: QLogic QLA2432
SAN storage MSA 1500cs 2Gbps 6 x 300GB RAID5 Array

Using Optimal compression and VCB SAN backups the following is a typical performance.
30 of 30 VMs processed (4 failed, 0 warnings)
Total size of VMs to backup: 597.68 GB
Processed size: 563.02 GB
Avg. performance rate: 8 MB/s
Start time: 13/04/2009 00:00:14
End time: 13/04/2009 19:00:57

The fastest of the VMs that backed up was the following. This VM includes around 30GB whitespace so I expected it to be a faster backup.
15 of 15 files processed
Total VM size: 40.00 GB
Processed size: 40.00 GB
Avg. performance rate: 56 MB/s
Backup mode: VCB SAN
Start time: 15/04/2009 18:01:40
End time: 15/04/2009 18:13:54

[Gostev]

Usually first thing we recommend to test is VCB file retrieval performance. To do that, you should run VCB job with command line from the Veeam Backup job logs. To do this, create new test VCB job and run the full backup (as a test VM, use VM that you observe slowest speed with), then go to Help | Support information, look for the latest job log, open it and look for "vcbmounter", copy that command line, substitute asterisks with the actual password, change backup path to real folder on your VCB proxy, and run the resulting command line. Note the speed of this operation, basically Veeam Backup cannot work faster than VCB is able to retrieve data from SAN.

If it appears that you are getting great VCB data retrieval performance, then the only thing left to suspect is your target storage performance, or connection speed between Veeam Backup console and your target storage.

As for real-world stats, I am collecting those in the following thread:
Veeam Backup 3.0 : Feedback From The Field

Hope this helps!