V6 Direct SAN mode backup speed (source at 99%)

[asloan]

Hey guys:

So I just installed V6, which I like a lot so far. One of my requirements for my backup infrastructure is to use the Direct SAN method to backup our VM's. My environment consists of a physical machine that has Veeam installed and has local storage which is acting as the onsite backup storage. This server is directly connected into the SAN network with four gig nics on different subnets and is setup for MPIO. The storage, a Dell MD3220i, is setup the same with different subnets on each iscsi port.

As a test to see what kind of performance I would get from Direct SAN I created a test backup job with one VM inside and started the job. Before I realized that I had to set the backup method parameter on the backup proxy, it was backing up using NBD. I noticed that the backup was saturating the network connection with a sustained utilization.

I then set the proxy to exclusively use the Direct SAN method with no failover and saw that the network utilization was very sporadic. Comparing the NBD method over the Direct SAN method was that NBD took about 5 minutes to backup the VM that I was using but Direct SAN method was at 8 minutes and at only 2% complete.

Is there something else that needs to be set for the network utilization to be more utilized than 5% every 2 minutes?

Thanks

[Gostev]

Usually, the first thing you need to do in quest for better direct SAN access performance is uninstall MPIO software. Also, be sure to read the sticky FAQ for additional information on iSCSI performance tweaking. Thanks.

[Pauli]

hi asloan,

i had slower backup speed vith V6 than V5
Direct SAN method (manually select datastores)

[asloan]

@Gostev- I did the suggestions from the link that you posted. I want to say that MPIO is configured. I was having trouble with seeing all of the SAN disks at first by just configuring the iSCSI initiator itself. I installed the host drivers on the Dell Resource CD but I can't remember if the MPIO drivers were installed with that. I'll have to retry the backups and see if the backups go faster.

@Pauli- I did a new install of V6 because of some SKU mismatch when ordering. I did have to manually select the datastores for the proxy to see the datastroes under Direct SAN mode.

Thanks Guys.

[asloan]

So I am doing a backup using Direct SAN mode but I am still getting the sporadic utilization of the NIC's used on the SAN network. Also, when I uninstall the MPIO driver all of a sudden in disk management within Server Manager shows a lot of redundant drives.

Thanks

[Gostev]

@asloan, sounds good. Based on years of feedback, I don't believe you will be able to achieve good results with MPIO present. Also, please be sure to always include your bottleneck analysis numbers, as they make it easy to track the "weak" link in your backup infrastructure.

@Pauli, please do not hi jack Alexander's topic. If you'd like, feel free to create your own topic, and please be sure to include the bottleneck analysis numbers for your v6 jobs (from the real-time statistics window). You are, by far, the only user reporting REDUCTION of direct SAN access performance (all other feedback I've got to date reports performance increase - sometimes huge - which is actually expected since there was work specifically done in that direction, and results were confirmed with multiple beta customers). So, there was some other environment changes that is affecting v6 performance - seeing bottleneck numbers would definitely help.

[asloan]

All of the backup jobs that I did had the source as the the bottleneck at 99% and then the proxy at 77%. All others (network and target) were either 5% or 1%. What is being measured when the job is saying that the source is the bottleneck?

Thanks

[Gostev]

This is covered in great details in the FAQ topic.

[jmcmurry]

Usually, the first thing you need to do in quest for better direct SAN access performance is uninstall MPIO software. .

I have a question on this point. This isn't mentioned in the User Guide for 5.x or 6.0, nor anywhere in the sticky FAQ for Direct SAN Access. Is this relevant only for iSCSI or is disabling MPIO for FC devices recommended as well?

I'm not currently seeing any significant performance issues with several dozen FC LUNs using Windows Server 2008 R2 and MPIO using the Round Robin With Subset policy. Is this recommendation a universal best practice or simply the first thing to try when your performance is unacceptably poor?

Thanks!

[Gostev]

@James, it is definitely universal best practice for both FC and iSCSI SAN. I've first learnt about it years ago (back at VCB times). By this time, I lost count of how many customers reported this helped them.

@People, I have deleted a couple of posts unrelated to OP's issue - as per my earlier request to Pauli, please do not hijack this topic, as it is talking about very specific issue with the speed of data retrieval from SAN. Do feel free to create your own topics, and remember to include bottleneck numbers. Thank you.

[tsightler]

Back in the VCB days VMware had a chart that showed specifically what multipath solutions were supported with VCB (it was a short list!). As of VDDK 1.2.1 and later VMware states that you should check with your multipath vendor regarding their support of multipath when us(good luck with that). The support statement in the release notes specifically says:

VMware Consolidated Backup (VCB) has a writeup (knowledge base article) showing the support matrix for storage multipathing. VMware does not provide a similar support matrix for VDDK. Customers should seek this information from their backup software vendors.

Honestly, with VDDK this problem seems far less pronounced. I've personally had great success with some multipathing solutions (specifically Equallogic) with VDDK. That being said, there have also been many cases where clients have reported improved performance after removing MPIO. Because of this fact, and the VMware support statement, it is "best practice" to NOT use multipath unless you can find a specific support statement from your vendor the they support VDDK with their multipath implementation.