How to confirm VEEAM v7 is using DirectSan?

[michel_deschenes]

Hi,

I just installed VEEAM v7.0.3 for the first time on a physical WIndows 2012R2 and connected on my vCenter infrastructure.

I have attached my LUNs as read-only to my server and followed the online guides to setup Direct San backups. I even force the repository to use only Direct San. Though I am unsure if it really is using it. I am getting 40mbps throughput and there is a lot of network activity on my Physical server and my ESX server. There is also very little disk IO on my SAN from the VEEAM server. How would I go to confirm that VEEAM is indeed using Direct San access?

I wondered if the SAN access was simply just slow but testing with IOMeter gives me 9000+ IOPS and 120mbps/worker thread.

VEEAM server: IBM HS21, 8 cores, 32gb ram, 4gbps qlogic hba
SAN : DELL Compellent FC

Thanks for any insight!

[Vitaliy S.]

Hello Michel,

You can confirm that by looking at your job session (pick any VM) and search for [san] tag in the backup job log. Can you please tell me what is your bottleneck stats for the backup job?

Also can you please clarify how you forced your repository to use direct SAN access?

Thank you!

[michel_deschenes]

Hi Vitaliy,

Ok, my backups is indeed using the SAN then, on to trying to figure out why it's slow..

Bottleneck is Source. Which would lead to SAN access being slow but IOMeter gives me an average of 120mbps/worker thread using 64kb random reads. Also, the read IOPS from my Veeam server is really low.

I forced Direct SAN access by going into the backup proxy setting and setting transport mode to "Direct SAN access" and unchecking the "failover to network mode" checkbox.

[Vitaliy S.]

Michel, do you have the MPIO software installed on the backup proxy? Are you running the latest version of the FC driver?

[michel_deschenes]

I have the Windows 2012 MPIO feature installed. I have installed the latest HBA driver from windows update. I could try updating to the latest version.

Thing to note is that my OS and Repository are both running on the SAN also.

[Vitaliy S.]

Try temporarily disabling MPIO, as I've seen a couple of posts on these forums where our clients reported decreased direct SAN performance with MPIO enabled.

As regards having repository on the same SAN, then it is not the best practice to do so, as you definitely don't want to lose production VMs and your backups if your SAN goes down. I would recommend using backup copy jobs to offload your backups to some other storage device, just to be on the safe side.

[michel_deschenes]

I will try disabling MPIO and see how that goes and will update.

I do plan to use a backup copy on LTO4 tapes when I move into production (I'm currently just trying to set it up to replace our existing backup solution).
I also am setting up replication jobs to our second datacenter.

[Vitaliy S.]

In this case your backup strategy looks good to me Please keep us informed on your findings after playing with MPIO parameters.

[michel_deschenes]

Just disabled MPIO from within Windows, no real performance change.

One thing I noticed is that if I run a job with 1 VM in it, I get ~150mbps on my storage. If there are 4 VMs, I get ~350 mbps. It's as if each thread ran by VEEAM was slow even though it says it's bottleneck is source :S
Veeam reported an average of 86mbps when running a job with only 1 VM, 200mbps when running a job with 4 VMs (dropping by about 50 mbps everytime a VM was fully backed up).

1 VM = 5/27/2014 10:55:42 AM :: Load: Source 99% > Proxy 28% > Network 4% > Target 0%
4 VMs = 5/27/2014 11:14:13 AM :: Load: Source 99% > Proxy 55% > Network 6% > Target 0%

[michel_deschenes]

I just got a weird thing now.. I started backing up my file server which weighs about 6TB. My new bottleneck is network? How would that be when it's using Direct SAN Access?

[Vitaliy S.]

Michel, can you please post full bottleneck stats for this very job?

[michel_deschenes]

Took a while to complete.. 10TB had to be backed up at an average of 120mbps

5/28/2014 11:49:47 AM :: Busy: Source 86% > Proxy 54% > Network 34% > Target 0%

Bottleneck changed to Source again during the evening.. Weird that Network was busy though as it did say it was using SAN..

[Vitaliy S.]

Just to make sure we are on the same page, are you referring to Megabits per second (Mbps) or Megabytes per second (MBps) for this job run? If it is megabytes, then you have a pretty decent performance for the full job run.

I just got a weird thing now.. I started backing up my file server which weighs about 6TB. My new bottleneck is network? How would that be when it's using Direct SAN Access?

How did you configure your repository server? Do you have both roles of repository and proxy servers deployed on the same machine?

[michel_deschenes]

Yes, my veeam server is also proxy and repository. I have the ESX LUNs mounted on it (read-only).

[Vitaliy S.]

If you have both proxy and repository server (pointed to the local hard disk) on the same backup server, then data transfer should be LAN-free. And what about the first question?

[veremin]

My new bottleneck is network?

Even when executed on the same machine, VB&R source and target components communicate with each other via TCP. The traffic stays local, though. This communication has been identified as the bottleneck in your case. Thanks.

[michel_deschenes]

Oh sorry I didn't see the first question.

I am talking about MBps, so 200MBps would be 1.6gbps.

My problem is that every single thread are slow. I get a max of 50MBps (400mbps) per thread (which drops to 30MBps (240mbps) when processing a very large VM). Why can I higher speed when running multiple threads but 1 thread is stuck at a max of 400mbps. Storage is barely using and IOMeter gives me much better performance using 100% random read (125MBps (1gbps)) on a single thread.

[michel_deschenes]

Even when executed on the same machine, VB&R source and target components communicate with each other via TCP. The traffic stays local, though. This communication has been identified as the bottleneck in your case. Thanks.

Yeah I figured that out when checking the network usage, it had the VEEAM server as the source and destination, as if it was using the loopback to communicate with the agents. It did change to source as the bottleneck later on.

[Vitaliy S.]

Do you have any virtual proxy server to compare job performance of HotAdd vs. SAN backup modes? BTW, do you read and write data to the same SAN storage at the same time?