Veeam 6.5 - "Direct SAN" mode slow

[maverick964_uk]

We have new physical HP DL380 G7 servers built to perform Direct SAN Veeam backups.
Windows 2008 R2 SP3
Vsphere 5.1

All VMware datastore LUNs are presented to the host via the SAN. (IBM V7000 array and brocade switches)
Veeam 6.5 update3

The physical proxies write to a HP D2D CIFS share via teamed 10Gbit networking.

For some reason performance has dropped considerably and its hard to try and troubleshoot the delays.
After initial installation and testing we were seeing good performance....approx 150+ MB/s.

Now its gone down to about 10-20MB/s....

Veeam reports a mix of bottleneck with Source and Target however its unclear how Veeam works this out.

If someone can help with some troubleshooting that would be great.

[veremin]

Veeam reports a mix of bottleneck with Source and Target however its unclear how Veeam works this out.

Hi, Mike,

I'm wondering what backup mode you're using. Forward incremental or Reversed incremental one?

In case of reversed incremental, the said situation might be explainable - during the initial ("full") run no transformation is involved, and you see pretty decent backup speed. However, once the subsequent runs take place, and full backup starts moving forward, transforming on regular basis, the backup speed drops significantly, as the underlying "dedupe" device might not handle the random I/O well.

As to how bottleneck statistics is calculated, see the sticky FAQ topic.

Thanks.

[foggy]

Also, are you positive Direct SAN transport mode is still used during backup (i.e. no failover to network)?

[maverick964_uk]

For the moment we are doing FULLs as a solid test. We dont do any reverse incrementals, just normal ones.
Yes, transport mode is definately SAN...

[foggy]

Does the primary bottleneck change from run to run or you can distinguish some component as the most common weakest point reported? Could you probably recall some environmental changes performed recently that could affect the backup speed?

[veremin]

What job compression/deduplication settings you're using? Also, have you changed them recently? Thanks.

[maverick964_uk]

we havent changed any settings just added the new physical proxy servers into the environment.
We dont perform any compression or dedup as we leave this to the HP D2D.

I cant understand where the delay is. The SAN/array reports reads the same as Veeam. The proxies are under no load at all.

Its really hard to troubleshoot and try and find where the bottleneck could be.....

[foggy]

we havent changed any settings just added the new physical proxy servers into the environment.

What if you try and remove this proxy from the equation (to check whether it could affect the process somehow)?

[maverick964_uk]

sure the physical proxy will only make the process faster. My initial tests were good so I cant understand where the delays are.
The veeam server is a VM and the storage is the IBM V7000....

are there issues with backing up large LUNs? I'm sure I read there were performance issues with LUNs presented to datastores which were over 1TB....Vmware issue not a Veeam one....

We just dont know what to suggest as the solution goes into testing shortly and we dont want them throwing it out!!

How can we troubleshoot end to end???

[tsightler]

If possible I would suggest carving up a local repository of local disk and testing to that. If performance returns to "normal" then at least you know the problem is with the CIFS portions writing to the D2D, if not, then you know it's almost certainly the source data retrieval.

You say you have DL380 G7 "servers", i.e. plural. Are you designating specific proxies as a gateway for the CIFS traffic or just set to auto. Do you have a lot of tasks running in parallel? It's important to remember that, when writing to a CIFS target, only one server for each job must act as the "repository" for the CIFS share, so with multiple proxies you can end up with a reasonable amount of "cross proxy" traffic which can eat into the available bandwidth (although it sounds like you don't have a bandwidth issue). You might try disabling all but one proxy and running some tests just to get a baseline.

To really help we'll probably need more details like screenshots from a job run or something.

[maverick964_uk]

I did try local disk and this didnt show an improvement. It seems to me that its backend disk reads.
We have checked the V7000 and cant see any issues....could it be delays in vmware?? Knowing which blocks of data to read via Direct SAN mode??

We have 2 x HP D2Ds and 2 x physical proxies. All have 2 x 10Gbit ethernet teamed for 20Gbit.

We can use normal copy and paste of data to the D2D with excellent transfer rates.

Veeam just writes at a snails pace....
however we have seen up to 1-2GB/s rates in veeam during early testing. Cant understand why its all changed to 10-20MB/s....poor!

thanks

[tsightler]

I agree, definitely sounds like source retrieval. Might try something silly like forcing network mode just to change the data path, help to eliminate one more thing. With 10Gb network assuming the hosts also have 10Gb for the management network this should still give speeds measured in the >100MB/s. That would help to eliminate the SAN data path, at least between the proxy and the SAN. It's just a troubleshooting step.

[maverick964_uk]

What I have noticed is that disk management and diskpart take some time to scan the disks. There arent alot of disks either....??

Also, what are best practices for the settings on the proxy server?

ie should MPIO be used? Should we perform automount disable | scrub from diskpart?? Does it matter if LUNs are presented read only or read/write??
I've seen something about disabling disk management.....

please let me know as maybe the proxy delays are due to the slow response from disk management scanning ??

[maverick964_uk]

still doing tests but it all seems faster after doing the following:

disable MPIO
diskpart> automount disable

and a reboot....

[foggy]

While MPIO can affect performance in some situations, disabling automount has nothing to do with it. Actually, it is not even required, as starting from v6.1 Veeam B&R sets SAN policy to Offline instead of disabling automount.

[pkelly_sts]

I once resolved a major performance issue of direct-attach SAN backup on a G7 with Brocade cards/Switches simply by running a driver/firmware update on the FC card. Might be worth a shot if you haven't updated it already.

Paul