Direct SAN - multiple NIC on backup server - traffic problem

[seonet]

Hi,

I have problem with backup speed. Problem is that data transfer between proxy and repository is return via network - but both are on same server.

So traffic in backup is as:

30% from DirectSAN -> proxy
30% from proxy-> NIC
30% from NIC-> repostory

Again - both are on same server. So why proxy send this via NIC and dont save backup directly to repository files?

Thank you

Pavel

CASE: #00526465

[Vitaliy S.]

Hi Pavel,

Do you use local disks for the repository or this is an ISCSI initiated datastore that is mounted to the proxy server?

Thanks!

[seonet]

Hi,

I'm using local disks...

Pavel

[Vitaliy S.]

Based on your ticket number, I see that you proxy was not configured to work in SAN mode and that is why it was using network interface to retrieve data from the ESXi host. Could this be the reason why you see data on the network interface of your machine?

[seonet]

Please see last message in ticket. I sent new logs there. Thanks. In short:
(task was manually canceled due speed)

[Vitaliy S.]

Got it! Let's see what your engineer finds out after reviewing the latest log files package. Your setup is very common and I haven't heard anything about proxy server sending data to the repository via network in other environments, everything should be LAN-free.

[Andreas Neufert]

as i know we use internal network at the server because our services proxy/repositor communicating with each other over tcp. So a high network percentage at bottleneck isn't unexpected. Please check if no virus scanner scanning the network (like the yellow ones).
If you feel that the high load at network stack is a problem, work with our support as mentioned before.
There is also a microsoft option that not only first cpu handle network traffic. maybe this can help to reduce bottleneck persentage.

[Gostev]

Subscribing to the thread as I am very curios to find out what will this come down to. My guess is some unusual configuration around networking on that proxy/repository server. Pretty sure this will be an easy fix.

[seonet]

Support sent me next steps - set proxy and repo. with 127.0.0.1 or localhost. Tried both, but without effect on speed. Any next idea please?

[Vitaliy S.]

Hmm...I would recommend to run a WebEx session with our support engineer to verify the configuration and see how the traffic is routed.

[Andreas Neufert]

Virus Scanner installed?
Other Software than Windows, Veeam, SQL and Drivers?

[seonet]

Hi, on server I have not antivirus or undefault FW. There is some managment software for HP etc.

Tomorrow we have next remote session. We redirect traffic trought full dedicated NIC (before this was share hyperv virtual network port - but only for SAN). Speed is still poor - about 30MB/s. Storage is on 10Gbe to switch (store virtual 4630) - from sw. to backup server is there 1Gbe.

[Vitaliy S.]

Not sure antivirus can cause this, please keep us updated on your findings.

[seonet]

Ticket is escaleted to senior tech now. We will have new sesion in next days.

[seonet]

Hi,

so solution for this issue is set 8MB block size in veeam.

Do you know anybody why must so big?

We set teaming for two NICs and speed is about 170-200MB/s for multi VMs in one time. For one VM it is about 100MB/s - from HP idea is that this is maximum for one stream - HP think that NIC teaming cant split one SAN stream to two NICs.

[Peejay62]

Hi,

I'm curious, setting the blocksize in Veeam@8MB. Where did you do that? Is that choosing Wan Target fo storage optimization?
I also have some teamed NICs and keep having the feeling that it's underperforming. If i know where to set it I can give it a try.

thanks, Peter

[Vitaliy S.]

I'm a bit confused now...block size absolutely cannot affect backup traffic routing. Can you please elaborate a bit more on what you've found out with our engineer?

[seonet]

Hi,

I'm curious, setting the blocksize in Veeam@8MB. Where did you do that? Is that choosing Wan Target fo storage optimization?
I also have some teamed NICs and keep having the feeling that it's underperforming. If i know where to set it I can give it a try.

thanks, Peter

Right. Try it!:) Support wrote:

The blocksize is 8MB, 1MB, 512KB, 256KB, starting from Lan Target (16TB+) and down to WAN - you will need to run an active full, before the settings are applied, however.

[foggy]

I'm a bit confused now...block size absolutely cannot affect backup traffic routing. Can you please elaborate a bit more on what you've found out with our engineer?

As far as I can get, the block size change helped to address performance issues (target was the bottleneck), not the traffic routing that was introduced in the original post.

[seonet]

Hello, I have news for this. So, as HP tested SAN Lefthand in details. From tech. atributes - 256kB is max block size. So if I have 8MB BS in veeam, so LH is not affected with this settings. Bottleneck for this is elsewhere. Network? Backup repo? Do you have idea?

[Vitaliy S.]

Block size should not affect data routing, however what does bottleneck stats show after you have changed the data block size?

[seonet]

with blocksize 8MB - speed is full (2 NICs teaming) - 200Mb/s - one VM with more VMDKs on full.... after end one VMDK, other increase speed
with blocksize 512kB - speed is about 100MB/s for all VMDK in one time..... after end one VMDKS, speed of others IS NOT increased, bottleneck is 94-24-0-0

[Vitaliy S.]

Hi Pavel,

Thanks for the update, so judging by the numbers, the bottleneck is still source, maybe there is some throttling on storage level, so changing repository should not make a change. On a side note, how does hotadd backup mode perform with your SAN?

Thanks!