Comprehensive data protection for all workloads
Post Reply
seonet
Influencer
Posts: 17
Liked: never
Joined: Mar 11, 2014 11:04 pm
Full Name: Pavel
Contact:

Direct SAN - multiple NIC on backup server - traffic problem

Post by seonet »

Hi,

I have problem with backup speed. Problem is that data transfer between proxy and repository is return via network - but both are on same server.

So traffic in backup is as:

30% from DirectSAN -> proxy
30% from proxy-> NIC
30% from NIC-> repostory

Again - both are on same server. So why proxy send this via NIC and dont save backup directly to repository files?

Thank you

Pavel

CASE: #00526465
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Vitaliy S. »

Hi Pavel,

Do you use local disks for the repository or this is an ISCSI initiated datastore that is mounted to the proxy server?

Thanks!
seonet
Influencer
Posts: 17
Liked: never
Joined: Mar 11, 2014 11:04 pm
Full Name: Pavel
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by seonet »

Hi,

I'm using local disks...

Pavel
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Vitaliy S. »

Based on your ticket number, I see that you proxy was not configured to work in SAN mode and that is why it was using network interface to retrieve data from the ESXi host. Could this be the reason why you see data on the network interface of your machine?
seonet
Influencer
Posts: 17
Liked: never
Joined: Mar 11, 2014 11:04 pm
Full Name: Pavel
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by seonet »

Please see last message in ticket. I sent new logs there. Thanks. In short:
Image (task was manually canceled due speed)
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Vitaliy S. » 1 person likes this post

Got it! Let's see what your engineer finds out after reviewing the latest log files package. Your setup is very common and I haven't heard anything about proxy server sending data to the repository via network in other environments, everything should be LAN-free.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Andreas Neufert »

as i know we use internal network at the server because our services proxy/repositor communicating with each other over tcp. So a high network percentage at bottleneck isn't unexpected. Please check if no virus scanner scanning the network (like the yellow ones).
If you feel that the high load at network stack is a problem, work with our support as mentioned before.
There is also a microsoft option that not only first cpu handle network traffic. maybe this can help to reduce bottleneck persentage.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Gostev »

Subscribing to the thread as I am very curios to find out what will this come down to. My guess is some unusual configuration around networking on that proxy/repository server. Pretty sure this will be an easy fix.
seonet
Influencer
Posts: 17
Liked: never
Joined: Mar 11, 2014 11:04 pm
Full Name: Pavel
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by seonet »

Support sent me next steps - set proxy and repo. with 127.0.0.1 or localhost. Tried both, but without effect on speed. Any next idea please?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Vitaliy S. »

Hmm...I would recommend to run a WebEx session with our support engineer to verify the configuration and see how the traffic is routed.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Andreas Neufert »

Virus Scanner installed?
Other Software than Windows, Veeam, SQL and Drivers?
seonet
Influencer
Posts: 17
Liked: never
Joined: Mar 11, 2014 11:04 pm
Full Name: Pavel
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by seonet »

Hi, on server I have not antivirus or undefault FW. There is some managment software for HP etc.

Tomorrow we have next remote session. We redirect traffic trought full dedicated NIC (before this was share hyperv virtual network port - but only for SAN). Speed is still poor - about 30MB/s. Storage is on 10Gbe to switch (store virtual 4630) - from sw. to backup server is there 1Gbe.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Vitaliy S. »

Not sure antivirus can cause this, please keep us updated on your findings.
seonet
Influencer
Posts: 17
Liked: never
Joined: Mar 11, 2014 11:04 pm
Full Name: Pavel
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by seonet »

Ticket is escaleted to senior tech now. We will have new sesion in next days.
seonet
Influencer
Posts: 17
Liked: never
Joined: Mar 11, 2014 11:04 pm
Full Name: Pavel
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by seonet »

Hi,

so solution for this issue is set 8MB block size in veeam.

Do you know anybody why must so big?

We set teaming for two NICs and speed is about 170-200MB/s for multi VMs in one time. For one VM it is about 100MB/s - from HP idea is that this is maximum for one stream - HP think that NIC teaming cant split one SAN stream to two NICs.
Peejay62
Expert
Posts: 235
Liked: 37 times
Joined: Aug 06, 2013 10:40 am
Full Name: Peter Jansen
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Peejay62 »

Hi,

I'm curious, setting the blocksize in Veeam@8MB. Where did you do that? Is that choosing Wan Target fo storage optimization?
I also have some teamed NICs and keep having the feeling that it's underperforming. If i know where to set it I can give it a try.

thanks, Peter
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Vitaliy S. »

I'm a bit confused now...block size absolutely cannot affect backup traffic routing. Can you please elaborate a bit more on what you've found out with our engineer?
seonet
Influencer
Posts: 17
Liked: never
Joined: Mar 11, 2014 11:04 pm
Full Name: Pavel
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by seonet »

Peejay62 wrote:Hi,

I'm curious, setting the blocksize in Veeam@8MB. Where did you do that? Is that choosing Wan Target fo storage optimization?
I also have some teamed NICs and keep having the feeling that it's underperforming. If i know where to set it I can give it a try.

thanks, Peter
Right. Try it!:) Support wrote:
The blocksize is 8MB, 1MB, 512KB, 256KB, starting from Lan Target (16TB+) and down to WAN - you will need to run an active full, before the settings are applied, however.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by foggy »

Vitaliy S. wrote:I'm a bit confused now...block size absolutely cannot affect backup traffic routing. Can you please elaborate a bit more on what you've found out with our engineer?
As far as I can get, the block size change helped to address performance issues (target was the bottleneck), not the traffic routing that was introduced in the original post.
seonet
Influencer
Posts: 17
Liked: never
Joined: Mar 11, 2014 11:04 pm
Full Name: Pavel
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by seonet »

Hello, I have news for this. So, as HP tested SAN Lefthand in details. From tech. atributes - 256kB is max block size. So if I have 8MB BS in veeam, so LH is not affected with this settings. Bottleneck for this is elsewhere. Network? Backup repo? Do you have idea?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Vitaliy S. »

Block size should not affect data routing, however what does bottleneck stats show after you have changed the data block size?
seonet
Influencer
Posts: 17
Liked: never
Joined: Mar 11, 2014 11:04 pm
Full Name: Pavel
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by seonet »

with blocksize 8MB - speed is full (2 NICs teaming) - 200Mb/s - one VM with more VMDKs on full.... after end one VMDK, other increase speed
with blocksize 512kB - speed is about 100MB/s for all VMDK in one time..... after end one VMDKS, speed of others IS NOT increased, bottleneck is 94-24-0-0
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Direct SAN - multiple NIC on backup server - traffic pro

Post by Vitaliy S. »

Hi Pavel,

Thanks for the update, so judging by the numbers, the bottleneck is still source, maybe there is some throttling on storage level, so changing repository should not make a change. On a side note, how does hotadd backup mode perform with your SAN?

Thanks!
Post Reply

Who is online

Users browsing this forum: Google [Bot], Semrush [Bot] and 238 guests