Slow Network Speed, Fast Locally

Everything about backing up to tape

Slow Network Speed, Fast Locally

Veeam Logoby rreed » Thu Dec 31, 2015 7:56 pm

Afternoon all. We're running B&R v8 w/ all latest patches, etc. Remote server but in the same data center, (4) 1Gbps LACP, Dell R430, Dell TL2100 (LTO6) direct connected SAS, Win2k12R2. If we run a backup job pointing to files locally on the server itself it will run at full speed (actually reported processing rate 120MB-189MB out of LTO's 160MB). Super, great. Looks like our speed is there, it can certainly do it. Restores will even run 100-120MB. Backup a VM or files, makes no difference. Runs great.

If we point it to files on a proxy (VM, Dell EQL, 10Gbps SAN fabric), it'll run about 2-3MB. Point it to a file share, same thing. Dismal unusable speed. Copy those backup files locally to the tape server, and we're back to full speed. This creates additional time/work to leap-frog our backup files from repositories to the tape server, then backup to tape at speed, administrative overhead, oh and we don't have enough room to use the tape server as a repository anyway. I pitched a proposal for adding a big DAS array to it but that got laughed off the table so we're back to needing to backup to tape over the local network. 2-3MB/s is unusable, especially for an LTO-6.

I've applied the registry hack for disabling the hidden encryption over the network, no change. Robocopy files to the server runs at full 1Gbps for single data stream, 2-3+Gbps running multiple streams. Backup to the tape server's HDD as a repository (setup as proxy so straight to local E: rather than shared) runs at 145-175MB (1.1-1.4Gbps per job, interestingly enough). Nice and fast so we know our network speed is there as well; we know we can get data from our VMware stack to the tape server itself. It's getting data from VMware straight to tape that is suffering.

Any ideas please?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Slow Network Speed, Fast Locally

Veeam Logoby Dima P. » Sun Jan 03, 2016 9:33 pm

Hi,
If we point it to files on a proxy

Was that a file to tape job? How many files were selected as a source? Thanks
Dima P.
Veeam Software
 
Posts: 6263
Liked: 442 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Slow Network Speed, Fast Locally

Veeam Logoby skrause » Mon Jan 04, 2016 2:22 pm

I know this won't resolve the root cause of your issue but this might be a decent workaround: can you have your tape server connect directly to the EquaLogic SAN and use it as a repository that way? (Obviously you would still need to move files between the VMFS volume and the Windows LUN on your tape server).

I have found that anytime the volume that the tape library is pulling data from is not "local" storage on the tape server, performance is poor (though I get better performance than you see, about 20-25MB/s versus the 130 MB/s I average locally on LTO-5).
Steve Krause
Veeam Certified Architect
skrause
Expert
 
Posts: 296
Liked: 45 times
Joined: Mon Dec 08, 2014 2:58 pm
Full Name: Steve Krause

Re: Slow Network Speed, Fast Locally

Veeam Logoby rreed » Mon Jan 04, 2016 8:16 pm

Hi Dima. It doesn't seem to matter if we use a file to tape or backup copy/to tape across the network, it runs unusable slow either way. Doesn't seem to matter if it's a single File backup or multiple files, but I will need to test and confirm that and other details this week. All the same regarding backing up files or VM; even if we do a backup copy job to tape, if the data comes across the network to tape it's slow.

Skrause, that's the same thing we're finding. If data is not local to the tape server, it's unusable. Single digit MB/s. Only pulling from the server's local HDD will it run at speed. I've seen others report the same issue but aside from the hidden encryption registry hack (done, no change) I haven't found any resolution on this forum or anywhere else on the internets. Since we know we can get data to the tape server at speed, we know the network speed is there. We know we can get data from the tape server to the tape at speed, we know the tape speed is there. It's going directly across network to tape that isn't. I'll be opening a support case this week.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Slow Network Speed, Fast Locally

Veeam Logoby skrause » Tue Jan 05, 2016 1:31 pm

When I was looking at things, I kind of went on the assumption that the issue is caused by the LTO drivers and how they handle traffic that comes in over network versus the storage bus. If it does end up being something with Veeam that can be resolved it would make some of my job a bit easier (though I already copy everything I send to tape to the repository connected to the tape server for on disk retention so it would not change much.)
Steve Krause
Veeam Certified Architect
skrause
Expert
 
Posts: 296
Liked: 45 times
Joined: Mon Dec 08, 2014 2:58 pm
Full Name: Steve Krause

Re: Slow Network Speed, Fast Locally

Veeam Logoby Dima P. » Tue Jan 05, 2016 4:57 pm

I was under impression that current file to tape performance over network suffers from slow indexing when you have a lot of files as a source. Might be reasonable to test performance for one single file but large size to recognize the average throughput.

Once you got the case ID, kindly, post it to this thread. Thanks.
Dima P.
Veeam Software
 
Posts: 6263
Liked: 442 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Slow Network Speed, Fast Locally

Veeam Logoby rreed » Tue Jan 05, 2016 5:01 pm

I've never done more than 4-5 files at a time for File to Tape. Is that considered a lot? And it's just backing up .vbk files if that matter. At that, same performance if I attempt to setup a Backup Copy to tape, if that also matters. I'll report back once I have a firm grip on exact circumstances and a case number.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Slow Network Speed, Fast Locally

Veeam Logoby skrause » Tue Jan 05, 2016 5:04 pm

I've had backup to tape jobs where it was only processing one 1.4TB .VBK (not processing incrementals, just using latest chain) that took 34 hours when it was pulling it over the network but takes about 4 hours when pulling it from a backup copy living on storage attached to the tape server.
Steve Krause
Veeam Certified Architect
skrause
Expert
 
Posts: 296
Liked: 45 times
Joined: Mon Dec 08, 2014 2:58 pm
Full Name: Steve Krause

Re: Slow Network Speed, Fast Locally

Veeam Logoby Dima P. » Tue Jan 05, 2016 5:07 pm

Thanks. What was written as bottleneck? Any chance you have a global throttling rules, that might affect the bandwidth consumption, enabled in VBR?
Dima P.
Veeam Software
 
Posts: 6263
Liked: 442 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Slow Network Speed, Fast Locally

Veeam Logoby rreed » Tue Jan 05, 2016 5:14 pm

No throttling rules for us, our infrastructure is all 10Gb SAN/core, (4)x1Gb TOR to the tape server. All w/in the same data center/rack. When we write to tape from the local tape server, bottleneck is target which is what we would expect over 6Gb SAS. Once this particular tape job is complete I'll run the same job over the network and report back to see what the bottleneck is.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Slow Network Speed, Fast Locally

Veeam Logoby Dima P. » Tue Jan 05, 2016 5:21 pm

Ok, I am out of ideas now. Kindly, open a case and let support team check your logs. Update this thread with the case ID, so we could check how it goes. Thanks!
Dima P.
Veeam Software
 
Posts: 6263
Liked: 442 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Slow Network Speed, Fast Locally

Veeam Logoby rreed » Fri Mar 25, 2016 8:20 pm

Thought I had updated this but apparently not. :) Ticket #01715660, been running your TapeLibTest.exe quite a bit, so far only confirming the same behavior as Veeam; network to local HDD fast (200+ MB/s), local HDD to tape fast (180 MB/s), network to tape slow (~101 MB/s). Our network source is a CIFS share.

Since my last post here we've upgraded our tape server's network connection from (4) x 1Gbps LAG to a single 10Gbps connection. It's helped some but we're still not seeing LTO-6 speeds across a now full 10Gbps infrastructure. Around 101 MB/s average. The only things we've been able to try w/ our tape drive is changing block sizes, but not enough to make a real discernible difference I'm afraid. Registry hack for netuse but it made little, if any difference.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Slow Network Speed, Fast Locally

Veeam Logoby rreed » Fri Mar 25, 2016 9:30 pm

Target is always bottleneck BTW, which it of course should be, but not at ~101 MB/s I wouldn't think when it can do 160-180 MB/s.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Slow Network Speed, Fast Locally

Veeam Logoby ccatlett1984 » Wed Apr 06, 2016 5:01 pm

ccatlett1984
Enthusiast
 
Posts: 83
Liked: 9 times
Joined: Thu Oct 31, 2013 5:11 pm
Full Name: Chris Catlett

Re: Slow Network Speed, Fast Locally

Veeam Logoby rreed » Wed Apr 06, 2016 5:51 pm

Tried the same in v8 and v9 (our Veeam engineer added it our v9 tape server). No discernible difference overall, or at least I don't really know for sure, since he added a slew of registry hacks. It certainly didn't help us in v8. Also, some of our tape jobs increased nicely since the hacks, some are about the same or at best only modestly better.

Close enough, we're kind of done messing w/ it, it is what it is, and we'll just be on w/ our lives. They only run once a month so if it takes a couple weeks to get everything archived off to tape, that's fine. We have warranty on the drive to hopefully help alleviate all the shoe shining it will be doing. Oh well. Close enough. :wink:
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Next

Return to Tape



Who is online

Users browsing this forum: No registered users and 6 guests