Slow Network Speed, Fast Locally

[rreed]

Afternoon all. We're running B&R v8 w/ all latest patches, etc. Remote server but in the same data center, (4) 1Gbps LACP, Dell R430, Dell TL2100 (LTO6) direct connected SAS, Win2k12R2. If we run a backup job pointing to files locally on the server itself it will run at full speed (actually reported processing rate 120MB-189MB out of LTO's 160MB). Super, great. Looks like our speed is there, it can certainly do it. Restores will even run 100-120MB. Backup a VM or files, makes no difference. Runs great.

If we point it to files on a proxy (VM, Dell EQL, 10Gbps SAN fabric), it'll run about 2-3MB. Point it to a file share, same thing. Dismal unusable speed. Copy those backup files locally to the tape server, and we're back to full speed. This creates additional time/work to leap-frog our backup files from repositories to the tape server, then backup to tape at speed, administrative overhead, oh and we don't have enough room to use the tape server as a repository anyway. I pitched a proposal for adding a big DAS array to it but that got laughed off the table so we're back to needing to backup to tape over the local network. 2-3MB/s is unusable, especially for an LTO-6.

I've applied the registry hack for disabling the hidden encryption over the network, no change. Robocopy files to the server runs at full 1Gbps for single data stream, 2-3+Gbps running multiple streams. Backup to the tape server's HDD as a repository (setup as proxy so straight to local E: rather than shared) runs at 145-175MB (1.1-1.4Gbps per job, interestingly enough). Nice and fast so we know our network speed is there as well; we know we can get data from our VMware stack to the tape server itself. It's getting data from VMware straight to tape that is suffering.

Any ideas please?

[Dima P.]

Hi,

If we point it to files on a proxy

Was that a file to tape job? How many files were selected as a source? Thanks

[skrause]

I know this won't resolve the root cause of your issue but this might be a decent workaround: can you have your tape server connect directly to the EquaLogic SAN and use it as a repository that way? (Obviously you would still need to move files between the VMFS volume and the Windows LUN on your tape server).

I have found that anytime the volume that the tape library is pulling data from is not "local" storage on the tape server, performance is poor (though I get better performance than you see, about 20-25MB/s versus the 130 MB/s I average locally on LTO-5).

[rreed]

Hi Dima. It doesn't seem to matter if we use a file to tape or backup copy/to tape across the network, it runs unusable slow either way. Doesn't seem to matter if it's a single File backup or multiple files, but I will need to test and confirm that and other details this week. All the same regarding backing up files or VM; even if we do a backup copy job to tape, if the data comes across the network to tape it's slow.

Skrause, that's the same thing we're finding. If data is not local to the tape server, it's unusable. Single digit MB/s. Only pulling from the server's local HDD will it run at speed. I've seen others report the same issue but aside from the hidden encryption registry hack (done, no change) I haven't found any resolution on this forum or anywhere else on the internets. Since we know we can get data to the tape server at speed, we know the network speed is there. We know we can get data from the tape server to the tape at speed, we know the tape speed is there. It's going directly across network to tape that isn't. I'll be opening a support case this week.

[skrause]

When I was looking at things, I kind of went on the assumption that the issue is caused by the LTO drivers and how they handle traffic that comes in over network versus the storage bus. If it does end up being something with Veeam that can be resolved it would make some of my job a bit easier (though I already copy everything I send to tape to the repository connected to the tape server for on disk retention so it would not change much.)

[Dima P.]

I was under impression that current file to tape performance over network suffers from slow indexing when you have a lot of files as a source. Might be reasonable to test performance for one single file but large size to recognize the average throughput.

Once you got the case ID, kindly, post it to this thread. Thanks.

[rreed]

I've never done more than 4-5 files at a time for File to Tape. Is that considered a lot? And it's just backing up .vbk files if that matter. At that, same performance if I attempt to setup a Backup Copy to tape, if that also matters. I'll report back once I have a firm grip on exact circumstances and a case number.

[skrause]

I've had backup to tape jobs where it was only processing one 1.4TB .VBK (not processing incrementals, just using latest chain) that took 34 hours when it was pulling it over the network but takes about 4 hours when pulling it from a backup copy living on storage attached to the tape server.

[Dima P.]

Thanks. What was written as bottleneck? Any chance you have a global throttling rules, that might affect the bandwidth consumption, enabled in VBR?

[rreed]

No throttling rules for us, our infrastructure is all 10Gb SAN/core, (4)x1Gb TOR to the tape server. All w/in the same data center/rack. When we write to tape from the local tape server, bottleneck is target which is what we would expect over 6Gb SAS. Once this particular tape job is complete I'll run the same job over the network and report back to see what the bottleneck is.

[Dima P.]

Ok, I am out of ideas now. Kindly, open a case and let support team check your logs. Update this thread with the case ID, so we could check how it goes. Thanks!

[rreed]

Thought I had updated this but apparently not. Ticket #01715660, been running your TapeLibTest.exe quite a bit, so far only confirming the same behavior as Veeam; network to local HDD fast (200+ MB/s), local HDD to tape fast (180 MB/s), network to tape slow (~101 MB/s). Our network source is a CIFS share.

Since my last post here we've upgraded our tape server's network connection from (4) x 1Gbps LAG to a single 10Gbps connection. It's helped some but we're still not seeing LTO-6 speeds across a now full 10Gbps infrastructure. Around 101 MB/s average. The only things we've been able to try w/ our tape drive is changing block sizes, but not enough to make a real discernible difference I'm afraid. Registry hack for netuse but it made little, if any difference.

[rreed]

Target is always bottleneck BTW, which it of course should be, but not at ~101 MB/s I wouldn't think when it can do 160-180 MB/s.

[ccatlett1984]

Try this, did wonders for us.
http://www.v-strange.de/index.php/veeam ... o-veeam-v8

[rreed]

Tried the same in v8 and v9 (our Veeam engineer added it our v9 tape server). No discernible difference overall, or at least I don't really know for sure, since he added a slew of registry hacks. It certainly didn't help us in v8. Also, some of our tape jobs increased nicely since the hacks, some are about the same or at best only modestly better.

Close enough, we're kind of done messing w/ it, it is what it is, and we'll just be on w/ our lives. They only run once a month so if it takes a couple weeks to get everything archived off to tape, that's fine. We have warranty on the drive to hopefully help alleviate all the shoe shining it will be doing. Oh well. Close enough.

[skrause]

From reading that article, the registry hack would only have an impact on tape jobs that transport over a network that is set to encrypt traffic (if there is a public IP address on the proxies/repositories it will encrypt network traffic by default). It also would make it so all of your network traffic is unencrypted which you might not like if you are transmitting on outside networks.

Which basically makes it a non-starter for us as we have to have public addresses on some of our machines (heavily firewalled off, but still) and we traverse networks that are owned by other entities on campus that we have no control over so we are required to encrypt all network communications by default.