v9 - Tape performance reports

Everything about backing up to tape

Re: v9 - Tape performance reports

Veeam Logoby rreed » Mon Apr 04, 2016 2:35 pm

#01715660 for us. 10G network infrastructure for the most part, Dell TL2000 6G SAS connected to Dell PE430 physical server, pulling from EMC Data Domain and Dell DR dedupe devices - DR's are 10G and DD's are 1G.

We were seeing around 20-30 or so MB/s files to tape - don't even try GFS w/ a dedupe source because of the transformations it does, those were 2-3 MB/s, stick to file to tape and live w/ it - I think our DR file to tape jobs (10G) were maybe around 50-60 MB/s.

Before anyone scoffs it off to pulling files from a dedupe device please, we can robocopy files from dedupe to the local tape server HDD at around 100 MB/s from the old 1G connected DD's and ~200-375 MB/s from our 10G connected DR's. Network speed appears to be there. We can backup files to tape from tape server HDD to tape at around 160 MB/s, so tape speed appears to be there (hardware encryption and compression enabled since, they're tapes, and coming from dedupe they're not compressed). Pulling across the network to tape, dismal speeds.

I've been working w/ patient and helpful first tier support for a week or two to experiment w/ different block sizes at the tape drive w/ mixed and inconclusive results. Overall it seemed like V9's default of 1MB blocks might be too large since it seemed to perform better in our setup at somewhere around middle of the road block size. We're currently set at 131072 in Veeam drive properties but per below, I think one of the registry hacks a higher engineer did was to set a different block size. Our fresh V9 install had set them to 1048576. Using Veeams tape performance testing utility did seem to confirm that on average somewhere around middle or lower block size gave best overall performance.

Last week support graciously got us escalated to an engineer, whom after a webex, very respectfully, threw a slew of registry hacks, and jacked w/ the tape server's network card, and got us down to around 12 MB/s copying files from the network to the tape server HDD, and around 823 KB/s tape speed, at which point he declared we clearly have network issues that we need to straighten out and call them back.

After putting back the tape server's NIC to where it was before the webex, we have our network speed back and surprisingly this morning our tape jobs are running pretty decent speed! Around 109-133 MB/s off our 10G DR's, we were getting much lower before. I don't know if it was experimenting w/ block size in Veeam or all the registry hacks that did it, but we're doing reasonably well this morning. I would argue it was not our network since there was no real change there (I had to put it back the way it was before the engineer). I'm going to let the monthly tape jobs run and see how overall performance comes out, and may even try the next smaller block size to see what difference that makes. We're much better, though I can't say what exactly may have fixed it other than try experimenting w/ block size, run a few jobs, and see what works best for you.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: v9 - Tape performance reports

Veeam Logoby rreed » Mon Apr 04, 2016 2:49 pm

Forgot, files to tape job, pulling .vbk's, anywhere from 12-13G to 4-5TB, only around as much 2-3 files at a time. Never seemed to matter what size, same performance. As mentioned though this mornign we're at least in three digit speed, not quite 160 MB/s of LTO-6 yet but way better than 20-30 MB/s.

At that, it would be great if GFS jobs simply did a straight file pull rather than synthetic transform. Kind of like a file to tape GFS job. :wink:
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: v9 - Tape performance reports

Veeam Logoby msmith_uk » Mon Apr 04, 2016 2:55 pm

Dima P. wrote:Hi Matt,

Could you collect and share the info described in this post. Thanks.



Thanks, as requested

0. Support case ID
01742008


1. Background on your infrastructure setup (i.e. type of the library, how it's connected to tape proxy, generation of tape drives)
Physical backup server backing up to SAN with tape drive connected direct to backup server using LTO5 tapes tape drive is a Dell TL2000

2. What type of tape job were you using during the performance testing (file to tape / backup to tape / file from tape restore / backup from tape restore)
Backup to tape

3. Bottleneck stats from job details
Source 0% > Proxy 0% > Network 0% > Target 72%

4. Average size of the file and number of backed up/restored files (a rough estimate is good enough)
18 objects transferring around 17.3gb 19h19m @15mbs

5. Any other useful information:
When backing up to tape it looks as if data rate is being capped at 14mbs
msmith_uk
Lurker
 
Posts: 2
Liked: never
Joined: Fri Apr 01, 2016 3:31 pm
Full Name: matt smith

Re: v9 - Tape performance reports

Veeam Logoby danc » Tue Apr 05, 2016 7:01 am

0. Support case ID - 01745678

1. Background on your infrastructure setup (i.e. type of the library, how it's connected to tape proxy, generation of tape drives)
- HP StoreEver MSL4048 G3 Series LT05 Tape Library, connected to a backup proxy server

2. What type of tape job were you using during the performance testing (file to tape / backup to tape / file from tape restore / backup from tape restore)
- Backup to Tape

3. Bottleneck stats from job details
- Source 0% > Proxy 9% > Network 0% > Target 0%

4. Average size of the file and number of backed up/restored files (a rough estimate is good enough)
- Size of the file - Around 600GB

5. Any other useful information:
When backing up to tape backups don't seem to go higher than 4mb/s
danc
Influencer
 
Posts: 11
Liked: never
Joined: Wed Apr 15, 2015 7:19 am
Full Name: Dan

Re: v9 - Tape performance reports

Veeam Logoby ccatlett1984 » Wed Apr 06, 2016 4:48 pm

Following this article and setting the reg key gave me a 30% increase in tape speed.
Before: 30Mbit/s
After: 50Mbit/s (with spikes to 65Mbit/s)
http://www.v-strange.de/index.php/veeam ... o-veeam-v8

Locally attached (SCSI) LTO4 Autoloader, reading from a synology RS2414RP+ on the same switch.
ccatlett1984
Enthusiast
 
Posts: 83
Liked: 9 times
Joined: Thu Oct 31, 2013 5:11 pm
Full Name: Chris Catlett

Re: v9 - Tape performance reports

Veeam Logoby DaveWatkins » Fri Apr 15, 2016 1:48 am

I'm not entirely sure I've "lost" performance after upgrading to v9, because we've only just migrated to Veeam, but we're also getting pretty poor transfer speeds to Tape.

Tape Drive: SAS connected Dell TL4000 with 2 x LTO6 drives
Job Type: Backup to Tape
Bottleneck Stats: 0% on everything except 4% on target
Average file size: About 100GB I guess, we have some 3TB files from the exchange server, and lots of 40GB for basic windows servers

Tape drives and SAN repo are connected to the same machine

With a single job using both drives in parallel, we get about 60Mb/s sustained, so 30MB/s for each drive. BE was doing double that when it was reading directly from the SAN over FC, this is from a dedicated SAN that holds out backups. Oddly even the jobs we're putting to tape from remote repos (over 1Gb WAN connections) run at the same speed

I've haven't logged a job yet as I still have some testing to do, but so far it doesn't look good
DaveWatkins
Expert
 
Posts: 273
Liked: 68 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: v9 - Tape performance reports

Veeam Logoby rreed » Tue Apr 19, 2016 6:28 pm

Blargh.. Today found some old .vbk files floating around on plain old Windows share (VM, Equallogic SAN, all 10Gbps infrastructure, LTO-6 tape). No problem, it was one of my old proxies; added it back to the console as a Windows Server, created a file to tape job, and pointed to the proxy's HDD (server/local HDD as source). At 2:20 hr/min now, 54MB/s. Around 5-10 .vbk files, 1.4TB total. Ugh. Oh well. My other gripe, and I haven't checked if U1 fixes it (not yet installed, waiting for this job to finish haha), is the Statistics window only shows read speed!!! W/ utmost love and respect to all, I don't care how fast it reads, I only want to know how fast it writes to tape. :wink:
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: v9 - Tape performance reports

Veeam Logoby Didi7 » Thu Apr 21, 2016 11:18 am 2 people like this post

Our current configuration on one of our VBR servers:

Veeam B&R v9.0.0.1491 installed on a physical Windows Server 2012 R2 (ProLiant DL380 G6) and an MSL4048 HP Library currently filled with 2x LTO4 tape drives FC connected. One 'File to Tape Backup' Job and one 'File from Tape Restore' Job was run. The backup files are located on the same VBR 9.x Server.

Unfortunately I cannot restore files from LTO6 tape media written with VBR 8.x at the moment, as the library is still filled with LTO4 drives for a couple of days. Then I could try to restore VBR 8.x LTO6 tape media with VBR 9.x !

Here are the results for backup and restore on LTO4 with VBR 9.x ...

Image

Image

Pretty good, if you ask me.

Write and Restore Performance on a physical VBR 8.x Server onto LTO6 tape media, which is located in an MSL2024 SAS-tape-library gives me 160MB/s troughput by the way.

Regards,
Didi7
Using Veeam Backup & Replication 9.5 Update 2 on every backup server here!
Didi7
Expert
 
Posts: 241
Liked: 16 times
Joined: Fri Oct 17, 2014 8:09 am

Re: v9 - Tape performance reports

Veeam Logoby rreed » Thu Apr 21, 2016 1:14 pm

Having everything all located on the same physical server - and FC connected! - , I wish we could that here. But, alas, we have to pull our source files from across the local network.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: v9 - Tape performance reports

Veeam Logoby Didi7 » Thu Apr 21, 2016 1:47 pm

Well, whether tape drives are FC connected or SAS connected, really doesn't make any difference, as even LTO6 doesn't work faster than 160MB/s, so I get the maximum on VBR 8.x with LTO6. I was even more impressed at how fast LTO4 is on VBR 8.x server with a local backup repository and data, which is saved to tape media, which is physically connected to the VBR-server itself.
Using Veeam Backup & Replication 9.5 Update 2 on every backup server here!
Didi7
Expert
 
Posts: 241
Liked: 16 times
Joined: Fri Oct 17, 2014 8:09 am

Re: v9 - Tape performance reports

Veeam Logoby StevenMeier » Fri Apr 22, 2016 1:21 am

We have a job open and tape through put from repository is rubbish.
Case #01719917

We did find some minor things when looking very closely and have resolved those, but the likely hood of these minor issues affecting tape through put is unlikely.
StevenMeier
Service Provider
 
Posts: 25
Liked: never
Joined: Fri Apr 22, 2016 1:06 am
Full Name: Steven Meier

Re: v9 - Tape performance reports

Veeam Logoby Didi7 » Fri Apr 22, 2016 10:21 am

The tape or library is connected how to the Veeam Backup Server or is the Veeam Backup server virtual? Is the repository data local to the Veeam backup server. What kind of tape hardware do you have and how is it connected to the Veeam Backup server?
Using Veeam Backup & Replication 9.5 Update 2 on every backup server here!
Didi7
Expert
 
Posts: 241
Liked: 16 times
Joined: Fri Oct 17, 2014 8:09 am

Re: v9 - Tape performance reports

Veeam Logoby StevenMeier » Mon Apr 25, 2016 8:31 pm

Hi.

The tape library is a 4 Drive lto 6 library sas attached to two cards , so library is logically partitioned.

The servers is a dl 380 and its ownly role in life is to service veeam tape jobs.


Its connected via an LACP cisco ether channels to switches(2 nics) which are then ether channeled to the switches core switches (2 nics) where the repository Server is based and storage is san attached.

Its all veeam 9 and latest patch levels.

Testing has shown that backup copy jobs between these two servers the network is not the issue....the jobs fly.(we did this by setting a temp repository on tape Server and then from repository server sent the data) so the only different thing in the equation is the Tape device and veeam tape data mover mechanism....its very slow.

Cheers
Steve
StevenMeier
Service Provider
 
Posts: 25
Liked: never
Joined: Fri Apr 22, 2016 1:06 am
Full Name: Steven Meier

Re: v9 - Tape performance reports

Veeam Logoby rreed » Mon Apr 25, 2016 8:36 pm

Hey Steve, do I understand you to indicate network source-->taper server=fast; tape server-->tape=fast; network source-->tape (directly to tape bypassing local tape server storage)=slow?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: v9 - Tape performance reports

Veeam Logoby StevenMeier » Tue Apr 26, 2016 8:45 pm

The Physical Server with SAN attached repositories sits on core switches this Server has an lacp cisco therchannel set up(2 nics).

The core switches are connected via a cisco etherchannel to the edge switches where the physical server resides and has the tape library attached.

This Server does no other veeam role other than Tape jobs.

Testing has shown via the backup copy from the main repository server to this tape server (by creating a test veeam repository)job that the network can shift enough data to provide the tape devices enough data.

Unfortunately the tape jobs run very slowly....the throughput stays well under what the backup copy job can do.

This library is a 4 drive library with 4 lto6 drives in it.(we are only using two at the moment for this testing.

It appears that the tape data mechanism for moving data is not very efficient.

How can we tell this....well

We are still using our backup exec infrastructure on another library.(dual lto6)

This backs up data of same repository server (not at same time as other library) to a server with the library attached and backup exec. This is backup exec 2012 which we want to decommission but looks like we cannot . The data is backed up to this box with no issues of network.
StevenMeier
Service Provider
 
Posts: 25
Liked: never
Joined: Fri Apr 22, 2016 1:06 am
Full Name: Steven Meier

PreviousNext

Return to Tape



Who is online

Users browsing this forum: No registered users and 1 guest