Discussions specific to tape backups
WinstonWolf
Expert
Posts: 187
Liked: 4 times
Joined: Jan 06, 2011 8:33 am
Contact:

Veeam Tapejob needs very long Time

Post by WinstonWolf » Apr 26, 2016 7:26 am

Hello,
I have an big Problem with Veeam Backup To Tape Times ( virtual Full Backup to Tape )
Here the environment :

Backup Storage : Netapp FAS2554 with 2 aggregats ( each Aggregat 22 x SATA 4 TB HDDs with 7200K )
Network : 10Gbit Switch Cisco , 10 Gbit Dual Network Card in the Backup Server , 10 Gbit Connection between Server and Storage
Tape Drive : HP MSL 4048 with 3 LTO 5 Tape Drives ( 2x Fibre Channel , 1 x SAS ) direct connected with the Backup Server
ISCSI Connection : ISCSI LUNs on Netapp Storage - ISCSI Connector in W2008R2 Server - Backup Repositorys are the ISCSI Drives in Windows

My Tape Job creates on Saturday Morning an new Mediaset and therefore an new "Virtual Full Backup to Tape"
In the Job Summary i see : Processed 7,8 TB , Read 10,7 TB , Transferred 7,8 TB ( 1,4x ) , Processing Rate : 73 MB/s , Bottleneck Target , Duration : 43:44:09

In the Details of the Job the longest Times are for : Processing xxxxxx.vbk

When the Tape Job runs we start the Resource Manager in Windows and see the Following Picture :

Image

As you can see we have an read rate from 53MBs , i think it is a little to poor for an 10Gbit Connection with 22 x 4 TB SATA HDDs ?

Here you can See the IOPS from the Storage :

Image

For Example : i have 22 x 4 TB SATA HDDs with 7200rpm then i have 80 IOPS x 22 = 1760 IOPS
For me then 250 IOPS as you can see is too poor .

Where it can be the bottleneck in this System ? Perhaps the Switch , Network Card Settings ?

The MTU Size of the Storage ISCSI is set to 9000 MTU.

Thanks for suggestions .

Michael

Dima P.
Product Manager
Posts: 9905
Liked: 789 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Veeam Tapejob needs very long Time

Post by Dima P. » Apr 27, 2016 4:51 pm

Hi Michael,
Processed 7,8 TB , Read 10,7 TB , Transferred 7,8 TB ( 1,4x ) , Processing Rate : 73 MB/s , Bottleneck Target , Duration : 43:44:09
I doubt its networks since bottleneck stats shows ‘target’. Do you have hardware encryption enabled on the tape library?

WinstonWolf
Expert
Posts: 187
Liked: 4 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Veeam Tapejob needs very long Time

Post by WinstonWolf » Apr 28, 2016 4:34 pm

Where i can enable it ?

v.eremin
Product Manager
Posts: 16192
Liked: 1322 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam Tapejob needs very long Time

Post by v.eremin » Apr 29, 2016 9:12 am

It's enabled on the library itself. The particular setting depends on the library manufacturer.

However, based on your question, you don't appear to use it.

Thanks.

WinstonWolf
Expert
Posts: 187
Liked: 4 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Veeam Tapejob needs very long Time

Post by WinstonWolf » May 02, 2016 6:41 am

Some News for this Case .
I opend an Support Ticket and sent Veeam Support the Logfile for the Problem Job .
They write me back that the Target ( the FC LTO5 Tape Drive ) is the Bottleneck .
But as you can see here on the Job from the Last Weekend , the Target is only with 23% loaded .

Image

Image

What is responsible for the Long Backup Times ? Or is it normal with backing up with an LTO 5 FC Drive only 80MB/s are possible ?

Thanks
Michael

v.eremin
Product Manager
Posts: 16192
Liked: 1322 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam Tapejob needs very long Time

Post by v.eremin » May 02, 2016 9:09 am

There should be a tool checking tape device performance that is available for support team. You may ask for more thorough tests to check whether given metrics are maximum, indeed. Thanks.

WinstonWolf
Expert
Posts: 187
Liked: 4 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Veeam Tapejob needs very long Time

Post by WinstonWolf » May 03, 2016 9:07 am

Here are the Outputs from the TapeLibTest.exe Tool :

Code: Select all

FC Drive 1 in MSL2024 : 

LOG: Initializing log file ...
----------Tape params----------
Device: \\.\Tape0
Total space:1541439225856
Free space: 26525827072
Eom reserved space: 0
Support EOM: false
Block size: 524288
----------Drive params----------
ECC: true
Compression: true
DataPadding: false
ReportSetmarks: false
DefaultBlockSize: 131072
MaximumBlockSize: 524288
MinimumBlockSize:2
MaximumPartitionCount: 2
FeaturesLow: 20127671
FeaturesHigh: 306704957
----------Media params----------
Capacity: 1541439225856
Remaining: 26525827072
BlockSize: 524288
PartitionCount:  1
WriteProtected:  false
Software EOM: 5368709120
Ignore software EOM: false
--------------------------------
Start backup D:\TapeTest\Testfile.iso
Backup to tape D:\TapeTest\Testfile.iso with [b]average speed 92.66MB/s[/b]
Start restore D:\TapeTest\Testfile.iso in D:\TapeTest\Testfile.iso.restore
Restore from tape D:\TapeTest\Testfile.iso.restore with [b]average speed 45.55MB/s[/b]


FC Drive 2 in MSL2024 : 

LOG: Initializing log file ...
----------Tape params----------
Device: \\.\Tape1
Total space:1541439225856
Free space: 26907508736
Eom reserved space: 0
Support EOM: false
Block size: 524288
----------Drive params----------
ECC: true
Compression: false
DataPadding: false
ReportSetmarks: false
DefaultBlockSize: 131072
MaximumBlockSize: 524288
MinimumBlockSize:2
MaximumPartitionCount: 2
FeaturesLow: 20127671
FeaturesHigh: 306704957
----------Media params----------
Capacity: 1541439225856
Remaining: 26907508736
BlockSize: 524288
PartitionCount:  1
WriteProtected:  false
Software EOM: 5368709120
Ignore software EOM: false
--------------------------------
Start backup D:\TapeTest\Testfile.iso
Backup to tape D:\TapeTest\Testfile.iso with [b]average speed 85.28MB/s[/b]
Start restore D:\TapeTest\Testfile.iso in D:\TapeTest\Testfile.iso.restore
Restore from tape D:\TapeTest\Testfile.iso.restore with [b]average speed 41.91MB/s[/b]

SAS Drive in MSL4048 : 

LOG: Initializing log file ...
----------Tape params----------
Device: \\.\Tape2
Total space:1541439225856
Free space: 1541439225856
Eom reserved space: 0
Support EOM: false
Block size: 524288
----------Drive params----------
ECC: true
Compression: false
DataPadding: false
ReportSetmarks: false
DefaultBlockSize: 131072
MaximumBlockSize: 1048576
MinimumBlockSize:2
MaximumPartitionCount: 2
FeaturesLow: 20127671
FeaturesHigh: 306704957
----------Media params----------
Capacity: 1541439225856
Remaining: 1541439225856
BlockSize: 524288
PartitionCount:  1
WriteProtected:  false
Software EOM: 5368709120
Ignore software EOM: false
--------------------------------
Start backup D:\TapeTest\Testfile.iso
Backup to tape D:\TapeTest\Testfile.iso with [b]average speed 76.74MB/s[/b]
Start restore D:\TapeTest\Testfile.iso in D:\TapeTest\Testfile.iso.restore
Restore from tape D:\TapeTest\Testfile.iso.restore with[b] average speed 41.88MB/s[/b]

What you Think abot the Result ?

v.eremin
Product Manager
Posts: 16192
Liked: 1322 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam Tapejob needs very long Time

Post by v.eremin » May 03, 2016 1:37 pm

It appears to be more or less the same as the speed rate you get.

So, given that testing tool has shown the same data and there is no obvious bottleneck, I don't think there is anything else I can propose to you.

Thanks.

McClane
Expert
Posts: 104
Liked: 11 times
Joined: Jun 20, 2009 12:47 pm
Contact:

Re: Veeam Tapejob needs very long Time

Post by McClane » May 03, 2016 2:11 pm

You could try setting the blocksize for the tape drive to 64kB. I get about 160MB/s out of a LTO7 drive with LTO6 media at 64kB blocksize (default is 1MB).

larry
Expert
Posts: 387
Liked: 92 times
Joined: Mar 24, 2010 5:47 pm
Full Name: Larry Walker
Contact:

Re: Veeam Tapejob needs very long Time

Post by larry » May 05, 2016 12:48 pm

McClane wrote: I get about 160MB/s out of a LTO7 drive
what LTP tape drive are you using? What controller? I am looking at switch to LTO 7 and that speed I would be more than happy with :D

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Veeam Tapejob needs very long Time

Post by rreed » May 05, 2016 6:53 pm

Experiment w/ different block sizes on the drive (library, drive properties in Veeam). It's the only thing that helped at all w/ ours. Still not as good as it should be, but it's the only thing that helped.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

McClane
Expert
Posts: 104
Liked: 11 times
Joined: Jun 20, 2009 12:47 pm
Contact:

Re: Veeam Tapejob needs very long Time

Post by McClane » May 05, 2016 10:21 pm 1 person likes this post

It's an IBM LTO7 in a Quantum Scalar i500 connected to a Brocade FC Switch connected via QLOGIC FC HBAs to the server.

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Veeam Tapejob needs very long Time

Post by rreed » May 06, 2016 1:13 pm

Dang, my expectations of that would be super fast. Try experimenting w/ different block sizes on the tape and see what works. Use a test tape, and erase it before every run.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

gmcoburn
Influencer
Posts: 15
Liked: never
Joined: Aug 28, 2014 5:23 am
Full Name: George Coburn
Contact:

Re: Veeam Tapejob needs very long Time

Post by gmcoburn » Jun 01, 2016 12:57 am

Hi McClane, are you saying that you get improved tape speed as a result of reducing the block size? What speeds were you getting before changing the block size? Do you still have your testing results on-hand?

We have a new TL4000 tape library with 2 half height IBM ULKT3580-HH7 LTO7 SAS drives connected to a Dell 6Gbps SAS adapter, but we are only seeing a processing rate of around 170MBps when using both drives in Backup to Tape jobs, with the repositories stored on 10Gb connected EqualLogic sans.

George

pkelly_sts
Expert
Posts: 568
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: Veeam Tapejob needs very long Time

Post by pkelly_sts » Jun 01, 2016 4:02 pm

If it helps expectations, I easily get 160Mb/s to a SAS LTO-6 drive (HP MSL2024) from 12 x SATA RAID-10 disks. Even from 4 x SATA Raid-5 I can easily get 120Mb/s to the same drive.

Main thing to bear in mind with LTO-7 is to ensure that you can hit the *minimum* streaming speed to avoid shoe-shining the tape/drive.

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests