Veeam Tapejob needs very long Time

Everything about backing up to tape

Veeam Tapejob needs very long Time

Veeam Logoby WinstonWolf » Tue Apr 26, 2016 7:26 am

Hello,
I have an big Problem with Veeam Backup To Tape Times ( virtual Full Backup to Tape )
Here the environment :

Backup Storage : Netapp FAS2554 with 2 aggregats ( each Aggregat 22 x SATA 4 TB HDDs with 7200K )
Network : 10Gbit Switch Cisco , 10 Gbit Dual Network Card in the Backup Server , 10 Gbit Connection between Server and Storage
Tape Drive : HP MSL 4048 with 3 LTO 5 Tape Drives ( 2x Fibre Channel , 1 x SAS ) direct connected with the Backup Server
ISCSI Connection : ISCSI LUNs on Netapp Storage - ISCSI Connector in W2008R2 Server - Backup Repositorys are the ISCSI Drives in Windows

My Tape Job creates on Saturday Morning an new Mediaset and therefore an new "Virtual Full Backup to Tape"
In the Job Summary i see : Processed 7,8 TB , Read 10,7 TB , Transferred 7,8 TB ( 1,4x ) , Processing Rate : 73 MB/s , Bottleneck Target , Duration : 43:44:09

In the Details of the Job the longest Times are for : Processing xxxxxx.vbk

When the Tape Job runs we start the Resource Manager in Windows and see the Following Picture :

Image

As you can see we have an read rate from 53MBs , i think it is a little to poor for an 10Gbit Connection with 22 x 4 TB SATA HDDs ?

Here you can See the IOPS from the Storage :

Image

For Example : i have 22 x 4 TB SATA HDDs with 7200rpm then i have 80 IOPS x 22 = 1760 IOPS
For me then 250 IOPS as you can see is too poor .

Where it can be the bottleneck in this System ? Perhaps the Switch , Network Card Settings ?

The MTU Size of the Storage ISCSI is set to 9000 MTU.

Thanks for suggestions .

Michael
WinstonWolf
Expert
 
Posts: 181
Liked: 4 times
Joined: Thu Jan 06, 2011 8:33 am

Re: Veeam Tapejob needs very long Time

Veeam Logoby Dima P. » Wed Apr 27, 2016 4:51 pm

Hi Michael,
Processed 7,8 TB , Read 10,7 TB , Transferred 7,8 TB ( 1,4x ) , Processing Rate : 73 MB/s , Bottleneck Target , Duration : 43:44:09

I doubt its networks since bottleneck stats shows ‘target’. Do you have hardware encryption enabled on the tape library?
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Veeam Tapejob needs very long Time

Veeam Logoby WinstonWolf » Thu Apr 28, 2016 4:34 pm

Where i can enable it ?
WinstonWolf
Expert
 
Posts: 181
Liked: 4 times
Joined: Thu Jan 06, 2011 8:33 am

Re: Veeam Tapejob needs very long Time

Veeam Logoby v.Eremin » Fri Apr 29, 2016 9:12 am

It's enabled on the library itself. The particular setting depends on the library manufacturer.

However, based on your question, you don't appear to use it.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Veeam Tapejob needs very long Time

Veeam Logoby WinstonWolf » Mon May 02, 2016 6:41 am

Some News for this Case .
I opend an Support Ticket and sent Veeam Support the Logfile for the Problem Job .
They write me back that the Target ( the FC LTO5 Tape Drive ) is the Bottleneck .
But as you can see here on the Job from the Last Weekend , the Target is only with 23% loaded .

Image

Image

What is responsible for the Long Backup Times ? Or is it normal with backing up with an LTO 5 FC Drive only 80MB/s are possible ?

Thanks
Michael
WinstonWolf
Expert
 
Posts: 181
Liked: 4 times
Joined: Thu Jan 06, 2011 8:33 am

Re: Veeam Tapejob needs very long Time

Veeam Logoby v.Eremin » Mon May 02, 2016 9:09 am

There should be a tool checking tape device performance that is available for support team. You may ask for more thorough tests to check whether given metrics are maximum, indeed. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Veeam Tapejob needs very long Time

Veeam Logoby WinstonWolf » Tue May 03, 2016 9:07 am

Here are the Outputs from the TapeLibTest.exe Tool :

Code: Select all
FC Drive 1 in MSL2024 :

LOG: Initializing log file ...
----------Tape params----------
Device: \\.\Tape0
Total space:1541439225856
Free space: 26525827072
Eom reserved space: 0
Support EOM: false
Block size: 524288
----------Drive params----------
ECC: true
Compression: true
DataPadding: false
ReportSetmarks: false
DefaultBlockSize: 131072
MaximumBlockSize: 524288
MinimumBlockSize:2
MaximumPartitionCount: 2
FeaturesLow: 20127671
FeaturesHigh: 306704957
----------Media params----------
Capacity: 1541439225856
Remaining: 26525827072
BlockSize: 524288
PartitionCount:  1
WriteProtected:  false
Software EOM: 5368709120
Ignore software EOM: false
--------------------------------
Start backup D:\TapeTest\Testfile.iso
Backup to tape D:\TapeTest\Testfile.iso with [b]average speed 92.66MB/s[/b]
Start restore D:\TapeTest\Testfile.iso in D:\TapeTest\Testfile.iso.restore
Restore from tape D:\TapeTest\Testfile.iso.restore with [b]average speed 45.55MB/s[/b]


FC Drive 2 in MSL2024 :

LOG: Initializing log file ...
----------Tape params----------
Device: \\.\Tape1
Total space:1541439225856
Free space: 26907508736
Eom reserved space: 0
Support EOM: false
Block size: 524288
----------Drive params----------
ECC: true
Compression: false
DataPadding: false
ReportSetmarks: false
DefaultBlockSize: 131072
MaximumBlockSize: 524288
MinimumBlockSize:2
MaximumPartitionCount: 2
FeaturesLow: 20127671
FeaturesHigh: 306704957
----------Media params----------
Capacity: 1541439225856
Remaining: 26907508736
BlockSize: 524288
PartitionCount:  1
WriteProtected:  false
Software EOM: 5368709120
Ignore software EOM: false
--------------------------------
Start backup D:\TapeTest\Testfile.iso
Backup to tape D:\TapeTest\Testfile.iso with [b]average speed 85.28MB/s[/b]
Start restore D:\TapeTest\Testfile.iso in D:\TapeTest\Testfile.iso.restore
Restore from tape D:\TapeTest\Testfile.iso.restore with [b]average speed 41.91MB/s[/b]

SAS Drive in MSL4048 :

LOG: Initializing log file ...
----------Tape params----------
Device: \\.\Tape2
Total space:1541439225856
Free space: 1541439225856
Eom reserved space: 0
Support EOM: false
Block size: 524288
----------Drive params----------
ECC: true
Compression: false
DataPadding: false
ReportSetmarks: false
DefaultBlockSize: 131072
MaximumBlockSize: 1048576
MinimumBlockSize:2
MaximumPartitionCount: 2
FeaturesLow: 20127671
FeaturesHigh: 306704957
----------Media params----------
Capacity: 1541439225856
Remaining: 1541439225856
BlockSize: 524288
PartitionCount:  1
WriteProtected:  false
Software EOM: 5368709120
Ignore software EOM: false
--------------------------------
Start backup D:\TapeTest\Testfile.iso
Backup to tape D:\TapeTest\Testfile.iso with [b]average speed 76.74MB/s[/b]
Start restore D:\TapeTest\Testfile.iso in D:\TapeTest\Testfile.iso.restore
Restore from tape D:\TapeTest\Testfile.iso.restore with[b] average speed 41.88MB/s[/b]


What you Think abot the Result ?
WinstonWolf
Expert
 
Posts: 181
Liked: 4 times
Joined: Thu Jan 06, 2011 8:33 am

Re: Veeam Tapejob needs very long Time

Veeam Logoby v.Eremin » Tue May 03, 2016 1:37 pm

It appears to be more or less the same as the speed rate you get.

So, given that testing tool has shown the same data and there is no obvious bottleneck, I don't think there is anything else I can propose to you.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Veeam Tapejob needs very long Time

Veeam Logoby McClane » Tue May 03, 2016 2:11 pm

You could try setting the blocksize for the tape drive to 64kB. I get about 160MB/s out of a LTO7 drive with LTO6 media at 64kB blocksize (default is 1MB).
McClane
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Sat Jun 20, 2009 12:47 pm

Re: Veeam Tapejob needs very long Time

Veeam Logoby larry » Thu May 05, 2016 12:48 pm

McClane wrote: I get about 160MB/s out of a LTO7 drive
what LTP tape drive are you using? What controller? I am looking at switch to LTO 7 and that speed I would be more than happy with :D
larry
Expert
 
Posts: 372
Liked: 89 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

Re: Veeam Tapejob needs very long Time

Veeam Logoby rreed » Thu May 05, 2016 6:53 pm

Experiment w/ different block sizes on the drive (library, drive properties in Veeam). It's the only thing that helped at all w/ ours. Still not as good as it should be, but it's the only thing that helped.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Veeam Tapejob needs very long Time

Veeam Logoby McClane » Thu May 05, 2016 10:21 pm 1 person likes this post

It's an IBM LTO7 in a Quantum Scalar i500 connected to a Brocade FC Switch connected via QLOGIC FC HBAs to the server.
McClane
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Sat Jun 20, 2009 12:47 pm

Re: Veeam Tapejob needs very long Time

Veeam Logoby rreed » Fri May 06, 2016 1:13 pm

Dang, my expectations of that would be super fast. Try experimenting w/ different block sizes on the tape and see what works. Use a test tape, and erase it before every run.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Veeam Tapejob needs very long Time

Veeam Logoby gmcoburn » Wed Jun 01, 2016 12:57 am

Hi McClane, are you saying that you get improved tape speed as a result of reducing the block size? What speeds were you getting before changing the block size? Do you still have your testing results on-hand?

We have a new TL4000 tape library with 2 half height IBM ULKT3580-HH7 LTO7 SAS drives connected to a Dell 6Gbps SAS adapter, but we are only seeing a processing rate of around 170MBps when using both drives in Backup to Tape jobs, with the repositories stored on 10Gb connected EqualLogic sans.

George
gmcoburn
Influencer
 
Posts: 15
Liked: never
Joined: Thu Aug 28, 2014 5:23 am
Full Name: George Coburn

Re: Veeam Tapejob needs very long Time

Veeam Logoby pkelly_sts » Wed Jun 01, 2016 4:02 pm

If it helps expectations, I easily get 160Mb/s to a SAS LTO-6 drive (HP MSL2024) from 12 x SATA RAID-10 disks. Even from 4 x SATA Raid-5 I can easily get 120Mb/s to the same drive.

Main thing to bear in mind with LTO-7 is to ensure that you can hit the *minimum* streaming speed to avoid shoe-shining the tape/drive.
[New Sig: PLEASE get GFS tape support for incrementals!!!]
pkelly_sts
Expert
 
Posts: 504
Liked: 55 times
Joined: Thu Jun 13, 2013 10:08 am
Full Name: Paul Kelly

Next

Return to Tape



Who is online

Users browsing this forum: Yahoo [Bot] and 7 guests