Veeam Tapejob needs very long Time

[WinstonWolf]

Hello,
I have an big Problem with Veeam Backup To Tape Times ( virtual Full Backup to Tape )
Here the environment :

Backup Storage : Netapp FAS2554 with 2 aggregats ( each Aggregat 22 x SATA 4 TB HDDs with 7200K )
Network : 10Gbit Switch Cisco , 10 Gbit Dual Network Card in the Backup Server , 10 Gbit Connection between Server and Storage
Tape Drive : HP MSL 4048 with 3 LTO 5 Tape Drives ( 2x Fibre Channel , 1 x SAS ) direct connected with the Backup Server
ISCSI Connection : ISCSI LUNs on Netapp Storage - ISCSI Connector in W2008R2 Server - Backup Repositorys are the ISCSI Drives in Windows

My Tape Job creates on Saturday Morning an new Mediaset and therefore an new "Virtual Full Backup to Tape"
In the Job Summary i see : Processed 7,8 TB , Read 10,7 TB , Transferred 7,8 TB ( 1,4x ) , Processing Rate : 73 MB/s , Bottleneck Target , Duration : 43:44:09

In the Details of the Job the longest Times are for : Processing xxxxxx.vbk

When the Tape Job runs we start the Resource Manager in Windows and see the Following Picture :



As you can see we have an read rate from 53MBs , i think it is a little to poor for an 10Gbit Connection with 22 x 4 TB SATA HDDs ?

Here you can See the IOPS from the Storage :



For Example : i have 22 x 4 TB SATA HDDs with 7200rpm then i have 80 IOPS x 22 = 1760 IOPS
For me then 250 IOPS as you can see is too poor .

Where it can be the bottleneck in this System ? Perhaps the Switch , Network Card Settings ?

The MTU Size of the Storage ISCSI is set to 9000 MTU.

Thanks for suggestions .

Michael

[Dima P.]

Hi Michael,

Processed 7,8 TB , Read 10,7 TB , Transferred 7,8 TB ( 1,4x ) , Processing Rate : 73 MB/s , Bottleneck Target , Duration : 43:44:09

I doubt its networks since bottleneck stats shows ‘target’. Do you have hardware encryption enabled on the tape library?

[WinstonWolf]

Where i can enable it ?

[veremin]

It's enabled on the library itself. The particular setting depends on the library manufacturer.

However, based on your question, you don't appear to use it.

Thanks.

[WinstonWolf]

Some News for this Case .
I opend an Support Ticket and sent Veeam Support the Logfile for the Problem Job .
They write me back that the Target ( the FC LTO5 Tape Drive ) is the Bottleneck .
But as you can see here on the Job from the Last Weekend , the Target is only with 23% loaded .





What is responsible for the Long Backup Times ? Or is it normal with backing up with an LTO 5 FC Drive only 80MB/s are possible ?

Thanks
Michael

[veremin]

There should be a tool checking tape device performance that is available for support team. You may ask for more thorough tests to check whether given metrics are maximum, indeed. Thanks.

[WinstonWolf]

Here are the Outputs from the TapeLibTest.exe Tool :

Code: Select all

FC Drive 1 in MSL2024 : 

LOG: Initializing log file ...
----------Tape params----------
Device: \\.\Tape0
Total space:1541439225856
Free space: 26525827072
Eom reserved space: 0
Support EOM: false
Block size: 524288
----------Drive params----------
ECC: true
Compression: true
DataPadding: false
ReportSetmarks: false
DefaultBlockSize: 131072
MaximumBlockSize: 524288
MinimumBlockSize:2
MaximumPartitionCount: 2
FeaturesLow: 20127671
FeaturesHigh: 306704957
----------Media params----------
Capacity: 1541439225856
Remaining: 26525827072
BlockSize: 524288
PartitionCount:  1
WriteProtected:  false
Software EOM: 5368709120
Ignore software EOM: false
--------------------------------
Start backup D:\TapeTest\Testfile.iso
Backup to tape D:\TapeTest\Testfile.iso with [b]average speed 92.66MB/s[/b]
Start restore D:\TapeTest\Testfile.iso in D:\TapeTest\Testfile.iso.restore
Restore from tape D:\TapeTest\Testfile.iso.restore with [b]average speed 45.55MB/s[/b]


FC Drive 2 in MSL2024 : 

LOG: Initializing log file ...
----------Tape params----------
Device: \\.\Tape1
Total space:1541439225856
Free space: 26907508736
Eom reserved space: 0
Support EOM: false
Block size: 524288
----------Drive params----------
ECC: true
Compression: false
DataPadding: false
ReportSetmarks: false
DefaultBlockSize: 131072
MaximumBlockSize: 524288
MinimumBlockSize:2
MaximumPartitionCount: 2
FeaturesLow: 20127671
FeaturesHigh: 306704957
----------Media params----------
Capacity: 1541439225856
Remaining: 26907508736
BlockSize: 524288
PartitionCount:  1
WriteProtected:  false
Software EOM: 5368709120
Ignore software EOM: false
--------------------------------
Start backup D:\TapeTest\Testfile.iso
Backup to tape D:\TapeTest\Testfile.iso with [b]average speed 85.28MB/s[/b]
Start restore D:\TapeTest\Testfile.iso in D:\TapeTest\Testfile.iso.restore
Restore from tape D:\TapeTest\Testfile.iso.restore with [b]average speed 41.91MB/s[/b]

SAS Drive in MSL4048 : 

LOG: Initializing log file ...
----------Tape params----------
Device: \\.\Tape2
Total space:1541439225856
Free space: 1541439225856
Eom reserved space: 0
Support EOM: false
Block size: 524288
----------Drive params----------
ECC: true
Compression: false
DataPadding: false
ReportSetmarks: false
DefaultBlockSize: 131072
MaximumBlockSize: 1048576
MinimumBlockSize:2
MaximumPartitionCount: 2
FeaturesLow: 20127671
FeaturesHigh: 306704957
----------Media params----------
Capacity: 1541439225856
Remaining: 1541439225856
BlockSize: 524288
PartitionCount:  1
WriteProtected:  false
Software EOM: 5368709120
Ignore software EOM: false
--------------------------------
Start backup D:\TapeTest\Testfile.iso
Backup to tape D:\TapeTest\Testfile.iso with [b]average speed 76.74MB/s[/b]
Start restore D:\TapeTest\Testfile.iso in D:\TapeTest\Testfile.iso.restore
Restore from tape D:\TapeTest\Testfile.iso.restore with[b] average speed 41.88MB/s[/b]

What you Think abot the Result ?

[veremin]

It appears to be more or less the same as the speed rate you get.

So, given that testing tool has shown the same data and there is no obvious bottleneck, I don't think there is anything else I can propose to you.

Thanks.

[McClane]

You could try setting the blocksize for the tape drive to 64kB. I get about 160MB/s out of a LTO7 drive with LTO6 media at 64kB blocksize (default is 1MB).

[larry]

I get about 160MB/s out of a LTO7 drive

what LTP tape drive are you using? What controller? I am looking at switch to LTO 7 and that speed I would be more than happy with

[rreed]

Experiment w/ different block sizes on the drive (library, drive properties in Veeam). It's the only thing that helped at all w/ ours. Still not as good as it should be, but it's the only thing that helped.

[McClane]

It's an IBM LTO7 in a Quantum Scalar i500 connected to a Brocade FC Switch connected via QLOGIC FC HBAs to the server.

[rreed]

Dang, my expectations of that would be super fast. Try experimenting w/ different block sizes on the tape and see what works. Use a test tape, and erase it before every run.

[gmcoburn]

Hi McClane, are you saying that you get improved tape speed as a result of reducing the block size? What speeds were you getting before changing the block size? Do you still have your testing results on-hand?

We have a new TL4000 tape library with 2 half height IBM ULKT3580-HH7 LTO7 SAS drives connected to a Dell 6Gbps SAS adapter, but we are only seeing a processing rate of around 170MBps when using both drives in Backup to Tape jobs, with the repositories stored on 10Gb connected EqualLogic sans.

George

[pkelly_sts]

If it helps expectations, I easily get 160Mb/s to a SAS LTO-6 drive (HP MSL2024) from 12 x SATA RAID-10 disks. Even from 4 x SATA Raid-5 I can easily get 120Mb/s to the same drive.

Main thing to bear in mind with LTO-7 is to ensure that you can hit the *minimum* streaming speed to avoid shoe-shining the tape/drive.

[gmcoburn]

Thanks for the feedback. I decided to not be lazy yesterday and did my own tests with block sizes ranging from 64KB to 1MB and also found that 64KB block provided me with the highest throughput to tape at 160MBps per tape drive. It also appeared to reduce the shoe-shining. This was just two separate File to Tape jobs, started at the same time, backing up a 42GB file.

Of note the 1MB block size average speed was 130MBps, with the other block sizes speeds being fairly linear up to the 64KB - 160MBps speed.

Oddly there was no difference in speed with Veeam software encryption turned on or off. The backup job does say hardware encryption, but we do not have hardware encryption enabled.

For us there are so many element in play, data\disk read speed, data processing speed for synthetic fulls etc, that I don't think we can expect to approach the full speed of LTO7 drives which is specced at being 300MBps.

For example I am currently running a Backup to Tape job of a single repository (not configured as per-vm) that contains 9 sets of data comprising of 2 Backup (Reverse Incremental) and 7 Backup Copy jobs. The job is set to only back up latest full backups, and synthetic backup built on same day the job has started and with encryption turned on, so in the end the size to go to tape is around 4TB. With parallel processing so 2 drives are in use, at the 50% mark it has taken 3 hours 45 minutes with a Veeam Processing rate of 176MB/s.

This is well under what I had hoped for, as I have another 2 repositories with another 13TB of data that I want to get onto tape within a reasonable time frame.

Any assistance or tips are appreciated.

George

[rreed]

Don't recall if it's been mentioned in your thread, but in several others there are reports of pulling tape backups from across the network being much slower than locally connected HDD's of the physical tape server itself, despite things like 10Gbps infrastructure, fast storage, etc. I understand Veeam to be aware and investigating the issue. It sounds like you've found what others have, that setting block size lower seems to help improve tape performance across the network, counter-intuitively.

[gmcoburn]

Thanks for the heads up rreed.

We do have an infrastructure that is all network connected with the repository server VM connecting to the storage via iscsi and a physical tape drive, all connected at 10Gbps locally. I'll see if I can find the other posts you are referring to so I can get an idea of the issues other are experiencing. I will likely also raise a support ticket with Veeam.

The backup I mention above is now up to 75%, 6 hours and 30 minutes in at a Veeam processing rate of 142MB/s. The job is now on the last backup set in the repository, so just a single drive in operation, but data streaming to that single drive is now down to 70MB/s.

[rreed]

An interesting point in your setup - do I understand your physical tape server is connected to storage via iSCSI rather than just file/CIFS share, etc? Because that's one thing that was suggested to us (we're just using Windows file share to our Windows repositories/CIFS to our dedupe devices). For us the limited SAN repository space we have was labor-prohibitive to connect our tape server to iSCSI infrastructure.

We went round and around w/ support and found the only thing that would help tape across network performance was fiddling w/ the tape drives' block size. Eventually we just gave up and accepted it as-is until they come out w/ a viable solution. As mentioned, last I heard they are aware and working on it but I would still recommend getting a hold of them to help increase visibility, especially since I don't think anyone else is connecting to their storage via iSCSI.

[gmcoburn]

No, we have a repository server which is a Windows 2012 R2 VM, which is connected via VMXNET3 vnics with jumbos frames and RSS enabled, by iscsi to 15TB luns on an EqualLogic san stack. Note that they aren't vmdks but physical luns.

Our tape drive is a separate physical server connected at 10G to the same switch stack as the VM and storage. Its actually in a different room, external to the datacenter.

At this stage I would prefer not to change the repository setup so that the physical tape drive server connects via iSCSi to the storage, as due to other reasons this would require a reorganisation of our national Veeam infrastructure and this would cause "issues".

[rreed]

Haha, so we're on the same page RE: iSCSI exposure to other servers then. RDM's for your repository VM's then? For our Windows repositories they're just vmdk's but it hasn't mattered whether that, CIFS shares, whatever. At the OS level we could pull data from any storage down to the tape server at speed. Backup to tape from tape server at speed. Pull files across the network to tape is slow. Anyways, sounds like a very similar setup to everyone else who's experiencing the same issue. Get you a support ticket going just to make sure nothing is missed, etc. They may do some registry hacks for Veeam to tune it a bit and investigate your tape server's NIC settings.

As a test, stick you a large file or two on your tape server, big as you can fit, and run a test file to tape job from there. See what your speed difference is.