Discussions related to exporting backups to tape and backing up directly to tape.
Post Reply
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Hello ,
We have an Performance Problem with Veeam Backup to Tape .
Here our Environment :
Veeam backup Server connected via Aruba 10Gbit Switch with the Backend Metrocluster and also with the E-2860 Netapp Backup to Disk Repository .
Veeam Backup Server also connected via 10 Gbit Network with an Veeam Backup Proxy Server which is Connected over SAS to an MSL4048 Tape Library with 2 x LTO8 Tape Drives.
This is also our Tape Server.
An LTO8 Drive can be an Write speed from 360 MB/s .
I can see over the task Manager on the Tape Server that over the 10Gbit Network Adapter the recieve Speed is 500MB till 1Gb but the Tape Drive has an Write Speed from only 115 MB/s .
The Tape Library , SAS HBA and the Drives has the newest Firmware and the Windows Server has the newest HP Driver for both installed.

What can here the Problem ? Should we use higher Block Sizes ?

Thanks
Michael
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by Dima P. »

Hello Michael,

Is that a file to tape or backup to tape job? What type of encryption are you using with your tape setup: software or hardware? Any chance you are using compression for tape jobs? Thank you!
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Hello ,
It is an Backup to Tape Job . I use no encryption . I use no compressin because the backup Files are comressed by the Backup to Disk job too.

Is it a difference if I backup synthetic Fulls or Active Fulls to tape? At the moment we backup Active Fulls to tape.

Thanks
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Ticket on Veeam is already opened : Case # 03892941 - Poor Tape write Performance on an MSL2024 with 2 x LTO8
Michael
Logfiles are uploaded .
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by Dima P. »

Thanks for the details. There will be a performance decrease during synthesized full back while original backup chain is stored on the deduplication repository (since data must be hydrated prior storing the full backup on tape).
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Ok , I understand .
That means when i have synthetic Full Backup instead of active Full Backups the i become better performance ?

Thanks
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Hello ?
Has anybody an Solution for my poor Backup Performance ?
Thanks
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by Regnor »

WinstonWolf wrote: Dec 03, 2019 3:43 pm That means when i have synthetic Full Backup instead of active Full Backups the i become better performance ?
No, synthetic operations will decrease performance.
Also a Synthetic Full on a ReFS drive will have less performance than an Active Full (to tape).
500MB till 1Gb
Is it 500MB/s to 1GB/s, or 500Mbit/s to 1Gbit/s?
And do you see peaks or cuts in the network performance?

Some general questions:
1. What performance do you get when restoring from your Repository? (Full-VM/Disk)
2. Can you check the performance when directly running a file/backup to tape job from your Tape server?
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Hello ,
Yesterday i creat an new "TEST" Backup to Disk Job .
I let the Job run and after this i let the Job runs 2 times more. Means i have a Active Full and 2 incrementals .
This Job i backup to Tape . I have an Write Performance of 290MB to Tape .
But when i try to backup an existing Backup Job ( 14 Days Retention Period , inremental forever , with weekly Active Fulls ) the i have an write Speed to Tape only from 110 MB
Here an Screenshot when i try to restore the complete VM over 10Gbit LAN Network from the Backup Repository to the Metrocluster over over my backup Proxy Server:
Image

Michael
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Here is the Speed when i try to Backup an File to Tape .
Image

Image
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Hello again ,
Here the Plan of our Backup to tape System :
Image
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by Regnor »

It looks like your disk repository is the bottleneck here. Do you have duplication enabled on the NetApp or inside Windows?
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Hello ,
I have neighter Dupliction on the E-Series Backup Repository nor on the Windows 2019 Veeam backup Server .
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by Dima P. »

Can you test the performance with file to tape job from a mentioned proxy server (i.e. try to backup a single file from the proxy server) and share the output? Did i get it right that within NetApp you host ReFS volumes and those are used as repository? Thanks!
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by Regnor »

Can you post a screenshot from the job statistics of the slow backup to tape job?
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Which are the best Settings on Veeam Backup Server and on the E-Series for best read Performance ?
We use 3 ISCSI Drives which are 3 Volumes in one Discpool on the E-Series . Flow Control is on Netapp Side on Aruba Switch Side and also on W2019 Server Side on .
Is it possible that we become better Performance when we insert SSD Drives as Cache in the E-Series ?

Thanks
tmckinnon
Novice
Posts: 8
Liked: 3 times
Joined: Mar 15, 2012 9:20 pm
Full Name: Travis McKinnon
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by tmckinnon »

I would suggest obtain a baseline for network I/O & the tape library as we can use it to identify points of contention.

The tape library can be benchmarked via the HPE Library and Tape Tools - https://www.hpe.com/au/en/product-catal ... 06729.html
This will show if you have any issues with HBAs, Cables, Tape Media and/or the library itself.

Next I would try writing a local file from the tape server to tape, maybe 200-500 GB if you can or a old .vbk backup.

Get that data and we'll look at the back end (Network/NetApp etc)
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Hello ,
Sorry for the long time waiting for the answer.
Now after some Tests we found the reason for this behaviour .

Test 1 : Copy an local *.vbk File ( 20GB ) from the Tape Proxy as "File to Tape Job" = 300 MB/s Tape Write Speed
Test 2 : Copy an local *.vbk File ( 20GB ) from the Main Backup Proxy as "File to Tape Job" = 300 MB/s Tape Write Speed
Test 3 : Create an Active Full Backup to Disk Job with 4 VMs , Create an Tape Job as "Backup to Tape" and backup this "Backup to Disk Job" to Tape = 300 MB/s Tape Write Speed
Test 4 : Create an "File to Tape" Backup to Tape Job with an *.vbk File direct from within the Backup Repository = 300 MB/s Tape Write Speed

I think the Problem is that we have only two Tape Drives , therefore i create two Backup to Tape Jobs .
Every "Backup to Tape" Job inserts 4 "Backup to Disk" Jobs .

That means we have simultaneously run two "Backup to Tape" Jobs which have to Backup altogether 8 "Backup to Disk" Jobs .

I think this is the reason why we have not the 300 MB/s per Tape Job .
Because we have two Jobs with 150-160MB/s
Can it be the reason ? Is there the Limit ?


Michael
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by Dima P. » 1 person likes this post

Michael,

There should be no limit. Data is written to a single tape drive sequentially, so there is difference how many disk jobs you set as a source for backup to tape job. Based on your tests above - performance looks good, so I'd raise a support case and let our support team investigate your original setup. They can say precisely what is acting as a bottleneck based on the debug logs.
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Poor Tape write Performance on an MSL2024 with 2 x LTO8

Post by WinstonWolf »

Hello ,
Now we make an new Setting .
We set the "Max concurrent tasks setting" per Repository from 6 to 4 . Maybe this brings more speed at Backup to Tape .

Michael
Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests