v9 - Tape performance reports

Everything about backing up to tape

v9 - Tape performance reports

Veeam Logoby kkuszek » Tue Jan 19, 2016 2:54 pm

I just started my first restore from tape since switching to v9 from v8. I need 1 file and as we know normal v8 is to read the whole tape.
it seems to have such a low processing rate to me though. The restore has been running for 18 hours and is at 18%. 100 hours for a restore? processing is pretty steady around 23 MB/S but this is dumping to idle local disk array so I struggle to see why streaming from tape isn't saturating the lto drive speed or even coming closer than 10%.
Funny enough since the job started I kicked off a daily incremental to tape and that's rockin on at 150MB/S while not changing the restore rate of the job in the other drive.

So other tape users - what kind of restore rates are you seeing and what's your configuration? Is this kind of processing rate typical to you?
kkuszek
Enthusiast
 
Posts: 92
Liked: 5 times
Joined: Fri Mar 13, 2015 3:12 pm
Full Name: Kurt Kuszek

Re: v9 restore from tape: how fast does it process for you

Veeam Logoby v.Eremin » Tue Jan 19, 2016 4:07 pm

What component is identified as a major bottleneck? And you're restoring to repository I guess, so what is the connection between tape server and repository?
v.Eremin
Veeam Software
 
Posts: 13543
Liked: 1002 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: v9 restore from tape: how fast does it process for you

Veeam Logoby kkuszek » Tue Jan 19, 2016 4:09 pm

How do I view that on restore from tape job? I don't see the traditional statistics and bottleneck display?
kkuszek
Enthusiast
 
Posts: 92
Liked: 5 times
Joined: Fri Mar 13, 2015 3:12 pm
Full Name: Kurt Kuszek

Re: v9 restore from tape: how fast does it process for you

Veeam Logoby kkuszek » Tue Jan 19, 2016 9:23 pm

P.S.,
I included this in my last ticket.
The tape drive is directly connected to the vmware host running veeam and passed through to the veeam VM. This is on a dedicated host with a 6 disk dedicated local raid 10 array for storage.
kkuszek
Enthusiast
 
Posts: 92
Liked: 5 times
Joined: Fri Mar 13, 2015 3:12 pm
Full Name: Kurt Kuszek

Re: v9 restore from tape: how fast does it process for you

Veeam Logoby v.Eremin » Wed Jan 20, 2016 2:31 pm

So, Virtual Machine having tape drive added as pass-through device to it is playing the role of tape server, what about repository server? Where is located in this scheme?
v.Eremin
Veeam Software
 
Posts: 13543
Liked: 1002 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: v9 restore from tape: how fast does it process for you

Veeam Logoby kkuszek » Wed Jan 20, 2016 2:40 pm

Hi v.Eremin,
This is a rather small environment so backup duty is entirely within 1 machine. It's just an essentials plus deployment, nothing huge.
It's a single dell r710 running vmware, and veeam is the only vm on there with it. I actually have the HBA the tape drive is connected to added as a pass-through device because vmware does not support multi-lun targets strangely enough. The r710 has 6x 3.5in drive slots in the front I populated with 4tb hgst sas drives in raid 10.
kkuszek
Enthusiast
 
Posts: 92
Liked: 5 times
Joined: Fri Mar 13, 2015 3:12 pm
Full Name: Kurt Kuszek

Re: v9 restore from tape: how fast does it process for you

Veeam Logoby goletsa » Fri Feb 12, 2016 9:51 am 1 person likes this post

HP LTO4, SAS HBA, on the same machine, where veeam installed.
Write was about 100MB\s on Veeam v8
But restore rate only 20MB\s on v9 (Restore files from tape, about 400 files 1-3GB per tape (750GB), 3 tapes contain 2.1TB, 3days to restore).
Image
goletsa
Novice
 
Posts: 3
Liked: 1 time
Joined: Fri Feb 12, 2016 9:31 am
Full Name: Alexey Golets

Re: v9 restore from tape: how fast does it process for you

Veeam Logoby Dima P. » Sat Feb 13, 2016 11:28 pm

Alexy,
Restore files from tape, about 400 files 1-3GB per tape (750GB)

What is the average size of the file? Can you open a case at support and let me know the case id, so I could share it with the dev team? Thank you in advance.
Dima P.
Veeam Software
 
Posts: 6494
Liked: 452 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

[MERGED] TL2000 Painfully Slow - v9

Veeam Logoby mongie » Tue Feb 16, 2016 4:40 am

Trying to run tape backup jobs on a new V9 upgraded server.

This was slow in V8 and is just as slow in V9. 6Gb SAS conected LTO6 drive in Dell TL2000 library. I get write speeds of less than 20MB/s. Normally around 12MB/s. Target is the bottleneck... but I don't see why.

I'm wondering if there is anything I can do? Should I log a ticket with Dell or Veeam?
mongie
Expert
 
Posts: 149
Liked: 23 times
Joined: Mon May 16, 2011 4:00 am
Location: Brisbane, Australia
Full Name: Alex Macaronis

Re: TL2000 Painfully Slow - v9

Veeam Logoby v.Eremin » Tue Feb 16, 2016 8:28 am

What kind you job are you running? Files to tape or backups to tape one? What is used as a source for the tape job? Can you provide us with full bottleneck statistics (the one with percent distribution)? Thanks.
v.Eremin
Veeam Software
 
Posts: 13543
Liked: 1002 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: TL2000 Painfully Slow - v9

Veeam Logoby Lulworth » Tue Feb 16, 2016 2:43 pm

I get the same here using a HP 1x8 G2 Autoloader using a LTO 6 drive connected via SAS.

I am running tests at the moment but I never get above 60MBs which is slow compared to what I used to get using backup exec.

An example load for a job is 14/02/2016 13:37:27 :: Load: Source 0% > Proxy 3% > Network 0% > Target 10%
Lulworth
Novice
 
Posts: 3
Liked: never
Joined: Wed Jan 27, 2016 2:01 pm
Full Name: Matthew Bartlett

Re: v9 restore from tape: how fast does it process for you

Veeam Logoby kkuszek » Tue Feb 16, 2016 3:13 pm

Dima,

What is the average size of the file?


I believe that is exactly what OP meant by their post. I interpreted that as each tape containing around 400 files, and they average between 1-3gb in size for each file with a total use of around 750gb per tape and the complete restore was 2.1TB in size.
kkuszek
Enthusiast
 
Posts: 92
Liked: 5 times
Joined: Fri Mar 13, 2015 3:12 pm
Full Name: Kurt Kuszek

Re: TL2000 Painfully Slow - v9

Veeam Logoby Dima P. » Tue Feb 16, 2016 4:37 pm

Hi Matthew,

To clarify, are you backing files via File to tape job or backups via Backup to tape job? If it’s a file to tape job what is the average size of the files (just estimate for better understanding)? Thanks
Dima P.
Veeam Software
 
Posts: 6494
Liked: 452 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: TL2000 Painfully Slow - v9

Veeam Logoby Lulworth » Tue Feb 16, 2016 5:12 pm

HI Dima,

I am using File to Tape. I am copying to tape the backups made from Veeam to my storage area. These are the VBK, VIB and VRB files and vary in size between 10's of MB to 600GB in size so not a lot of small files.

It's currently taking 25 hours to transfer 2.6TB.

Thanks,

Matt
Lulworth
Novice
 
Posts: 3
Liked: never
Joined: Wed Jan 27, 2016 2:01 pm
Full Name: Matthew Bartlett

Re: v9 - Tape performance reports

Veeam Logoby Dima P. » Tue Feb 16, 2016 6:39 pm

Guys,
I’ve merged both performance report threads into one for tracking purposes. Support and QA teams will review all cases posted above.

To all newcomers.
If you see the performance decrease in file to tape job, backup to tape job or restore from tape please open a support case and post the case ID – we will check it. So, please provide:

0. Support case ID
1. Background on your infrastructure setup (i.e. type of the library, how it's connected to tape proxy, generation of tape drives)
2. What type of tape job were you using during the performance testing (file to tape / backup to tape / file from tape restore / backup from tape restore)
3. Bottleneck stats from job details
4. Average size of the file and number of backed up/restored files (a rough estimate is good enough)
5. Any other useful information

Thanks.
Dima P.
Veeam Software
 
Posts: 6494
Liked: 452 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Next

Return to Tape



Who is online

Users browsing this forum: No registered users and 17 guests