Discussions specific to tape backups
stewsie
Enthusiast
Posts: 62
Liked: 2 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Another question about tape backup performance

Post by stewsie » Nov 29, 2018 8:21 am

The performance of tape backup jobs for me can be very very slow. Sometimes as slow as 4 MB/s. I have recently ran a tape job to use the repository as the source and the throughput for the job was 94 MB/s. I am now running another job using the repository and again the job is running at 94 MB/s. What on earth happens when I use a backup or copy job as the source? It seems to be doing a lot of stuff on disk compared to using the repository as the tape job type.

v.Eremin
Veeam Software
Posts: 15412
Liked: 1164 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Another question about tape backup performance

Post by v.Eremin » Nov 29, 2018 1:57 pm

Are those jobs configured identically (besides source, of course)? What about virtual full settings? What bottleneck statistics show? And can you also shed some light on your backup infrastructure - how everything is configured? Thanks!

stewsie
Enthusiast
Posts: 62
Liked: 2 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Another question about tape backup performance

Post by stewsie » Nov 29, 2018 3:40 pm

Jobs are identical using the same storage but different repositories. There are no virtual full backups configured. The backups are incremental. Backup infra is EMC 5300 and CIFS. We are experiencing issues at the moment with copy performance and I have a support call open but tape performance is always slow apart from when I have now used the repository backup

v.Eremin
Veeam Software
Posts: 15412
Liked: 1164 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Another question about tape backup performance

Post by v.Eremin » Nov 29, 2018 3:54 pm

Hmm, had to guess based on forum correspondence. So, kindly, keep working with our support team on finding the root cause of the degraded performance. Thanks!

mkretzer
Expert
Posts: 413
Liked: 82 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer » Nov 29, 2018 8:29 pm 1 person likes this post

Sorry but tape functionality is just bad. Case 02772195.

I am quite frustrated about the whole situation.

I just do not get why a backup solution which is 10 years old can do streaming multiple files at the same time (thus using the full performance of the source storage) but not Veeam. It just reads one file, single stream (yes it know about datamoverlocalfastpath - it changes nothing about the basic issue).

How in the world can i keep a LTO 8 tape streaming with someting like this?

We used a HUS110 with 96 disks as source and tape speed was often below 100 MB/s. Now we switched to a G200 midrange storage with 128 disks canhich can do > 1 GB/s when used as source for copy jobs but with single steam tape jobs it just does the 100 MB/s again!

This is just unacceptable. I do not want to purchase SSD storage just to keep my tape streaming.

Again, i am a HUGE veeam fan and i have convinced alot of people to use Veeam but i would never reccomend it if Tape is important...

Dima P.
Veeam Software
Posts: 8949
Liked: 664 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Another question about tape backup performance

Post by Dima P. » Nov 30, 2018 8:22 pm

mkretzer,

May I ask if you are using bundled SQL server for your Veeam B&R installation or by any change you have installed at least SQL Server Standard edition for your backup server database? Thank you in advance.

mkretzer
Expert
Posts: 413
Liked: 82 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer » Nov 30, 2018 10:27 pm

Hello Dima,

no we installed SQL 2016 (and now 2017) Enterprise.

Markus

mkretzer
Expert
Posts: 413
Liked: 82 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer » Dec 03, 2018 9:04 pm

@dima p. Why did you ask?

Dima P.
Veeam Software
Posts: 8949
Liked: 664 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Another question about tape backup performance

Post by Dima P. » Dec 04, 2018 10:59 am

mkretzer,

Thanks! In most of the cases performance issues are caused by SQL Express bundled with Veeam B&R installation (due to the it's limitation of 4GB per instance). I've checked both support cases and it looks like you are investigating why source is marked as bottleneck. Please let us know how the investigation goes. Cheers!

mkretzer
Expert
Posts: 413
Liked: 82 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer » Dec 04, 2018 8:01 pm

We did 5 tests today:

- GFS Backup: ~100 MB/s
- Non-GFS Backup: We got ~250 MB/s and bottleneck switching between source and target
- The requested diskspd test with one thread: 436 MB/s
- The requested diskspd test with three threads: 1620 MB/s
- A second diskspd test with three threads from the second part of the SOBR (same backend system, but lower volume load at the time): 2188 MB/s

This test shows exactly why i find single stream backups absolutely unacceptable: Our storage can provide so much IO/MB/s which is just not used by Veeam.

Markus

Regnor
Veeam ProPartner
Posts: 252
Liked: 53 times
Joined: Jan 31, 2011 11:17 am
Contact:

Re: Another question about tape backup performance

Post by Regnor » Dec 05, 2018 8:18 am

What did you test with GFS and Non-GFS backup? Are those tape jobs assigned to different (GFS) media pools or tape hardware?

In most of our environments I can see Veeam achieving 80-100% of the performance of a single drive, so in general it should be faster.

mkretzer
Expert
Posts: 413
Liked: 82 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer » Dec 05, 2018 6:45 pm

All exactly the same. I used a tape one time in a GFS pool and then in a non-GFS pool with the same drive hardware. Before we used LTO-6 and also did not reach the advertised speed.

You really have seen a LTO8 time stream at 300-360 MB/s with GFS and non-Flash source storage?

Dima P.
Veeam Software
Posts: 8949
Liked: 664 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Another question about tape backup performance

Post by Dima P. » Dec 05, 2018 9:02 pm

mkretzer,

Thanks for sharing! How your tape library is connected to the repository? Since you are talking about GFS tape jobs mind me asking need to know how the source backup jobs are configured (are they set to perform periodic full backups or not)? Cheers!

mkretzer
Expert
Posts: 413
Liked: 82 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer » Dec 07, 2018 6:25 am

Dima,

also by FC (8 GBit).
We have periodic fulls 1-2 days before the GFS backup. I have the feeling this helps a bit.

Best would be to have GFS take only these fulls as a normal job would - normal jobs are much faster. Can we in some way disable tape synthetic generation with GFS?

Markus

Regnor
Veeam ProPartner
Posts: 252
Liked: 53 times
Joined: Jan 31, 2011 11:17 am
Contact:

Re: Another question about tape backup performance

Post by Regnor » Dec 07, 2018 7:10 pm

Synthetic backups can really slow down the performance of your jobs. With what RAID level is your disk array configured?

I can provide you some numbers for LTO8 next week.

Post Reply

Who is online

Users browsing this forum: Rmachado and 6 guests