Discussions related to exporting backups to tape and backing up directly to tape.
Post Reply
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Another question about tape backup performance

Post by stewsie »

The performance of tape backup jobs for me can be very very slow. Sometimes as slow as 4 MB/s. I have recently ran a tape job to use the repository as the source and the throughput for the job was 94 MB/s. I am now running another job using the repository and again the job is running at 94 MB/s. What on earth happens when I use a backup or copy job as the source? It seems to be doing a lot of stuff on disk compared to using the repository as the tape job type.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Another question about tape backup performance

Post by veremin »

Are those jobs configured identically (besides source, of course)? What about virtual full settings? What bottleneck statistics show? And can you also shed some light on your backup infrastructure - how everything is configured? Thanks!
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Another question about tape backup performance

Post by stewsie »

Jobs are identical using the same storage but different repositories. There are no virtual full backups configured. The backups are incremental. Backup infra is EMC 5300 and CIFS. We are experiencing issues at the moment with copy performance and I have a support call open but tape performance is always slow apart from when I have now used the repository backup
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Another question about tape backup performance

Post by veremin »

Hmm, had to guess based on forum correspondence. So, kindly, keep working with our support team on finding the root cause of the degraded performance. Thanks!
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer » 1 person likes this post

Sorry but tape functionality is just bad. Case 02772195.

I am quite frustrated about the whole situation.

I just do not get why a backup solution which is 10 years old can do streaming multiple files at the same time (thus using the full performance of the source storage) but not Veeam. It just reads one file, single stream (yes it know about datamoverlocalfastpath - it changes nothing about the basic issue).

How in the world can i keep a LTO 8 tape streaming with someting like this?

We used a HUS110 with 96 disks as source and tape speed was often below 100 MB/s. Now we switched to a G200 midrange storage with 128 disks canhich can do > 1 GB/s when used as source for copy jobs but with single steam tape jobs it just does the 100 MB/s again!

This is just unacceptable. I do not want to purchase SSD storage just to keep my tape streaming.

Again, i am a HUGE veeam fan and i have convinced alot of people to use Veeam but i would never reccomend it if Tape is important...
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Another question about tape backup performance

Post by Dima P. »

mkretzer,

May I ask if you are using bundled SQL server for your Veeam B&R installation or by any change you have installed at least SQL Server Standard edition for your backup server database? Thank you in advance.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer »

Hello Dima,

no we installed SQL 2016 (and now 2017) Enterprise.

Markus
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer »

@dima p. Why did you ask?
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Another question about tape backup performance

Post by Dima P. »

mkretzer,

Thanks! In most of the cases performance issues are caused by SQL Express bundled with Veeam B&R installation (due to the it's limitation of 4GB per instance). I've checked both support cases and it looks like you are investigating why source is marked as bottleneck. Please let us know how the investigation goes. Cheers!
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer »

We did 5 tests today:

- GFS Backup: ~100 MB/s
- Non-GFS Backup: We got ~250 MB/s and bottleneck switching between source and target
- The requested diskspd test with one thread: 436 MB/s
- The requested diskspd test with three threads: 1620 MB/s
- A second diskspd test with three threads from the second part of the SOBR (same backend system, but lower volume load at the time): 2188 MB/s

This test shows exactly why i find single stream backups absolutely unacceptable: Our storage can provide so much IO/MB/s which is just not used by Veeam.

Markus
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Another question about tape backup performance

Post by Regnor »

What did you test with GFS and Non-GFS backup? Are those tape jobs assigned to different (GFS) media pools or tape hardware?

In most of our environments I can see Veeam achieving 80-100% of the performance of a single drive, so in general it should be faster.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer »

All exactly the same. I used a tape one time in a GFS pool and then in a non-GFS pool with the same drive hardware. Before we used LTO-6 and also did not reach the advertised speed.

You really have seen a LTO8 time stream at 300-360 MB/s with GFS and non-Flash source storage?
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Another question about tape backup performance

Post by Dima P. »

mkretzer,

Thanks for sharing! How your tape library is connected to the repository? Since you are talking about GFS tape jobs mind me asking need to know how the source backup jobs are configured (are they set to perform periodic full backups or not)? Cheers!
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer »

Dima,

also by FC (8 GBit).
We have periodic fulls 1-2 days before the GFS backup. I have the feeling this helps a bit.

Best would be to have GFS take only these fulls as a normal job would - normal jobs are much faster. Can we in some way disable tape synthetic generation with GFS?

Markus
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Another question about tape backup performance

Post by Regnor »

Synthetic backups can really slow down the performance of your jobs. With what RAID level is your disk array configured?

I can provide you some numbers for LTO8 next week.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer »

RAID 6 14+2 in a hitachi pool which effectively is a RAID 60.
We also had RAID 6 6+2 with another storage system.

Even when there are no other writes (for which RAID 6(0) is bad) and the array could deliver >2 GB/S we only get 120 - 150 MB/s.
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Another question about tape backup performance

Post by Regnor »

Snythetic Operations perform rather bad on Parity RAID systems. What diskspd parameters did support tell you to run?

Regarding tape performance; i dont have any numbers for LTO8 currently but LTO7. Single drive performance was about 260-280MB/s, both drives in parallel up to 560MB/s.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer »

1 thread:
diskspd.exe -c10G -b512K -r4K -Sh -d600 M:\test.dat > C:\temp\diskspd_1t.log

3 threads:
diskspd.exe -c10G -b512K -r4K -Sh -t3 -d600 M:\test.dat > C:\temp\diskspd_3t.log

Ok your tests were with GFS? What is the source backup type? Forward incremental with synthetic fulls?
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Another question about tape backup performance

Post by Regnor »

No the jobs were standard with forward incremental and active fulls.
I don't have any GFS Jobs for comparison but if I look at virtual fulls, which are synthetic, performance decreases; 150MB/s to 250MB/s for LTO7.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer »

... which prooves my point: We have about the same difference. Non-GFS runs at high speed when there is nothing else going on on the storage (nearly 300 MB/s).
Since the storage can provide > 1 GB/s with multiple streams (for example when doing backup copies) it would also solve this for GFS.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Another question about tape backup performance

Post by Dima P. »

mkretzer,

Thank you for updating this thread with the investigation results. We will review the details with RnD team.
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Another question about tape backup performance

Post by Regnor »

Multiple streams would be a great enhancement, also for backup jobs.

For the moment I would recommend you to try the GFS job from a RAID10 array, which should give you a better throughput.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Another question about tape backup performance

Post by mkretzer »

Backup jobs are only an issue if you have very few VMs. At least here you can backup multiple VM at once.

RAID 10 is no Option for us. We just had to up our backup storage to 600 TB. With RAID 10 we would loose sooooo much space and also our storage vendor told us multiple times "do not use R10 with Nearline Disks, it is much to unsafe even with the most modern storage error detection algorythms"
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Another question about tape backup performance

Post by Regnor »

Do you have a whitepaper on that statement? I've never heard of anything like that for RAID10.
We normally go for RAID6 or RAID60 but if performance matters RAID10 is the better choice.
Post Reply

Who is online

Users browsing this forum: No registered users and 27 guests