Another question about tape backup performance

[stewsie]

The performance of tape backup jobs for me can be very very slow. Sometimes as slow as 4 MB/s. I have recently ran a tape job to use the repository as the source and the throughput for the job was 94 MB/s. I am now running another job using the repository and again the job is running at 94 MB/s. What on earth happens when I use a backup or copy job as the source? It seems to be doing a lot of stuff on disk compared to using the repository as the tape job type.

[veremin]

Are those jobs configured identically (besides source, of course)? What about virtual full settings? What bottleneck statistics show? And can you also shed some light on your backup infrastructure - how everything is configured? Thanks!

[stewsie]

Jobs are identical using the same storage but different repositories. There are no virtual full backups configured. The backups are incremental. Backup infra is EMC 5300 and CIFS. We are experiencing issues at the moment with copy performance and I have a support call open but tape performance is always slow apart from when I have now used the repository backup

[veremin]

Hmm, had to guess based on forum correspondence. So, kindly, keep working with our support team on finding the root cause of the degraded performance. Thanks!

[mkretzer]

Sorry but tape functionality is just bad. Case 02772195.

I am quite frustrated about the whole situation.

I just do not get why a backup solution which is 10 years old can do streaming multiple files at the same time (thus using the full performance of the source storage) but not Veeam. It just reads one file, single stream (yes it know about datamoverlocalfastpath - it changes nothing about the basic issue).

How in the world can i keep a LTO 8 tape streaming with someting like this?

We used a HUS110 with 96 disks as source and tape speed was often below 100 MB/s. Now we switched to a G200 midrange storage with 128 disks canhich can do > 1 GB/s when used as source for copy jobs but with single steam tape jobs it just does the 100 MB/s again!

This is just unacceptable. I do not want to purchase SSD storage just to keep my tape streaming.

Again, i am a HUGE veeam fan and i have convinced alot of people to use Veeam but i would never reccomend it if Tape is important...

[Dima P.]

mkretzer,

May I ask if you are using bundled SQL server for your Veeam B&R installation or by any change you have installed at least SQL Server Standard edition for your backup server database? Thank you in advance.

[mkretzer]

Hello Dima,

no we installed SQL 2016 (and now 2017) Enterprise.

Markus

[mkretzer]

@dima p. Why did you ask?

[Dima P.]

mkretzer,

Thanks! In most of the cases performance issues are caused by SQL Express bundled with Veeam B&R installation (due to the it's limitation of 4GB per instance). I've checked both support cases and it looks like you are investigating why source is marked as bottleneck. Please let us know how the investigation goes. Cheers!

[mkretzer]

We did 5 tests today:

- GFS Backup: ~100 MB/s
- Non-GFS Backup: We got ~250 MB/s and bottleneck switching between source and target
- The requested diskspd test with one thread: 436 MB/s
- The requested diskspd test with three threads: 1620 MB/s
- A second diskspd test with three threads from the second part of the SOBR (same backend system, but lower volume load at the time): 2188 MB/s

This test shows exactly why i find single stream backups absolutely unacceptable: Our storage can provide so much IO/MB/s which is just not used by Veeam.

Markus

[Regnor]

What did you test with GFS and Non-GFS backup? Are those tape jobs assigned to different (GFS) media pools or tape hardware?

In most of our environments I can see Veeam achieving 80-100% of the performance of a single drive, so in general it should be faster.

[mkretzer]

All exactly the same. I used a tape one time in a GFS pool and then in a non-GFS pool with the same drive hardware. Before we used LTO-6 and also did not reach the advertised speed.

You really have seen a LTO8 time stream at 300-360 MB/s with GFS and non-Flash source storage?

[Dima P.]

mkretzer,

Thanks for sharing! How your tape library is connected to the repository? Since you are talking about GFS tape jobs mind me asking need to know how the source backup jobs are configured (are they set to perform periodic full backups or not)? Cheers!

[mkretzer]

Dima,

also by FC (8 GBit).
We have periodic fulls 1-2 days before the GFS backup. I have the feeling this helps a bit.

Best would be to have GFS take only these fulls as a normal job would - normal jobs are much faster. Can we in some way disable tape synthetic generation with GFS?

Markus

[Regnor]

Synthetic backups can really slow down the performance of your jobs. With what RAID level is your disk array configured?

I can provide you some numbers for LTO8 next week.

[mkretzer]

RAID 6 14+2 in a hitachi pool which effectively is a RAID 60.
We also had RAID 6 6+2 with another storage system.

Even when there are no other writes (for which RAID 6(0) is bad) and the array could deliver >2 GB/S we only get 120 - 150 MB/s.

[Regnor]

Snythetic Operations perform rather bad on Parity RAID systems. What diskspd parameters did support tell you to run?

Regarding tape performance; i dont have any numbers for LTO8 currently but LTO7. Single drive performance was about 260-280MB/s, both drives in parallel up to 560MB/s.

[mkretzer]

1 thread:
diskspd.exe -c10G -b512K -r4K -Sh -d600 M:\test.dat > C:\temp\diskspd_1t.log

3 threads:
diskspd.exe -c10G -b512K -r4K -Sh -t3 -d600 M:\test.dat > C:\temp\diskspd_3t.log

Ok your tests were with GFS? What is the source backup type? Forward incremental with synthetic fulls?

[Regnor]

No the jobs were standard with forward incremental and active fulls.
I don't have any GFS Jobs for comparison but if I look at virtual fulls, which are synthetic, performance decreases; 150MB/s to 250MB/s for LTO7.

[mkretzer]

... which prooves my point: We have about the same difference. Non-GFS runs at high speed when there is nothing else going on on the storage (nearly 300 MB/s).
Since the storage can provide > 1 GB/s with multiple streams (for example when doing backup copies) it would also solve this for GFS.

[Dima P.]

mkretzer,

Thank you for updating this thread with the investigation results. We will review the details with RnD team.

[Regnor]

Multiple streams would be a great enhancement, also for backup jobs.

For the moment I would recommend you to try the GFS job from a RAID10 array, which should give you a better throughput.

[mkretzer]

Backup jobs are only an issue if you have very few VMs. At least here you can backup multiple VM at once.

RAID 10 is no Option for us. We just had to up our backup storage to 600 TB. With RAID 10 we would loose sooooo much space and also our storage vendor told us multiple times "do not use R10 with Nearline Disks, it is much to unsafe even with the most modern storage error detection algorythms"

[Regnor]

Do you have a whitepaper on that statement? I've never heard of anything like that for RAID10.
We normally go for RAID6 or RAID60 but if performance matters RAID10 is the better choice.