V10 Tape performance improvements & release notes

[mkretzer]

Hello,

according to release notes "We’ve made multiple changes under-the-hood to significantly accelerate Backup to Tape synthetic full backup offload performance when using enterprise-grade backup storage with Windows-based backup repositories."

Does that mean that reading from synthetically created VBKs was not optimized? We do synthetic fulls every sunday for all our jobs and then at 23:30 we start the GFS tape job which just takes these VBKs without having to synthesize something.

I ask this because we did a quick test in the middle of the week and bottleneck was 100 % target (for the first time) and now with sundays run bottleneck is mostly source again - i would have hoped for better performance from V10 (even if there is another bug as described in main forum).

Markus

[Gostev]

Yep, reading from synthetic VBKs is specifically what was optimized in v10. Thanks!

[mkretzer]

So... Would it be more efficient to *not* generate VBKs before the backup? Sounds silly to me...
There was no optimisation in reading plain VBKs from the disk?

To be honest i am very disapointed: My original issue is not fixed and i still think reading with multiple streams from multiple files would be the correct course of action.

[Gostev]

Optimization of reading plain VBKs from the disk is the next step that developers are currently working on for v11. For v10, we prioritized synthetic VBK performance optimizations, because this was the cause for vast majority of support cases in regards to tape backup performance. Thanks!

[mkretzer]

I really hope so... Testing synthetic performance now...

[mkretzer]

What in the world??
Tape backup that i started in the middle of the week which needed to create synthetic tape backup is running at an average of 390 mb/s OVER NIGHT WHILE ALL OTHER BACKUPS RAN....Currently it went up to 450 MB/s. I *never* saw that tape speed with Veeam. Last weeks backup from ReFS synthetics ran at 178 MB/s. Now Bottleneck is 0 % source, network, proxy, 4% target (before after backup run 94 % source, 3% proxy, 20% network, 48 % target).
In other words with V10 you have to do the exact oposite than with 9.5:
- Don't do synthetics on the same day as the tape backup starts
- Have as many incrementals as possible in the current chain when the backup starts

[Gostev]

Well, I told you proper tape offload performance it's totally possible to achieve without introducing all the drawbacks of multiplexing different backup files onto the same tape

[mkretzer]

Gostev I should not have doubted you .

Do I understand correctly that the way you read the data for tape will be used for normal restores as well? Will this take years to implement? I think the benefit could be extremely high!

[soncscy]

@mkretzer,

These are really impressive numbers...can you give brief run-down of your setup to use as a reference? Tape is continuing to be a really popular subject for our potentials, and would love to have something "solid" to point them at, especially when discussions come up regarding classic backup strats and more modern methods.

"- Have as many incrementals as possible in the current chain when the backup starts"

Can you elaborate on this part too? I'm not sure I follow why more increments would benefit the virtual full process.

[mkretzer]

Our Backup Storage is a G200 with 192 disks.

To be honest i do not know how exactly the optimization is done but our first test was with only one full and one incremental and backup speed seemed to be much lower. My guess is that it gets faster the more files the data is coming from (even on the same disk).
Perhaps @gostev can tell us if that is really the way it is done.

Right now our backup speed is mainly limited by the bug i mentioned in tape-f29/first-v10-issue-tape-backup-sw ... 65423.html . Bottleneck is still 0 % source after 40 TB. If this bug is fixed i guess we could get our 60 TB backed up on the weekend instead of between monday and friday (which is very bad for us as we only have a few hours in the week for restores from tape).

[Gostev]

Do I understand correctly that the way you read the data for tape will be used for normal restores as well? Will this take years to implement? I think the benefit could be extremely high!

Yes, that's the plan. Implementation is not that hard in this case, primary concern is reliability and the amount of testing required when introducing these kind of major changes to the holy grail of the engine. It's about changing extremely polished code that was not touched for years...

[mkretzer]

But from what i see here the benefits could be HUGE! So V11? When will that come?

Can you explain a bit how the optimisation in tape logic works? I mean we can do try & error and try with less or more incrementals but it would help to know how the optimisation works!

[Gostev]

V11 will come When It's Ready™

Unfortunately I cannot share a lot of details, as there are multiple optimizations and a lot of them are know-hows we prefer not to share publicly. However, based on what I know, the number of incremental files should not have any impact.

[mkretzer]

Ok - thats strange. I will test that as soon as the other bug (pausing between objects) is fixed.

[DerOest]

Just wanted to chime in and say:

v10 has blown my mind!
We run with 2 LTO7 tape drives, before we only got ~80MB/s per stream from our ReFS repository (it was very fast in the beginning, but performance degraded more and more the longer the backup retention, Server2016).
With v10 Veeam maxes out both LTO7 drives again, with ~290 MB/s per stream!

I have no idea how Veeam manages to read ReFS that fast again, as even manually copying files is extremly slow.
It's wonderful to have tape-out time reduced by >3x again!

[mdiver]

I also want to cite the release notes: "We've made multiple changes under-the-hood to significantly accelerate Backup to Tape synthetic full backup offload performance when using enterprise-grade backup storage with Windows-based backup repositories."

Nothing seems to be like it was before under this hood!

With V10 we get 890MB/s (!) with 3 LTO-7 drives in parallel backing up from a ReFS SOBR (data-locality) with 7 extents across 3 hosts. Awesome!
9.5U4 gave us a maximum (!) of 250MB/s in exactly the same installation using the same data. This is in exess of 3x here as well.

[RGijsen]

What exactly is 'enterprise storage' in this regard? With regards to Veeam, does it distinguish between let's say an entry level SAN like a MSA2052 with SSDs compared to a fully fledged all flash Netapp or Nimble or the like? How should I interpret this? Does Veeam just use more streams towards ReFS when latency is low enough for example?
I'm trying to wrap my head around, as an expected limitation, or 'feature' of ReFS with blockcloning is that your sequential data gets scattered more and more all over your storage during time. Even with SSD's random access is considerably slower than sequential. Of course, with a 100 'spindle' all flash NVME based beast on 128Gb fiber, you might still fill the wires at random, but that's probably not what Veeam is aiming at.

So I'd like to see a bit elaboration here on when this ReFS improvement can roughly be expected.

[Stephan23]

Is the tape performance increase limited to ReFS repositories?

[Dima P.]

Does Veeam just use more streams towards ReFS when latency is low enough for example?

When building the synthetic full for tape backup we've started to request data via a special queue via multiple threads. The way how this queue is handled fully depends on the storage operating system and hardware configuration. For example any RAID system can handle such queues pretty well.

Is the tape performance increase limited to ReFS repositories?

No, it's not limited to ReFS.

[perjonsson1960]

Folks,

It seems that the tape backup performance has increased significantly between B&R v9.5 and v10. Last time I did a backup to tape, using v9.5 U4b, it took 38h 11m to write 30.9 TB to four LTO-7 tapes, and now it took 23h 28m to write 30 TB, also to four LTO-7 tapes. The hardware is the same. This is a significant improvement!

PJ

[Egor Yakovlev]

Thanks for feedback, Per!

Yes, v10 has performance improvements for Backup to Tape (major part is related to the synthesized backup full creation) and NDMP backup to Tape (up to x3 times faster)!

/Cheers!

[mkretzer]

If only the switch between objects would be faster we really could get even better performance - at times the job stalls completely for 30 seconds or so

[Dima P.]

mkretzer,

switch between objects

To clarify, you are referring to tasks in the job or actual backup files?

[Gostev]

What exactly is 'enterprise storage' in this regard?

Basically, not Synologies and QNAPs of the world... but rather more serious stuff with many spindles + proper enterprise-grade RAID controller.

[mkretzer]

@Dima P. actual per-VM files. We back up > 60 TB in three days now and some of these VMs are huge and everything goes extremly fast and then there are alot of VMs < 30 GB. In total the job backs up ~ 1200 VMs and when the small VMs are backed up the processing rate drops by a good 100 MB/s on average as the switch between objects causes the backup to stall for a short time.

[Dima P.]

mkretzer,

Thanks for the details, will discuss your case with RnD team. Cheers!

[mkretzer]

Case is 04050943. The current theory is that it is the amount of history of encrypted tape backups. Disabling encryption did not help but that was expected according to support. So currently we have no workaround.

Here you see how it behaves:
https://imgur.com/a/tQHr1Ry

In the screenshot you can see that both tape drives currently has no running tasks as it takes time to switch between objects

[veremin]

We've identified the issue and created a hotfix for it - after few days of tests we will share it with you (within existing support case). Thanks!

[mkretzer]

Very nice! I am eager seeing the speed with the patch!