Using tape as a backup target
Post Reply
mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

V10 Tape performance improvements & release notes

Post by mkretzer »

Hello,

according to release notes "We’ve made multiple changes under-the-hood to significantly accelerate Backup to Tape synthetic full backup offload performance when using enterprise-grade backup storage with Windows-based backup repositories."

Does that mean that reading from synthetically created VBKs was not optimized? We do synthetic fulls every sunday for all our jobs and then at 23:30 we start the GFS tape job which just takes these VBKs without having to synthesize something.

I ask this because we did a quick test in the middle of the week and bottleneck was 100 % target (for the first time) and now with sundays run bottleneck is mostly source again - i would have hoped for better performance from V10 (even if there is another bug as described in main forum).

Markus

Gostev
SVP, Product Management
Posts: 26919
Liked: 4373 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V10 Tape performance improvements & release notes

Post by Gostev »

Yep, reading from synthetic VBKs is specifically what was optimized in v10. Thanks!

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer »

So... Would it be more efficient to *not* generate VBKs before the backup? Sounds silly to me...
There was no optimisation in reading plain VBKs from the disk?

To be honest i am very disapointed: My original issue is not fixed and i still think reading with multiple streams from multiple files would be the correct course of action.

Gostev
SVP, Product Management
Posts: 26919
Liked: 4373 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V10 Tape performance improvements & release notes

Post by Gostev »

Optimization of reading plain VBKs from the disk is the next step that developers are currently working on for v11. For v10, we prioritized synthetic VBK performance optimizations, because this was the cause for vast majority of support cases in regards to tape backup performance. Thanks!

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer »

I really hope so... Testing synthetic performance now...

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer » 1 person likes this post

What in the world??
Tape backup that i started in the middle of the week which needed to create synthetic tape backup is running at an average of 390 mb/s OVER NIGHT WHILE ALL OTHER BACKUPS RAN....Currently it went up to 450 MB/s. I *never* saw that tape speed with Veeam. Last weeks backup from ReFS synthetics ran at 178 MB/s. Now Bottleneck is 0 % source, network, proxy, 4% target (before after backup run 94 % source, 3% proxy, 20% network, 48 % target).
In other words with V10 you have to do the exact oposite than with 9.5:
- Don't do synthetics on the same day as the tape backup starts
- Have as many incrementals as possible in the current chain when the backup starts

Gostev
SVP, Product Management
Posts: 26919
Liked: 4373 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V10 Tape performance improvements & release notes

Post by Gostev »

Well, I told you proper tape offload performance it's totally possible to achieve without introducing all the drawbacks of multiplexing different backup files onto the same tape ;)

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer »

Gostev I should not have doubted you :-).

Do I understand correctly that the way you read the data for tape will be used for normal restores as well? Will this take years to implement? I think the benefit could be extremely high!

soncscy
Expert
Posts: 228
Liked: 91 times
Joined: Aug 04, 2019 2:57 pm
Full Name: Harvey Carel
Contact:

Re: V10 Tape performance improvements & release notes

Post by soncscy »

@mkretzer,

These are really impressive numbers...can you give brief run-down of your setup to use as a reference? Tape is continuing to be a really popular subject for our potentials, and would love to have something "solid" to point them at, especially when discussions come up regarding classic backup strats and more modern methods.

"- Have as many incrementals as possible in the current chain when the backup starts"

Can you elaborate on this part too? I'm not sure I follow why more increments would benefit the virtual full process.

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer »

Our Backup Storage is a G200 with 192 disks.

To be honest i do not know how exactly the optimization is done but our first test was with only one full and one incremental and backup speed seemed to be much lower. My guess is that it gets faster the more files the data is coming from (even on the same disk).
Perhaps @gostev can tell us if that is really the way it is done.

Right now our backup speed is mainly limited by the bug i mentioned in tape-f29/first-v10-issue-tape-backup-sw ... 65423.html . Bottleneck is still 0 % source after 40 TB. If this bug is fixed i guess we could get our 60 TB backed up on the weekend instead of between monday and friday (which is very bad for us as we only have a few hours in the week for restores from tape).

Gostev
SVP, Product Management
Posts: 26919
Liked: 4373 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V10 Tape performance improvements & release notes

Post by Gostev »

mkretzer wrote: Mar 14, 2020 4:44 pmDo I understand correctly that the way you read the data for tape will be used for normal restores as well? Will this take years to implement? I think the benefit could be extremely high!
Yes, that's the plan. Implementation is not that hard in this case, primary concern is reliability and the amount of testing required when introducing these kind of major changes to the holy grail of the engine. It's about changing extremely polished code that was not touched for years...

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer »

But from what i see here the benefits could be HUGE! So V11? When will that come?

Can you explain a bit how the optimisation in tape logic works? I mean we can do try & error and try with less or more incrementals but it would help to know how the optimisation works!

Gostev
SVP, Product Management
Posts: 26919
Liked: 4373 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V10 Tape performance improvements & release notes

Post by Gostev »

V11 will come When It's Ready™ :D

Unfortunately I cannot share a lot of details, as there are multiple optimizations and a lot of them are know-hows we prefer not to share publicly. However, based on what I know, the number of incremental files should not have any impact.

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer »

Ok - thats strange. I will test that as soon as the other bug (pausing between objects) is fixed.

DerOest
Enthusiast
Posts: 60
Liked: 27 times
Joined: Oct 30, 2015 10:10 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by DerOest » 5 people like this post

Just wanted to chime in and say:

v10 has blown my mind!
We run with 2 LTO7 tape drives, before we only got ~80MB/s per stream from our ReFS repository (it was very fast in the beginning, but performance degraded more and more the longer the backup retention, Server2016).
With v10 Veeam maxes out both LTO7 drives again, with ~290 MB/s per stream!

I have no idea how Veeam manages to read ReFS that fast again, as even manually copying files is extremly slow.
It's wonderful to have tape-out time reduced by >3x again!

mdiver
Service Provider
Posts: 88
Liked: 15 times
Joined: Nov 04, 2009 2:08 pm
Location: Heidelberg, Germany
Contact:

Re: V10 Tape performance improvements & release notes

Post by mdiver » 3 people like this post

I also want to cite the release notes: "We've made multiple changes under-the-hood to significantly accelerate Backup to Tape synthetic full backup offload performance when using enterprise-grade backup storage with Windows-based backup repositories."

Nothing seems to be like it was before under this hood!

With V10 we get 890MB/s (!) with 3 LTO-7 drives in parallel backing up from a ReFS SOBR (data-locality) with 7 extents across 3 hosts. Awesome!
9.5U4 gave us a maximum (!) of 250MB/s in exactly the same installation using the same data. This is in exess of 3x here as well.

RGijsen
Expert
Posts: 100
Liked: 15 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: V10 Tape performance improvements & release notes

Post by RGijsen »

What exactly is 'enterprise storage' in this regard? With regards to Veeam, does it distinguish between let's say an entry level SAN like a MSA2052 with SSDs compared to a fully fledged all flash Netapp or Nimble or the like? How should I interpret this? Does Veeam just use more streams towards ReFS when latency is low enough for example?
I'm trying to wrap my head around, as an expected limitation, or 'feature' of ReFS with blockcloning is that your sequential data gets scattered more and more all over your storage during time. Even with SSD's random access is considerably slower than sequential. Of course, with a 100 'spindle' all flash NVME based beast on 128Gb fiber, you might still fill the wires at random, but that's probably not what Veeam is aiming at.

So I'd like to see a bit elaboration here on when this ReFS improvement can roughly be expected.

Stephan23
Enthusiast
Posts: 38
Liked: 2 times
Joined: Jun 03, 2015 8:32 am
Full Name: Stephan
Contact:

Re: V10 Tape performance improvements & release notes

Post by Stephan23 »

Is the tape performance increase limited to ReFS repositories?

Dima P.
Product Manager
Posts: 11872
Liked: 1063 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: V10 Tape performance improvements & release notes

Post by Dima P. »

Does Veeam just use more streams towards ReFS when latency is low enough for example?
When building the synthetic full for tape backup we've started to request data via a special queue via multiple threads. The way how this queue is handled fully depends on the storage operating system and hardware configuration. For example any RAID system can handle such queues pretty well.
Is the tape performance increase limited to ReFS repositories?
No, it's not limited to ReFS.

perjonsson1960
Expert
Posts: 187
Liked: 15 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

[MERGED] Performance of backup to tape

Post by perjonsson1960 »

Folks,

It seems that the tape backup performance has increased significantly between B&R v9.5 and v10. Last time I did a backup to tape, using v9.5 U4b, it took 38h 11m to write 30.9 TB to four LTO-7 tapes, and now it took 23h 28m to write 30 TB, also to four LTO-7 tapes. The hardware is the same. This is a significant improvement! :D

PJ

Egor Yakovlev
Veeam Software
Posts: 1621
Liked: 374 times
Joined: Jun 14, 2013 9:30 am
Full Name: Egor Yakovlev
Contact:

Re: Performance of backup to tape

Post by Egor Yakovlev » 1 person likes this post

Thanks for feedback, Per!

Yes, v10 has performance improvements for Backup to Tape (major part is related to the synthesized backup full creation) and NDMP backup to Tape (up to x3 times faster)!

/Cheers!

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer »

If only the switch between objects would be faster we really could get even better performance - at times the job stalls completely for 30 seconds or so

Dima P.
Product Manager
Posts: 11872
Liked: 1063 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: V10 Tape performance improvements & release notes

Post by Dima P. »

mkretzer,
switch between objects
To clarify, you are referring to tasks in the job or actual backup files?

Gostev
SVP, Product Management
Posts: 26919
Liked: 4373 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: V10 Tape performance improvements & release notes

Post by Gostev » 1 person likes this post

RGijsen wrote: Apr 06, 2020 6:52 amWhat exactly is 'enterprise storage' in this regard?
Basically, not Synologies and QNAPs of the world... but rather more serious stuff with many spindles + proper enterprise-grade RAID controller.

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer »

@Dima P. actual per-VM files. We back up > 60 TB in three days now and some of these VMs are huge and everything goes extremly fast and then there are alot of VMs < 30 GB. In total the job backs up ~ 1200 VMs and when the small VMs are backed up the processing rate drops by a good 100 MB/s on average as the switch between objects causes the backup to stall for a short time.

Dima P.
Product Manager
Posts: 11872
Liked: 1063 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: V10 Tape performance improvements & release notes

Post by Dima P. »

mkretzer,

Thanks for the details, will discuss your case with RnD team. Cheers!

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer »

Case is 04050943. The current theory is that it is the amount of history of encrypted tape backups. Disabling encryption did not help but that was expected according to support. So currently we have no workaround.

Here you see how it behaves:
https://imgur.com/a/tQHr1Ry

In the screenshot you can see that both tape drives currently has no running tasks as it takes time to switch between objects

veremin
Product Manager
Posts: 17911
Liked: 1675 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: V10 Tape performance improvements & release notes

Post by veremin »

We've identified the issue and created a hotfix for it - after few days of tests we will share it with you (within existing support case). Thanks!

mkretzer
Expert
Posts: 678
Liked: 159 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: V10 Tape performance improvements & release notes

Post by mkretzer »

Very nice! I am eager seeing the speed with the patch!

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Ramalingam N and 24 guests