Veeam BR8, best practice for B2D2T full backups nightly?

[cerberus]

Hello,

We currently have 2 nightly jobs that run and backup our VMware infrastructure (vSphere/ESXi 6, 3 hosts, 30 VMs) over FC to a dedicated backup server which is then offloaded to a Dell ML6020 LTO7 (2x drives) fc library.We leverage VMware's CBT to only transfer changed blocks over FC network (8gb fc).

Backup server has 8x4TB 7.2k NLSAS (RAID6) for local storage and 6x480GB SSD (RAID5) used for write cache (about 90% of writes during full backup to disk are cached which helps with the RAID6 write penalty on our slow large disks).

We have a business requirement for offloading full backups to tape daily.

Both B2D backup jobs are setup to do Reverse Incremental so that I can run a B2T job after it finishes which gets me a full nightly backup offloaded to tape. Performance wise, is this the same as doing Incremental with Synthetic Full daily? Or is it better to do Incremental with Synthetic Full daily which would gain me the same end result and that is the last backup being a full backup which is then offloaded to tape? Would the backup to disk time decrease?

My issue is with performance when doing backup to tape, both jobs combined is just under 10TB of data and the source is the bottleneck when writing to LTO7 which is capable of doing 300MB/s each (we have 2 drives and 2 jobs in parallel, this should get close to 600MB/s). I am finding that I am getting about 300MB/s combined writing to both tapes. Anything I can do settings wise to increase this? I read in BR9 there is block settings we can change on the tape drive; has there been any testing done in terms of performance with small/large block size on tape? I am thinking larger block size would reduce the amount of read IOPS needed on the storage for the same amount of data.. which might have a potential to help tape throughput.. In BR8, what is the default block size that is used and what's available in BR9? I am thinking if I can increase the block size it could increase the throughput to tape...

Tape Library is a Dell ML6020 with LTO7 drives (IBM ULTRIUM-TD7). We have an active support with Veeam so upgrading to BR9 is also on the table.

[PTide]

Hi,

Performance wise, is this the same as doing Incremental with Synthetic Full daily? Or is it better to do Incremental with Synthetic Full daily which would gain me the same end result and that is the last backup being a full backup which is then offloaded to tape?

Forward mode is 3x less stresfull for your target storage. On the other hand daily synthetic will consume lots of space with the same end result so it'd be better to use virtual full backup. in case you decide to switch to forever forward.

Would the backup to disk time decrease?

Yes, forward incremental method is faster than reverse incremental.

both jobs combined is just under 10TB of data and the source is the bottleneck when writing to LTO7 which is capable of doing 300MB/s each (we have 2 drives and 2 jobs in parallel, this should get close to 600MB/s). I am finding that I am getting about 300MB/s combined writing to both tapes.

Have you tried running your jobs one by one instead of suimultaneous run? Any performance difference?

Thank you.

[cerberus]

Hey PTide,

Yes I did run a single backup job to tape and the throughput goes to about 250MB/s and the bottleneck changes to target (tape). When both jobs are writing to both LTO7 tape drives, the bottleneck is source and throughput is about 110MB/s per job. We want to utilize both tape drives to reduce the backup to tape window. This is why I was curious about the block size setting available in BR9 and if any testing with different block size setting was done/published by Veeam?

Also we have a business requirement for offloading a full backup to tape daily. It seems virtual full backups do not support this (looking at the screen on https://www.veeam.com/native-tape-support.html). It has day of week or monthly.

It seems the only option for "full backups to tape every day" is what I have right now, which is Reverse Incremental jobs. Interesting though, looking at that same page it seems it is now possible to start writing to tape right after a VM is backed up instead of waiting for the entire job to finish.. can you confirm this is correct?

Looks like I need to plan for BR9 immediately if that's the case

[tsightler]

I believe you can do a virtual full backup to tape every day if you select "Weekly on selected days" and then just select every day.

[rreed]

+1 on the block size. Looks like in v8 it was running at 64K but in v9 it's much larger (default settings, I don't recall that I ever went in an monkeyed w/ block size in our old v8 setup), but I need to confirm that here in a bit as I go through my tapes. We're running LTO-6.

Cerberus, are you backing up across the network to tape, or backing up from your tape server's local HDD to tape? We're having trouble w/ the network to tape = slow; local HDD to tape = fast in our environment. (4)x1Gbps connected tape server that's only seeing around 30-50MB/s but if we pull locally from the tape server's locally HDD it'll run 120-160MB/s for LTO-6's 120MB/s capability. We're about to swap out our (4)x1Gbps out for (2)x10Gbps so I haven't really pursued the 30-50MB/s speed issue just yet.

[cerberus]

We are backing up from local storage/server to tape via 8gb fibre channel. The tape library has 2x LTO7 drives which should be capable of doing 300MB/s but I am seeing 100-120MB/s per drive.

The source is being identified as the bottle neck. The local disks on the server are 8x4TB 7.2k NLSAS (RAID6) and there is just under 10TB of data being offloaded to tape. I spent couple of hours last night playing with this with no luck. When I run HDTune on the B2D volume I am getting 743MB/s read average.

I did confirm that in the device manager the LTO7 tapes have a default block size of 64KB
My B2D NTFS volume allocation size is 8192 bytes (8kb).
The stripe size on my PERC H730P RAID6 is 64kb.

What I was thinking to do next (over the weekend) is to blow away the B2D volume and re-create it with 64kb allocation unit size to see if that helps. I also have a call open to Dell to see if we can run some test from the server to the tape using something like ITDT to see what results we get.

My thought is, if HDTune is showing 743MB/s read avg, why is Veeam only able to get 100-120MB/s per drive in parallel and showing source as the bottleneck. I will probably also have to upgrade to BR9 to engage support since LTO7 is not supported in BR8. Waiting to see update 1 released to move to it which should be soon then hopefully the larger block size in BR9 will need less IOPS and source bottleneck wont' be destroying my tape throughput??

[PTide]

My thought is, if HDTune is showing 743MB/s read avg

Have you tried to copy those two .vbk files somewhere, what was the performance? I'm asking because I suspect that parallel reading of two large files (lots of random reads) might be the reason why source is the bottleneck.

[cerberus]

I was able to observe what happens when one of the 2 backup to tape jobs finishes, we notice that if we have a single B2T job running the speed is about 250MB/s.

Some screenshots below..
http://imgur.com/a/on9wQ

Do you think if I changed my B2D job from Reverse Incremental to Incremental and roll with "Forever forward and synthesizing full to tape" it should reduce the overall back time? Creating the full backup VBK during "Reverse Incremental" backup vs. synthesizing it on tape; which is better in terms of I/O? Our writes are good (since we cache them in SSD), if I can find a solution to the reads to squeeze a bit more throughput on tape would be great.

We are now planning to upgrade to BR9 this weekend and also change the B2D RAID stripe from 64kb to 256 or 512 and also the NTFS block size to 64kb then play with tape block settings (under device manager the LTO7 drives reports max block size to be 8mb and default is 64kb).