v9 - Tape performance reports

Everything about backing up to tape

Re: v9 - Tape performance reports

Veeam Logoby mongie » Wed Feb 17, 2016 8:24 pm

I've logged a ticket now... #01700980
mongie
Expert
 
Posts: 149
Liked: 23 times
Joined: Mon May 16, 2011 4:00 am
Location: Brisbane, Australia
Full Name: Alex Macaronis

[MERGED] : Tape LTO4 Expected speed?

Veeam Logoby larkrist » Thu Feb 18, 2016 8:29 pm

I wonder if there is something we can do to speed up the Backup to Tape Job.
We are having a StoreOnce with Catalyst-FC for first backup store. Then we want to copy this to an LTO4 TL also running on FC. Proxy has access to both.
We are getting a speed of 50MB/s
larkrist
Novice
 
Posts: 6
Liked: never
Joined: Thu Feb 18, 2016 8:22 pm
Full Name: Lars Kristoffersen

Re: Tape LTO4 Expected speed?

Veeam Logoby skrause » Thu Feb 18, 2016 9:01 pm

What does the Veeam console say is the bottleneck?

Also, are you using encryption? Software encryption puts a pretty big penalty on tape performance.
Steve Krause
Veeam Certified Architect
skrause
Expert
 
Posts: 296
Liked: 45 times
Joined: Mon Dec 08, 2014 2:58 pm
Full Name: Steve Krause

Re: v9 - Tape performance reports

Veeam Logoby v.Eremin » Fri Feb 19, 2016 9:36 am

Hi, Lars,

We try to keep all posts about tape jobs' performance within one thread for the purpose of convenience.

Please, follow the recommendations given by Dmitry.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: v9 - Tape performance reports

Veeam Logoby redhorse » Tue Mar 01, 2016 8:30 am

I do have the same situation:

1. Dell TL2000 with LTO5-Tapes, connected to physical Veeam-Server with all roles
2. File-to-Tape
3. Bottleneck-Detecting with no result
4. About one million files with average file size of 250 KByte

The processing rate is about 16 MB/s, is that normal?

I have to restore all the old file backups from backup exec to disk and then backup to tape with Veeam. Because it takes so much time my idea is to backup the volume with the restored files with Endpoint Protection and then to tape, does that make sense? ;-)
redhorse
Enthusiast
 
Posts: 84
Liked: 4 times
Joined: Tue Feb 19, 2013 8:08 am
Full Name: RH

Re: v9 - Tape performance reports

Veeam Logoby v.Eremin » Tue Mar 01, 2016 10:03 am

And you are already on version 9, right? If so, feel free to open a ticket with our support team and post its number here so that we can follow the case and check whether the performance you see is normal indeed. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: v9 - Tape performance reports

Veeam Logoby redhorse » Tue Mar 01, 2016 10:20 am

Yes V9, Case #01713666

Now, I started a file level backup with Veeam EP, this is faster and the file copy with one large file to tape should be much faster.
redhorse
Enthusiast
 
Posts: 84
Liked: 4 times
Joined: Tue Feb 19, 2013 8:08 am
Full Name: RH

Re: v9 - Tape performance reports

Veeam Logoby rreed » Wed Mar 02, 2016 5:27 pm

#01715660 for us. It's always been the same in v8 and in v9. Pulling across the network yields ~30-70 MB/s to our LTO-6. Pulling from the tape server's local HDD ran its full 120MB/s in v8, and v9 somehow reports 240MB/s from local HDD(?). File or Backup, doesn't matter. (4) x 1Gbps LAG, and I get that it's likely only running one data stream so only one NIC, but 30-70 MB/s doesn't saturate a 1Gbps link. Target is always the bottleneck w/ all jobs (File and Backup); Source 0%, Proxy 5-10%, Network 0%, Target 15% thereabouts across all network tape jobs.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: v9 - Tape performance reports

Veeam Logoby rreed » Wed Mar 02, 2016 7:46 pm

Question while we're on the topic of v9 tape performance, how is the processing rate calculated in the Statistics dialog box? Reason I ask is because we're running LTO-6, and in v8 when we'd run jobs off the tape server's HDD it would report right around 120 MB/s which is LTO-6's advertised speed. In v8 doing the same thing reports up through 250+ MB/s running one single tape drive! Do we have a bug there or ?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: v9 - Tape performance reports

Veeam Logoby Dima P. » Sun Mar 06, 2016 7:34 pm

Hi rreed,

This might be a bug indeed. I’ve seen several reports where tape average performance was not correctly calculated, so it’s better to report this issue to support team and let them investigate the logs.
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

[MERGED] Veeam v9 B2T to dual LTO7 drives slow?

Veeam Logoby cerberus » Wed Mar 09, 2016 2:48 am

Hello,

We have a Dell ML6020 LTO7 tape library (dual LTO7 drives) connected using 8gb FC to our dedicated backup server with local storage (8x4TB 7.2k RPM NLSAS in RAID10) with a 1.75TB SSD write I/O caching layer in our new Dell R730xd.

The RAID10 is formatted with 512kb stripe size and the NTFS volume is formatted with 64kb cluster size. Our tape drives are set to use native SCSI using largest block size for tape (524288).

We have 2 B2D jobs setup so that we can write to both tape drives at once (I know that in v9 there is the possibility to use single job to write to both tapes but I found the fill rate on tape was not acceptable so I am using two jobs). The local repository is setup for per-VM backup files.

The B2D jobs finish fast and in acceptable time (backing up VMware over FC using CBT). The B2T job is not so great, single LTO7 should go 300MB/s (600MB/s combined) but I am finding that I am barely getting the throughput to reach those numbers. Observing the B2T job I notice a very interesting DROP in performance when both jobs are running. If a single job is running it gets upwards of 270MB/s+.

This is a reverse incremental job, so last file is always the full VBK and since this is all fairly new (only 5 backup restore points) there shouldn't be much of fragmentation causing random reads. I would assume the B2T would be a very linear read opearation in this case.

I found kb2014 and ran diskspd tests, below they are..
- Reverse Incremental (notice the read here, through the roof) http://pastebin.com/1dRHPv7i
- Random Read http://pastebin.com/XHkfXfgM
- Linear Read http://pastebin.com/iKu8jfbH
- HDTune yields about 700MB/s read

Graph of first B2T job... http://imgur.com/VJyUMnc
Graph of 2nd B2T job... http://imgur.com/touOW7P

Both jobs show Target as bottleneck (37%) and also Proxy is up there right around 35%. Veeam v9 has all roles installed on a single dedicated backup server with local storage and tape library zoned through our brocade fc switch.

I spoke to Dell as well and they ran IBM's ITDT tool on both drives at same time and they were both getting 280MB/s in parallel.

No idea what to do about the complete performance drop in one of the B2T jobs when both are running in parallel. Any ideas what the issue is or what I could do to yield more out of both of our new LTO7 drives?

Case #01719643
cerberus
Enthusiast
 
Posts: 33
Liked: 3 times
Joined: Fri Aug 28, 2015 2:45 pm
Full Name: MD

[MERGED] Re: Veeam v9 B2T to dual LTO7 drives slow?

Veeam Logoby alanbolte » Wed Mar 09, 2016 3:35 am

What happens if you run two instances of the diskspd linear read test concurrently? Or if you run diskspd reading from one file while the tape job copies a different file to tape?
alanbolte
Expert
 
Posts: 635
Liked: 170 times
Joined: Mon Jun 18, 2012 8:58 pm
Full Name: Alan Bolte

[MERGED] Re: Veeam v9 B2T to dual LTO7 drives slow?

Veeam Logoby cerberus » Wed Mar 09, 2016 7:02 pm

I ran two instances of diskspd.exe while matching the -b (blocksize) to be the same as the setting on the backup to disk which is set to "Local target" and compression is "Optimal" This means 1024K block size on disk for Veeam files.

Code: Select all
Command Line: diskspd.exe -b1024K -h -d300 D:\Backups\Nightly Backup job for Servers\xxx-xxx.vm-1952016-03-08T193037.vbk

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |    133787811840 |       127590 |     425.29 |     425.29 | D:\Backups\Nightly Backup job for Servers\xxx-xxx.vm-1952016-03-08T193037.vbk (38GB)
------------------------------------------------------------------------------
total:      133787811840 |       127590 |     425.29 |     425.29

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |    133787811840 |       127590 |     425.29 |     425.29 | D:\Backups\Nightly Backup job for Servers\xxx-xxx.vm-1952016-03-08T193037.vbk (38GB)
------------------------------------------------------------------------------
total:      133787811840 |       127590 |     425.29 |     425.29

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 | D:\Backups\Nightly Backup job for Servers\xxx-xxx.vm-1952016-03-08T193037.vbk (38GB)
------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00

Code: Select all
Command Line: diskspd.exe -b1024K -h -d300 D:\Backups\Nightly Backup job for Applications\xxx-xxx.vm-2002016-03-08T190028.vbk

Total IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |    137800712192 |       131417 |     438.05 |     438.05 | D:\Backups\Nightly Backup job for Applications\xxx-xxx.vm-2002016-03-08T190028.vbk (90GB)
------------------------------------------------------------------------------
total:      137800712192 |       131417 |     438.05 |     438.05

Read IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |    137800712192 |       131417 |     438.05 |     438.05 | D:\Backups\Nightly Backup job for Applications\xxx-xxx.vm-2002016-03-08T190028.vbk (90GB)
------------------------------------------------------------------------------
total:      137800712192 |       131417 |     438.05 |     438.05

Write IO
thread |       bytes     |     I/Os     |     MB/s   |  I/O per s |  file
------------------------------------------------------------------------------
     0 |               0 |            0 |       0.00 |       0.00 | D:\Backups\Nightly Backup job for Applications\xxx-xxx.vm-2002016-03-08T190028.vbk (90GB)
------------------------------------------------------------------------------
total:                 0 |            0 |       0.00 |       0.00


Both jobs running simultaneously are getting 400MB/s+.
cerberus
Enthusiast
 
Posts: 33
Liked: 3 times
Joined: Fri Aug 28, 2015 2:45 pm
Full Name: MD

[MERGED] Veeam Slow backup to tape

Veeam Logoby msmith_uk » Fri Apr 01, 2016 3:40 pm

Hi all,

Im having a issue with my new veeam setup so far support have not been able to assist with resolving this, i hoped someone may have seen it or have a idea :)

Basically when backing up to tape we are getting speeds of 14mbs on Veeam, to compare Backup Exec we was getting around 50mbs.

Our setup is as follows Veeam backing up to the disk on the SAN from a physical server with a dell TL2000 attached to the physical via iSCSI backups the 'to disk jobs' to tape.

the only thing i have seen is it feels as though it has capped itself at 14mbs. though there are no options to show so.

Any pointers would be greatly appreciated.

Thanks
msmith_uk
Lurker
 
Posts: 2
Liked: never
Joined: Fri Apr 01, 2016 3:31 pm
Full Name: matt smith

Re: v9 - Tape performance reports

Veeam Logoby Dima P. » Sat Apr 02, 2016 1:23 pm

Hi Matt,

Could you collect and share the info described in this post. Thanks.
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

PreviousNext

Return to Tape



Who is online

Users browsing this forum: No registered users and 6 guests