v9 - Tape performance reports

Everything about backing up to tape

Re: v9 - Tape performance reports

Veeam Logoby cma@ahbl.ca » Tue Apr 26, 2016 11:14 pm

We're experiencing performance issues too. As requested:

0. Support case ID
01778189

1. Background on your infrastructure setup (i.e. type of the library, how it's connected to tape proxy, generation of tape drives)
StorageTek SL24 Tape Autloader, direct attached to Dell PE1850 server (Windows Server 2008 R2 Standard), LTO4 tape drive

2. What type of tape job were you using during the performance testing (file to tape / backup to tape / file from tape restore / backup from tape restore)
file to tape

3. Bottleneck stats from job details
Source: 0%
Proxy: 4%
Network: 0%
Target: 27%

4. Average size of the file and number of backed up/restored files (a rough estimate is good enough)
Average file size 133GB and approximately 20 files

5. Any other useful information
Prior to v9 u1, the backup job was averaging 80MB/s. We are now average 25MB/s


Observation: Not sure if this is related or coincidental, prior to upgrade to v9 u1, our Bottleneck stats always had stats for all 4 areas (Source, Proxy, Network, Target). With v9 u1, we only appear to have stats for Proxy and Target; Source and Network are always 0%. Is there a new method/algorithm in v9 for measuring these stats?
cma@ahbl.ca
Lurker
 
Posts: 2
Liked: never
Joined: Mon Apr 25, 2016 5:08 pm
Full Name: Chris Ma

Re: v9 - Tape performance reports

Veeam Logoby rreed » Wed Apr 27, 2016 1:14 pm

What's your source, Chris (CIFS share, Windows share, direct SAN, etc. on the network or the local tape server HDD?) Reason I ask is because U1 introduced a bug in CIFS shares getting piped to tape. Don't know if it would have affected speed, but if you look at the properties of a file to tape job that's sourced off a CIFS share it can lock the domain account credentials.

I do hope they fixed the goofy tape speed calculator in U1 though, while it was usually reasonably accurate while watching resource monitor, sometimes my LTO-6 would randomly report 320+ MB/s! We wish.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: v9 - Tape performance reports

Veeam Logoby cma@ahbl.ca » Thu Apr 28, 2016 7:44 pm

Hi rreed, thanks for the info on the CIFS bug. I'm aware of the bug as it impacted one of our backup jobs. We have since received the hotfix and that resolved that issue.

I have been working with Veeam support with our performance issue. After reviewing our logs, they have confirmed our performance slowness and had me check the following:

1. Please edit your File to Tape job and under Options, do you have Microsoft Volume Shadow Copy enabled?
No, it is not enabled.

2. Under Tape Infrastructure, Right click on your tape library and go to properties, do you have Use Native SCSI commands instead of Windows Driver enabled?
No, it is not enabled.

3. Under Tape Infrastructure, right click on Media Pool used for this job and go to properties. Under Options, do you have Parallel Processing for jobs using this media pool enabled?
No, it is not enabled.

I'm waiting to hear back.

Chris
cma@ahbl.ca
Lurker
 
Posts: 2
Liked: never
Joined: Mon Apr 25, 2016 5:08 pm
Full Name: Chris Ma

Re: v9 - Tape performance reports

Veeam Logoby HussainMahfood » Tue May 03, 2016 7:41 am

+1

I am facing same issue even with V9 update 1 max 67MB processing compared to other softwares tape backup 120MB
Tapes: LTO5
Tape Library: TL2000

:(
HussainMahfood
Enthusiast
 
Posts: 35
Liked: 7 times
Joined: Mon Jun 24, 2013 9:43 am
Full Name: Hussain Mahfood

Re: v9 - Tape performance reports

Veeam Logoby StevenMeier » Mon May 09, 2016 2:42 am

we are still working on tape issues ..I am beginning to get very concerned that there does not seem to be a veeam post about this issue.

It is obviously affecting a number of folks and symptoms seem very similar.

This lack of communication is a worry.
StevenMeier
Service Provider
 
Posts: 24
Liked: never
Joined: Fri Apr 22, 2016 1:06 am
Full Name: Steven Meier

Re: v9 - Tape performance reports

Veeam Logoby Dima P. » Mon May 09, 2016 12:32 pm

Hi Steven,

We are collecting and reviewing every single report in this thread. While some ‘configuration issues’ can be resolved together with the support team, DEVs are now working on improving the overall performance for all existing tape functionality (F2T, B2T and GFS jobs), so all these posts and support cases help a lot to identify the exact areas that needs to be investigated by R&D folks.
Dima P.
Veeam Software
 
Posts: 6263
Liked: 442 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: v9 - Tape performance reports

Veeam Logoby McClane » Wed May 11, 2016 10:03 am

0. 01796032
1. Quantum Scalar i500 with LTO7 drives connected via FC over Brocade switches. Repository is on IBM V3700 SAN with 12 SAS drives. Different FC HBAs for SAN and library
2. Backup from tape to repositry
3. Restore session does not have bottleneck display
4. One 3.4TB VBK
5. Ressource montor in windows tells me high disk usage, but no actually data transfer (I/O 100% on repository disk, but no actually bytes transferred for a long period of time)
McClane
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Sat Jun 20, 2009 12:47 pm

Re: v9 - Tape performance reports

Veeam Logoby Dima P. » Thu May 12, 2016 5:51 pm

McClane,

Can you clarify the average performance rate in your case?
Dima P.
Veeam Software
 
Posts: 6263
Liked: 442 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: v9 - Tape performance reports

Veeam Logoby McClane » Fri May 13, 2016 6:21 am

It seems that the performance counters have shown false data for about an hour. After that the rate was constantly 150MB/s for the next 6 hours. That only happens with big jobs. A 100GB restore displayed the over 100MB/s from the beginning.
McClane
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Sat Jun 20, 2009 12:47 pm

[MERGED] Backup files to tape very very slow

Veeam Logoby keesvoortman » Thu Jun 16, 2016 12:49 pm

Hello Veeam users,

I am using VEEAM Backup and Replication version 9.0.0.1491.
I'm backing up 2 folders to tape in a backup job.
One folder contains all VM backups.
The other folder contains about 400GB with tons of individual files (pdf, xls, doc, txt etc etc).

When backing up these files to tape I have to stop the job cause it takes sooooooooo long.
Withouth this folder the backup of the folder with VM's is during about 2 hours, so something's got wrong with the file folder....

Does anyone have an idea ?
Thank you in advance.
keesvoortman
Lurker
 
Posts: 1
Liked: never
Joined: Thu Jun 16, 2016 12:38 pm
Full Name: Kees Voortman

Re: v9 - Tape performance reports

Veeam Logoby Dima P. » Thu Jun 16, 2016 12:56 pm

Hi Kees,

Can you share the information described in this post? Thanks.
Dima P.
Veeam Software
 
Posts: 6263
Liked: 442 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

[MERGED] File to Tape Job is very slow to backup CIFS Shared

Veeam Logoby nfoug75 » Mon Sep 12, 2016 9:44 am

Hello,
I have a lot of file to save in full. 1 T 5 of files... It takes more that 1 million of files ... to save
It is very slow to backup all this files .... Veeam is not really fast for that... no drivers...

How can i proceed ?
nfoug75
Enthusiast
 
Posts: 25
Liked: 1 time
Joined: Thu Jun 09, 2016 11:20 am
Full Name: Nicolas FOUGEROUX

Re: v9 - Tape performance reports

Veeam Logoby Vitaliy S. » Mon Sep 12, 2016 10:10 am

Can you please provide a bit more on your setup (see Dima's post above)? Also what version of Veeam B&R are you using?
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1106 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: v9 - Tape performance reports

Veeam Logoby nfoug75 » Mon Sep 12, 2016 12:56 pm

It is the last version 9.0.0.1715.. It is a VM the server Veeam.

0. Support case ID
1. Background on your infrastructure setup (i.e. type of the library, how it's connected to tape proxy, generation of tape drives) - PV124T connected to a physical W2008 LTO4 drive
2. What type of tape job were you using during the performance testing (file to tape / backup to tape / file from tape restore / backup from tape restore) - File To tape Job
3. Bottleneck stats from job details - Proxy
4. Average size of the file and number of backed up/restored files (a rough estimate is good enough) - 1T4, 1 million 285000 files
5. Any other useful information

Source : 0 %
Proxy : 35 %
?etwork 0 %
Target 0 %
nfoug75
Enthusiast
 
Posts: 25
Liked: 1 time
Joined: Thu Jun 09, 2016 11:20 am
Full Name: Nicolas FOUGEROUX

Re: v9 - Tape performance reports

Veeam Logoby larry » Tue Sep 13, 2016 9:09 pm

LTO 7 drive with LTO 7 7 tape, local disk as source backup speed 303 MBS
LTO 7 drive with LTO6 tape Local disk as source backup - speed 153MBS

LTO 4 driveLTO4 tape local or remote server 53MBS

LTO 7 drive/Tape 6 remote server backup 148 mbs . I got this after being sure data stayed on 10 gig lan.Tape job would use the wrong path given a chance.Used IP addresses for everything (source,tape server)

CPU on tape server backing up local VeeamAgent.exe < 2% - Server idle.

CPU when backing up from remote. VeeamAgent.exe (remote server source CPU 25-30%) (tape server CPU 6%) not sure why the high CPU when I am doing Files and Folders to Tape. Picked remote server from drop down.

CPU when backing up from remote using unc like \\174.46.44.248\f$\Backups when selecting source.When picking source server I click add then type unc path. VeeamAgent.exe (remote server source 0%) (tape server 0%) both servers light cpu load when typing unc and not using pull down. LTO 7 drive/Tape 6 remote server backup 146 mbs.


My test files are Veeam backups about 1tb in 40 files, using file and folders to tape.

I switched my tape backups to Veeam backup copy to local disk, then local disk to tape. Both jobs finish before one tape from remote. The good side effect is now I get all my restore points on tape. My main speed issue was the data would take a 1 gig link at times and using all IP addresses for names fixed that. But I see where at night with other Veeam jobs running High CPU may be an issue.

Just sharing.
larry
Expert
 
Posts: 372
Liked: 89 times
Joined: Wed Mar 24, 2010 5:47 pm
Full Name: Larry Walker

PreviousNext

Return to Tape



Who is online

Users browsing this forum: No registered users and 6 guests