Remote tape library. Slow performance

Everything about backing up to tape

Re: Remote tape library. Slow performance

Veeam Logoby Dima P. » Tue Nov 11, 2014 2:39 pm

Hello all,

I’ve merged all existing remote tape library performance posts into one, as it might be caused by the similar issue. Since we are still investigating this problem, please open a support case with all the troubleshooting information you have and post the case ID to this thread. QA is looking into this.

Thank you.
Dima P.
Veeam Software
 
Posts: 6261
Liked: 441 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Remote tape library. Slow performance

Veeam Logoby nreutemann » Tue Nov 11, 2014 2:56 pm

d.popov wrote:Hello all,

I’ve merged all existing remote tape library performance posts into one, as it might be caused by the similar issue. Since we are still investigating this problem, please open a support case with all the troubleshooting information you have and post the case ID to this thread. QA is looking into this.

Thank you.


Hi all,

In my case, I got 2 IBM TS3100, one locally attached to the Veeam Server (via SAS), and the other one, on a remote site, with a link of fiber of 1Gbps (100 meters between sites).
The jobs pointing to both libraries shows the same througput, 25Mb/s.
If Im not wrong, Veeam use a proxy service (or something like that) to access to both libraries, one proxy on the same server and the other on the remote server, right?
Maybe the problem will be there, in the communication between Veeam and the proxy service.

Maybe, Im saying something that all Veem crew already knows, but I like to think and write.

Best regards,

Nicolas.
nreutemann
Enthusiast
 
Posts: 47
Liked: 6 times
Joined: Tue Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann

Re: Remote tape library. Slow performance

Veeam Logoby mpozar » Wed Nov 12, 2014 12:26 am

IS the Tape Job still trying to process a Synthesized Full?

We found that to get the performance back we had to modify the Schedule under the Media Pool for full backups and then change the default that is set to weekly on Saturday to Last Sunday of December (no other month) and then when we reran the Job we found that it was no longer trying to create the Synthesised Full on Tape and our backup speed had gone back to close to normal..


Have FUN!
Michael Pozar
mpozar
Enthusiast
 
Posts: 30
Liked: never
Joined: Sun Jan 01, 2006 1:01 am

Re: Remote tape library. Slow performance

Veeam Logoby nreutemann » Wed Nov 12, 2014 12:37 am

I will try that!

In the middle, Im running a File to tape backup from a server wich has a lot of video files, the throughput is near 65Mb/s so, seems the problem is with the veeam files.

Regards

Nicolas.
nreutemann
Enthusiast
 
Posts: 47
Liked: 6 times
Joined: Tue Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann

Re: Remote tape library. Slow performance

Veeam Logoby mpozar » Wed Nov 12, 2014 1:04 am

We monitored the Veeam Backup Server (Physical Windows 2012 R2 with locally attached SAS IBM TS3100 via Task Manager -> Performance Monitor and noticed that on disk access the .VBK, a collection of .VRB and .VSB( this is the Synthesised Full for Tape file) were being accessed.

After making the change to the Schedule on Media pool for full backups we found that the .VRB and the .VSB were no longer being accessed and our speed was back to nearly normal. In v7 we were getting between 78-81MB/s on average and now we are seeing 75-76MB/s average.

Another thing we noticed when the Tape backup Job was trying to process the Synthesized Full was that sometime it showed the NETWORK as being the bottleneck. Again when we looked at Performance Monitor we noticed that there were 5 Network streams running and their TOTAL throughput was around 25MB/s which was approximately what the Job was showing as the throughput.


Have FUN!
Michael Pozar
mpozar
Enthusiast
 
Posts: 30
Liked: never
Joined: Sun Jan 01, 2006 1:01 am

Re: Remote tape library. Slow performance

Veeam Logoby mpozar » Wed Nov 12, 2014 1:12 am

Quick question to Veeam.

Is there a way to DISABLE/Turn Off the Full Backup Schedule on the Media Pool for backups?

Us changing the Schedule as described in previous posts has allowed us to get our Tape backup Jobs throughput nearly back to normal but I am assuming that we are going to process a Full Backup onto Tape on the Last Sunday of December even though we do not require it since we are running Reverse Incremental backup Jobs and every backup is a FULL.

The Veeam documentation describes this Full Backup to tape as a feature for those that are NOT regularly creating a FULL backup i.e.. Forever Incremental , and thus this feature was added that would require no additional disk space to be able to create a Full Backup onto tape on a regular basis.

This is a nice feature for those running Forever Incremental and Tape Backups but based on our experience it is just too SLOWWWW....


Have FUN!
Michael Pozar
mpozar
Enthusiast
 
Posts: 30
Liked: never
Joined: Sun Jan 01, 2006 1:01 am

Re: Remote tape library. Slow performance

Veeam Logoby v.Eremin » Wed Nov 12, 2014 10:40 am

Michael, has you support engineer provided a private fix that we mentioned earlier in this thread? If so, is this issue reproducible on backup server with private fix installed? Thanks.
v.Eremin
Veeam Software
 
Posts: 13291
Liked: 973 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Remote tape library. Slow performance

Veeam Logoby caccia82 » Wed Nov 12, 2014 11:11 am

Hi,
slow performance also in my environment.

Previous situation:
Veeam B&R 7: Tape performance 70MB/s

Actual situation:
Veeam B&R 8: Tape performace 25MB/s
BottleNeck Network 95%

Tape Library: Ultrium 1760 LTO4 (SAS)
caccia82
Novice
 
Posts: 5
Liked: never
Joined: Thu Sep 11, 2014 10:17 pm

Re: Remote tape library. Slow performance

Veeam Logoby v.Eremin » Wed Nov 12, 2014 11:19 am

Are you also using reversed incremental mode? Have you applied the private fix mentioned above? Thanks.
v.Eremin
Veeam Software
 
Posts: 13291
Liked: 973 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

[MERGED] : Question about the new "Synthesized Full backup"

Veeam Logoby Maurice » Wed Nov 12, 2014 1:43 pm

Hello everyone,

i have a question about "Synthesized Full Backup" for Backup to Tape Jobs:
I'm using reversed Incremental for all Backup Jobs and i'm copying the Full Backups to Tape weekly.
I only selected a Full Media Pool for that specific Tape Job to just copy those Full Backups.
But with v8 the Tape Job takes ages (ok, about 4-5 Hours) for creating synthesized Full Backup and then starts writing data on Tape, even when there is no Backup-to-Disk Job running.
Is that behaviour normal even when there is a up-to-date Full Backup available or do i have to configure something specific inside that Tape Job to "deactivate" the "synthesized Full Backup"?

P.S.: Will there be a VMCE Course/Certification for v8?


Thanks & Greetings
Maurice
Maurice
Service Provider
 
Posts: 21
Liked: 2 times
Joined: Mon May 26, 2014 7:59 am
Location: Münster, Germany
Full Name: Maurice Galicic

Re: Remote tape library. Slow performance

Veeam Logoby v.Eremin » Wed Nov 12, 2014 1:54 pm

Hi, Maurice,

As mentioned above, kindly, open a ticket with our support team and request the said private fix.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13291
Liked: 973 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Remote tape library. Slow performance

Veeam Logoby caccia82 » Wed Nov 12, 2014 2:50 pm

No, i'm using Forever Incremental

I have not applied the patch. I'll open a ticket.
caccia82
Novice
 
Posts: 5
Liked: never
Joined: Thu Sep 11, 2014 10:17 pm

Re: Remote tape library. Slow performance

Veeam Logoby v.Eremin » Wed Nov 12, 2014 3:06 pm

It'll be the best option, indeed. Once you install the said fix, please, double-check whether or not the issue persists. Thanks.
v.Eremin
Veeam Software
 
Posts: 13291
Liked: 973 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Remote tape library. Slow performance

Veeam Logoby Maurice » Thu Nov 13, 2014 2:58 pm

v.Eremin wrote:Hi, Maurice,

As mentioned above, kindly, open a ticket with our support team and request the said private fix.

Thanks.


Hi Eremin,

thanks for Merging and answering.
My Case-ID is # 00678114.
I'm waiting now for that Patch, but what i noticed is that the synthesized Full Backup is HUGE!!!
I have 10 Jobs which i put on tape every Week and it used before v8 upgrade just One LTO-5 Tape for all Jobs.
After upgrading to v8 the Tape Job starts to synthesize Full Backups of all Jobs and after that it starts copying the first Job, but it only copies 33% of that Job cause the Tape is already full.
That is strange in my Opinion cause it looks like every change of the Backup-to-Disk Job (rev. incr. with 93 Retention Points) will be inside that synthesized Full Backup.

Is that correct and a wanted behaviour?


Thanks and Greetings
Maurice
Maurice
Service Provider
 
Posts: 21
Liked: 2 times
Joined: Mon May 26, 2014 7:59 am
Location: Münster, Germany
Full Name: Maurice Galicic

Re: Remote tape library. Slow performance

Veeam Logoby v.Eremin » Thu Nov 13, 2014 3:09 pm

The main issue is that a backup to tape job is currently copying .vrb files; which is not by design. The fact that it's trying to synthesize rollbacks is related to that issue. Once private fix addressing the first problem is installed, the second issue will go away automatically. Thanks.
v.Eremin
Veeam Software
 
Posts: 13291
Liked: 973 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

PreviousNext

Return to Tape



Who is online

Users browsing this forum: JasonBailey and 9 guests