Discussions specific to tape backups
Dima P.
Product Manager
Posts: 9782
Liked: 774 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Remote tape library. Slow performance

Post by Dima P. » Nov 11, 2014 2:39 pm

Hello all,

I’ve merged all existing remote tape library performance posts into one, as it might be caused by the similar issue. Since we are still investigating this problem, please open a support case with all the troubleshooting information you have and post the case ID to this thread. QA is looking into this.

Thank you.

nreutemann
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann
Contact:

Re: Remote tape library. Slow performance

Post by nreutemann » Nov 11, 2014 2:56 pm

d.popov wrote:Hello all,

I’ve merged all existing remote tape library performance posts into one, as it might be caused by the similar issue. Since we are still investigating this problem, please open a support case with all the troubleshooting information you have and post the case ID to this thread. QA is looking into this.

Thank you.
Hi all,

In my case, I got 2 IBM TS3100, one locally attached to the Veeam Server (via SAS), and the other one, on a remote site, with a link of fiber of 1Gbps (100 meters between sites).
The jobs pointing to both libraries shows the same througput, 25Mb/s.
If Im not wrong, Veeam use a proxy service (or something like that) to access to both libraries, one proxy on the same server and the other on the remote server, right?
Maybe the problem will be there, in the communication between Veeam and the proxy service.

Maybe, Im saying something that all Veem crew already knows, but I like to think and write.

Best regards,

Nicolas.

mpozar
Enthusiast
Posts: 30
Liked: 1 time
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Remote tape library. Slow performance

Post by mpozar » Nov 12, 2014 12:26 am

IS the Tape Job still trying to process a Synthesized Full?

We found that to get the performance back we had to modify the Schedule under the Media Pool for full backups and then change the default that is set to weekly on Saturday to Last Sunday of December (no other month) and then when we reran the Job we found that it was no longer trying to create the Synthesised Full on Tape and our backup speed had gone back to close to normal..


Have FUN!
Michael Pozar

nreutemann
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann
Contact:

Re: Remote tape library. Slow performance

Post by nreutemann » Nov 12, 2014 12:37 am

I will try that!

In the middle, Im running a File to tape backup from a server wich has a lot of video files, the throughput is near 65Mb/s so, seems the problem is with the veeam files.

Regards

Nicolas.

mpozar
Enthusiast
Posts: 30
Liked: 1 time
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Remote tape library. Slow performance

Post by mpozar » Nov 12, 2014 1:04 am

We monitored the Veeam Backup Server (Physical Windows 2012 R2 with locally attached SAS IBM TS3100 via Task Manager -> Performance Monitor and noticed that on disk access the .VBK, a collection of .VRB and .VSB( this is the Synthesised Full for Tape file) were being accessed.

After making the change to the Schedule on Media pool for full backups we found that the .VRB and the .VSB were no longer being accessed and our speed was back to nearly normal. In v7 we were getting between 78-81MB/s on average and now we are seeing 75-76MB/s average.

Another thing we noticed when the Tape backup Job was trying to process the Synthesized Full was that sometime it showed the NETWORK as being the bottleneck. Again when we looked at Performance Monitor we noticed that there were 5 Network streams running and their TOTAL throughput was around 25MB/s which was approximately what the Job was showing as the throughput.


Have FUN!
Michael Pozar

mpozar
Enthusiast
Posts: 30
Liked: 1 time
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Remote tape library. Slow performance

Post by mpozar » Nov 12, 2014 1:12 am

Quick question to Veeam.

Is there a way to DISABLE/Turn Off the Full Backup Schedule on the Media Pool for backups?

Us changing the Schedule as described in previous posts has allowed us to get our Tape backup Jobs throughput nearly back to normal but I am assuming that we are going to process a Full Backup onto Tape on the Last Sunday of December even though we do not require it since we are running Reverse Incremental backup Jobs and every backup is a FULL.

The Veeam documentation describes this Full Backup to tape as a feature for those that are NOT regularly creating a FULL backup i.e.. Forever Incremental , and thus this feature was added that would require no additional disk space to be able to create a Full Backup onto tape on a regular basis.

This is a nice feature for those running Forever Incremental and Tape Backups but based on our experience it is just too SLOWWWW....


Have FUN!
Michael Pozar

v.eremin
Product Manager
Posts: 16160
Liked: 1320 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by v.eremin » Nov 12, 2014 10:40 am

Michael, has you support engineer provided a private fix that we mentioned earlier in this thread? If so, is this issue reproducible on backup server with private fix installed? Thanks.

caccia82
Novice
Posts: 5
Liked: never
Joined: Sep 11, 2014 10:17 pm
Contact:

Re: Remote tape library. Slow performance

Post by caccia82 » Nov 12, 2014 11:11 am

Hi,
slow performance also in my environment.

Previous situation:
Veeam B&R 7: Tape performance 70MB/s

Actual situation:
Veeam B&R 8: Tape performace 25MB/s
BottleNeck Network 95%

Tape Library: Ultrium 1760 LTO4 (SAS)

v.eremin
Product Manager
Posts: 16160
Liked: 1320 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by v.eremin » Nov 12, 2014 11:19 am

Are you also using reversed incremental mode? Have you applied the private fix mentioned above? Thanks.

Maurice
Service Provider
Posts: 25
Liked: 2 times
Joined: May 26, 2014 7:59 am
Full Name: Maurice Galicic
Location: Münster, Germany
Contact:

[MERGED] : Question about the new "Synthesized Full backup"

Post by Maurice » Nov 12, 2014 1:43 pm

Hello everyone,

i have a question about "Synthesized Full Backup" for Backup to Tape Jobs:
I'm using reversed Incremental for all Backup Jobs and i'm copying the Full Backups to Tape weekly.
I only selected a Full Media Pool for that specific Tape Job to just copy those Full Backups.
But with v8 the Tape Job takes ages (ok, about 4-5 Hours) for creating synthesized Full Backup and then starts writing data on Tape, even when there is no Backup-to-Disk Job running.
Is that behaviour normal even when there is a up-to-date Full Backup available or do i have to configure something specific inside that Tape Job to "deactivate" the "synthesized Full Backup"?

P.S.: Will there be a VMCE Course/Certification for v8?


Thanks & Greetings
Maurice

v.eremin
Product Manager
Posts: 16160
Liked: 1320 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by v.eremin » Nov 12, 2014 1:54 pm

Hi, Maurice,

As mentioned above, kindly, open a ticket with our support team and request the said private fix.

Thanks.

caccia82
Novice
Posts: 5
Liked: never
Joined: Sep 11, 2014 10:17 pm
Contact:

Re: Remote tape library. Slow performance

Post by caccia82 » Nov 12, 2014 2:50 pm

No, i'm using Forever Incremental

I have not applied the patch. I'll open a ticket.

v.eremin
Product Manager
Posts: 16160
Liked: 1320 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by v.eremin » Nov 12, 2014 3:06 pm

It'll be the best option, indeed. Once you install the said fix, please, double-check whether or not the issue persists. Thanks.

Maurice
Service Provider
Posts: 25
Liked: 2 times
Joined: May 26, 2014 7:59 am
Full Name: Maurice Galicic
Location: Münster, Germany
Contact:

Re: Remote tape library. Slow performance

Post by Maurice » Nov 13, 2014 2:58 pm

v.Eremin wrote:Hi, Maurice,

As mentioned above, kindly, open a ticket with our support team and request the said private fix.

Thanks.
Hi Eremin,

thanks for Merging and answering.
My Case-ID is # 00678114.
I'm waiting now for that Patch, but what i noticed is that the synthesized Full Backup is HUGE!!!
I have 10 Jobs which i put on tape every Week and it used before v8 upgrade just One LTO-5 Tape for all Jobs.
After upgrading to v8 the Tape Job starts to synthesize Full Backups of all Jobs and after that it starts copying the first Job, but it only copies 33% of that Job cause the Tape is already full.
That is strange in my Opinion cause it looks like every change of the Backup-to-Disk Job (rev. incr. with 93 Retention Points) will be inside that synthesized Full Backup.

Is that correct and a wanted behaviour?


Thanks and Greetings
Maurice

v.eremin
Product Manager
Posts: 16160
Liked: 1320 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by v.eremin » Nov 13, 2014 3:09 pm

The main issue is that a backup to tape job is currently copying .vrb files; which is not by design. The fact that it's trying to synthesize rollbacks is related to that issue. Once private fix addressing the first problem is installed, the second issue will go away automatically. Thanks.

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests