Remote tape library. Slow performance

[nreutemann]

I got this "situation",

We got two TS3100 on our backup infrastructure. After we upgrade to v8, the jobs "backup to tape" shows a "network bottleneck", and the throughput is "flat" on 24Mb/s. The same jobs with the v7, has a troughput of 90Mb/s.
One of the library is conected to the Veeam Sever, the other one, on a remote server (the lan between both servers are 1Gbps).

Any idea?

Thanks in advance,

Nicolas.

Support case: ID#00674598

[Dima P.]

Nicolas,

From the top of my head - could you check if both tape servers are not listed in the global network traffic rules or in local network traffic rules (go to tape infrastructure, select servers right click server > properties and go to Traffic step)? Thank you.

[nreutemann]

Hi,

Now, yes, I add the subnet to "global network traffic rules".
I set up the throttling to 9999Mb/s.
I do this because that was the 1st thing I think "maybe, there is a rule wich constrains the traffic", but no, after the setup, the situation remains the same.

Thanks!

Nicolas.

(Im from Argentina, thats why my english is so poor, sorry).

[nreutemann]

Now, I delete all the rules about throttling, and the performance still remains the same (same numbers like before I create the throttling rules.)

The weird part of this, both jobs has the same performance, the one that are writing to the locally attached library as the another one, attached to the remote server.

[mpozar]

After our Upgrade from V7 to V8 we saw the same issue and found that in our environment; we run Reverse Incremental Backups, the Tape Backup job was processing a Synthetic Full on to the tape and had created a .vsb file and was processing the .VBK and some of our .VRB.

We could not find a way to turn of the creation of the Full on Tape, there is only an option to schedule. We then changed the schedule from the Default of doing it every Saturday to the Last Sunday of December.

When we re-ran the Tape Backup Job we found it was no longer creating a Full and the backup speed returned to about what it was previously but we now have another issue where the Tape Backup Jobs finished with an error message:

MediaPool not found (id: 00000000-0000-0000-0000-000000000000)

We now have a case open with Veeam Support regarding the above error and also trying to get an explanation of the changes to the Tape Backup Process.

As an update, we have managed to get the Tape Backup Job to work successfully by also selection to backup the Incremental; note that we a re running Reverse Incremental and the Veeam documentation makes no mention that I could find that Veeam supports backing up .VRB, and found that the Job now also backed up the .VRB files and then the Job completed without the above error.

Case # 00671757


Have FUN!
Michael Pozar

[veremin]

Hi all,

Currently, we're trying to reproduce the reported issues (network bottleneck and throttling rules). I believe we will update the topic, once we have more information.

As to backup to tape job unexpectedly copying .vrb files, we're already aware of this behavior and have created a private fix. Kindly, contact support team in order to get it.

Thanks.

[frinnst]

Hi!

So we upgraded to V8 as soon as it became available for the improved Tape support (yay!). We are, however, experiencing some *slow* performance - 4,6MB/s (Bottleneck: Target).
As mentioned in the topic we are using the remote library feature. The Veeam server is installed on a VM (2012r2) and the Tape hardware (HP 1x8 G2 Autoldr) is connected to a physical 2008r2 server.

So the performance is pretty poor. Any suggestions on how to debug this? We could move to the pysical server if that would improve things but i'd rather not if the performance is the same.

[veremin]

Fredrik,

We are, however, experiencing some *slow* performance - 4,6MB/s (Bottleneck: Target).
As mentioned in the topic we are using the remote library feature. The Veeam server is installed on a VM (2012r2) and the Tape hardware (HP 1x8 G2 Autoldr) is connected to a physical 2008r2 server.

So the performance is pretty poor. Any suggestions on how to debug this? We could move to the pysical server if that would improve things but i'd rather not if the performance is the same.

What type of tape job are you using? Backups to tape or files to tape one? What is selected as a source for it? Couple of big backup files or multiple small files? What performance have you had previously? Can you also provide full bottleneck statistics (the one with percent distribution). Thanks.

[nreutemann]

Hi, after read your post, I check the log of the jobs and found the same behaviour. The job has queued the .vbk, .vib and .vrb to backup to tape.

cya

Nicolas.

[veremin]

Hi Nicolas,

As mentioned above, kindly, open a ticket with our support team and you will be provided with a private fix soon. It resolves the issue with backup to tape job copying .vrb files.

Thanks.

[frinnst]

Can you also provide full bottleneck statistics (the one with percent distribution).

We are using "Backup to tape".

Bottleneck:
Source: 0%
Proxy: 15%
Network: 0%
Target: 99%

[nreutemann]

Hi, Nicolas,

As mentioned above, kindly, open a ticket with our support team and you will be provided with a private fix soon. It resolves the issue with backup to tape job copying .vrb files.

Thanks.

I already created an Support ticket, the case ID is on the first post of this thread.

Support case: ID#00674598

Best regards,

Nicolas.

[frinnst]

What type of tape job are you using? Backups to tape or files to tape one? What is selected as a source for it? Couple of big backup files or multiple small files?

We are testing a simple job that has one full backup (257gb) and two diff (5gb, 9gb)

[veremin]

Nicolas,
We've just sent a private fix to your support engineer. I believe he will provide you with it soon. Thanks.

[nreutemann]

We've just sent a private fix to your support engineer. I believe he will provide you with it soon. Thanks.

Hi Vladimir, Im testing now the fix.

Fix the .vrb problem, but the bottleneck remains, the speed is 25Mb/s

Best regards.

Nicolas.

[Dima P.]

Hello all,

I’ve merged all existing remote tape library performance posts into one, as it might be caused by the similar issue. Since we are still investigating this problem, please open a support case with all the troubleshooting information you have and post the case ID to this thread. QA is looking into this.

Thank you.

[nreutemann]

Hello all,

I’ve merged all existing remote tape library performance posts into one, as it might be caused by the similar issue. Since we are still investigating this problem, please open a support case with all the troubleshooting information you have and post the case ID to this thread. QA is looking into this.

Thank you.

Hi all,

In my case, I got 2 IBM TS3100, one locally attached to the Veeam Server (via SAS), and the other one, on a remote site, with a link of fiber of 1Gbps (100 meters between sites).
The jobs pointing to both libraries shows the same througput, 25Mb/s.
If Im not wrong, Veeam use a proxy service (or something like that) to access to both libraries, one proxy on the same server and the other on the remote server, right?
Maybe the problem will be there, in the communication between Veeam and the proxy service.

Maybe, Im saying something that all Veem crew already knows, but I like to think and write.

Best regards,

Nicolas.

[mpozar]

IS the Tape Job still trying to process a Synthesized Full?

We found that to get the performance back we had to modify the Schedule under the Media Pool for full backups and then change the default that is set to weekly on Saturday to Last Sunday of December (no other month) and then when we reran the Job we found that it was no longer trying to create the Synthesised Full on Tape and our backup speed had gone back to close to normal..


Have FUN!
Michael Pozar

[nreutemann]

I will try that!

In the middle, Im running a File to tape backup from a server wich has a lot of video files, the throughput is near 65Mb/s so, seems the problem is with the veeam files.

Regards

Nicolas.

[mpozar]

We monitored the Veeam Backup Server (Physical Windows 2012 R2 with locally attached SAS IBM TS3100 via Task Manager -> Performance Monitor and noticed that on disk access the .VBK, a collection of .VRB and .VSB( this is the Synthesised Full for Tape file) were being accessed.

After making the change to the Schedule on Media pool for full backups we found that the .VRB and the .VSB were no longer being accessed and our speed was back to nearly normal. In v7 we were getting between 78-81MB/s on average and now we are seeing 75-76MB/s average.

Another thing we noticed when the Tape backup Job was trying to process the Synthesized Full was that sometime it showed the NETWORK as being the bottleneck. Again when we looked at Performance Monitor we noticed that there were 5 Network streams running and their TOTAL throughput was around 25MB/s which was approximately what the Job was showing as the throughput.


Have FUN!
Michael Pozar

[mpozar]

Quick question to Veeam.

Is there a way to DISABLE/Turn Off the Full Backup Schedule on the Media Pool for backups?

Us changing the Schedule as described in previous posts has allowed us to get our Tape backup Jobs throughput nearly back to normal but I am assuming that we are going to process a Full Backup onto Tape on the Last Sunday of December even though we do not require it since we are running Reverse Incremental backup Jobs and every backup is a FULL.

The Veeam documentation describes this Full Backup to tape as a feature for those that are NOT regularly creating a FULL backup i.e.. Forever Incremental , and thus this feature was added that would require no additional disk space to be able to create a Full Backup onto tape on a regular basis.

This is a nice feature for those running Forever Incremental and Tape Backups but based on our experience it is just too SLOWWWW....


Have FUN!
Michael Pozar

[veremin]

Michael, has you support engineer provided a private fix that we mentioned earlier in this thread? If so, is this issue reproducible on backup server with private fix installed? Thanks.

[caccia82]

Hi,
slow performance also in my environment.

Previous situation:
Veeam B&R 7: Tape performance 70MB/s

Actual situation:
Veeam B&R 8: Tape performace 25MB/s
BottleNeck Network 95%

Tape Library: Ultrium 1760 LTO4 (SAS)

[veremin]

Are you also using reversed incremental mode? Have you applied the private fix mentioned above? Thanks.

[Maurice]

Hello everyone,

i have a question about "Synthesized Full Backup" for Backup to Tape Jobs:
I'm using reversed Incremental for all Backup Jobs and i'm copying the Full Backups to Tape weekly.
I only selected a Full Media Pool for that specific Tape Job to just copy those Full Backups.
But with v8 the Tape Job takes ages (ok, about 4-5 Hours) for creating synthesized Full Backup and then starts writing data on Tape, even when there is no Backup-to-Disk Job running.
Is that behaviour normal even when there is a up-to-date Full Backup available or do i have to configure something specific inside that Tape Job to "deactivate" the "synthesized Full Backup"?

P.S.: Will there be a VMCE Course/Certification for v8?


Thanks & Greetings
Maurice

[veremin]

Hi, Maurice,

As mentioned above, kindly, open a ticket with our support team and request the said private fix.

Thanks.

[caccia82]

No, i'm using Forever Incremental

I have not applied the patch. I'll open a ticket.

[veremin]

It'll be the best option, indeed. Once you install the said fix, please, double-check whether or not the issue persists. Thanks.

[Maurice]

Hi, Maurice,

As mentioned above, kindly, open a ticket with our support team and request the said private fix.

Thanks.

Hi Eremin,

thanks for Merging and answering.
My Case-ID is # 00678114.
I'm waiting now for that Patch, but what i noticed is that the synthesized Full Backup is HUGE!!!
I have 10 Jobs which i put on tape every Week and it used before v8 upgrade just One LTO-5 Tape for all Jobs.
After upgrading to v8 the Tape Job starts to synthesize Full Backups of all Jobs and after that it starts copying the first Job, but it only copies 33% of that Job cause the Tape is already full.
That is strange in my Opinion cause it looks like every change of the Backup-to-Disk Job (rev. incr. with 93 Retention Points) will be inside that synthesized Full Backup.

Is that correct and a wanted behaviour?


Thanks and Greetings
Maurice

[veremin]

The main issue is that a backup to tape job is currently copying .vrb files; which is not by design. The fact that it's trying to synthesize rollbacks is related to that issue. Once private fix addressing the first problem is installed, the second issue will go away automatically. Thanks.