Remote tape library. Slow performance

[veremin]

Update: Support team has created a KB article regarding this behaviour. Everyone experiencing this issue, please, check it out and download the private fix provided there. Thanks.

[Maurice]

The Path in the KB is a bit wrong.
It should be "C:\Program Files\Veeam\Backup and Replication\Backup" right?

Thanks for that fix. It worked and the Workaround with the Scheduling that was sended by the Support Team worked, too.

Great Job for the v8 by the way.
The new Features are great and i hope that the first few bugs are gone in no time.

Greetings
Maurice

[veremin]

You're right, indeed. The correct path is C:\Program Files\Veeam\Backup and Replication\Backup. I've asked already support team to update the KB.

Yes, we're trying to addressing any issues raised by users in timely fashion. Thanks be to you for drawing our attention to this or that particular problem; highly appreciated.

[nreutemann]

I create a new job, wich process the backup of only one VM, a new "backup to tape job".
Under "Media Pool", go to "Schedule" into the "media pool for full backups" and set it to "Monthly" on "Last" "Sunday" of "December" (the only month selected).
When I run the job, the throughput is 25Mb/s, the bottleneck is network.

So, I didnt see any change.

I do something wrong?

Best regards!

Our case is ID#00674598

[Dima P.]

Nicolas,

Please, continue investigation with Veeam Support team - it could be configuration specific bug and requires deep VBR log and infrastructure analysis for correct troubleshooting. Thank you.

[daxb]

Gotta say I'm pretty disappointed with Veeam support. I logged Case # 00679800 on the 16th of November as tape backups were stuck at 20MB/s following the upgrade and still haven't had anything useful from support. I found this post which appears to have resolved the issue though: http://www.v-strange.de/index.php/2-unc ... o-veeam-v8

So adding the reg key HKLM\Software\Veeam\Veeam Backup and Replication\DisablePublicIPTrafficEncryption = 1 seems to fix my issue.

Worth trying if you are experiencing slow tape transfer following installation of the KB1949 patch.

Would have saved me a lot of time if support had been as responsive as the author of the blog experienced.

[Dima P.]

Hello Dax,

I am sorry that you case resolution took longer than expected. The issue you experienced is now fully documented and applied for a fix in upcoming patch, so thank you opening a support case and providing all the troubleshooting information - it helped a lot in finding the root of this problem. Thanks again and sorry for any inconvenience caused.

[HHansen]

Hi,

I was wondering what kind of perfomance you other guys have on your tape setup with B&R 8.

I'm currently looking into replacing our HP data protector with Veeam since we already pay for it.

My setup is:

One physical server (2xCPU, 48GB RAM)
2xHP D2600 SAS disk enclosures, with 44TB storage (22 in raid 0+1), directly attached to the physical server
1x MSL 4048 library with 2x LTO5 drives, directly attached to the physical server, also using SAS

All backups are done to disk each night, and kept there for 1 week for fast restore.

Backup to tape is done on a daily basis, by backing up the data on the disks and setting retention.

With HP data protector, i get around 100-120MB/s to tape.

With Veeam i get around 40MB/s.

LTO5 is supposed to run at 140MB/s as far as i know.

Is this normal performance for Veeam or is there something wrong? Veeam says target is bottleneck.

Thanks

[Dima P.]

Henrik,
We have at least two bugs that may affect performance of your existing backup to tape jobs:

- If you using the backup to tape with .vrb files please refer to this KB article Backup to tape jobs fails when the source contains .vrb files
- If not, please check that encryption does not affect your performance by setting the custom regkey: at HKLM\Software\Veeam\Veeam Backup and Replication a DisablePublicIPTrafficEncryption DWORD with value = 1

[HHansen]

Henrik,
We have at least two bugs that may affect performance of your existing backup to tape jobs:

- If you using the backup to tape with .vrb files please refer to this KB article Backup to tape jobs fails when the source contains .vrb files
- If not, please check that encryption does not affect your performance by setting the custom regkey: at HKLM\Software\Veeam\Veeam Backup and Replication a DisablePublicIPTrafficEncryption DWORD with value = 1

The files im currently testing with is .bak files, and they are around 20-500GB in size each.

I've tried the regkey setting but it doesnt change anything in relation to performance. What can i expect to get in MB/s on my setup?

[vMario156]

- If not, please check that encryption does not affect your performance by setting the custom regkey: at HKLM\Software\Veeam\Veeam Backup and Replication a DisablePublicIPTrafficEncryption DWORD with value = 1

Thanks, the registry key increased our backup to tape Speed from 47MB/s to 150MB/s with a direct connected LTO6 SAS Drive (HP MSL 2024).

Cheers,
Mario

[HHansen]

Thanks for your input Mario,

How are your setup? And what type of backup are you running? Files? Which files, sizes etc?

[tochi]

Hi Guys,

Just installed V.8 for a customer... veeam server is a VM, tape is connected to a physical machine configured as "tape server" Network speed is 1Gbps (private/dedicated). I'm backing up VBK and VIB files. First testing, throughput is 1MB/s... ouch. VB$R is with latest patch 1. Unchecked hardware compression on backup job option, nothing changes, followed the registry change regarding DisableIPTrafficEncryption. It worked, now its throughput is around 11MB/s... still very slow... I'm using LTO4 SAS standalone drive. Please help.

Regards,

[veremin]

What type of tape job you're using? Backup to tape or file to tape one? What you're archiving to tapes - one big file or bunch of small ones? Just to make sure the device is capable of better performance can you try to perform similar operation a with a native tape vendor tool and see what speed you get? Thanks.

[tochi]

Thanks for quick response..
Job Type: File to Tape
Archiving: Backup Repository Folder: .vbk and .vib (around 15 files, 12GB) took 21 minutes to finish.
Tape device is installed in a W2K8 SP2, I don't think native backup to tape is available.
Backup Summary says Network is the bottleneck 99%

[veremin]

I was talking about utility provided by your tape device manufacturer. Anyway, what type of repository you're using? And where is it located? Thanks.

[tochi]

Sorry, but we don't have any utility for that... its a Dell PowerVault LTO4 standalone drive. Backup jobs repository is of course directed to a Disk, the one that is having a slow performance is the "file to tape" job. The Veeam server is installed on a VM, the tape is installed on a tape server through a private lan connection running at 1Gbps. The tape drive is used in Sym BE before Veeam, and its throughput is running around 1.5GB/Min or at least 25MB/s, right now we're seeing 10MB/s on veeam file to tape job.

[veremin]

Feel free to open a ticket with our support team and let them confirm your environment. Thanks.

[tochi]

Thanks,I just did... I'll also update things here when its resolved.

[karthik108417]

Hi,
I faced the same issue after upgrading to v8
I had my backup server running on win2008 R2 connected to IBM tape library (FC) drive and the backup to tape job was really slow running at 24mb/s

After a long troubleshooting, I upgraded my backup server OS to Win2012 R2 and ran the tape job again and now I can see the speed has drastically increased it's running at 90-120 mb/s and now the job finishes within the time.

Regards
Karthikeyan B

[veremin]

Thank you for taking time and sharing the found resolution. By the way, was upgrade the only thing that helped to increase performance? Did support team suggest it for specific reason or it was rather one of recommended troubleshooting steps? Thanks.

[karthik108417]

Hi,

Yes upgrading the operating system to Win2012 R2 only resolved my issue.

I didn't open a case with support for this issue.

Regards
Karthikeyan B

[Dima P.]

Karthikeyan,
Was thinking about your resolution – any chance you have noticed the difference in drivers versions for Win 2008R2 and Win 2012R2 operating systems? Or it was the same driver for both?

[karthik108417]

Hello,

Yes I didn't go with Microsoft Drivers for Tape Library, I downloaded IBM Tape library drivers and installed it.

Regards
Karthikeyan B

[veremin]

Hmm, then, as Dmitry's tried to point out, the drivers' replacement (not the OS upgrade) might have been the reason of increased performance. Thanks.

[gosnells]

finally! Very glad to have found this Veeam reg tweak.

8core Xeon, 32GB mem
2012R2
ML6010 (8Gbps FC)
multi-homed
2GbE LAG to production/mgnt network
10GbE to iSCSI SAN network
10GbE to CIFS backup network

Veeam 8, 8p1 & 8u2
No matter what the source (local Perc array, 16disk iSCSI Target, CIFS dedupe appliance), performance was averaging <35MB/s rendering multiple TB backups unworkable. No joy with consistent acceptable tape library backup performance.

Until now. Reg tweak applied. Decent, consistent performance - 145+MB/s - i.e. what the LTO6 should be capable of.

Thank you.

[habibalby]

[SupportID=01844091]

Hi Guys,
My backup structure is layout and backup to tape between 9MB ~ 30MB...

Physical Sever Dell R720
Windows 2012 R2
8 Cores Dual CPUs
16 GB RAM
Dell Tape Library LTO 2000 with two drives connected via SAS.

Physical server is added to Veeam as Remote Tape Library
MUBBACK001 is just configured as Proxy (perhaps to increase the backup performance)


Veeam Media server is a VM with 20 GB Ram & 8 vCPUs Cores.
4 vNICs;
Production where all servers are connected.
vSphere is Where ESXi & vCenter Management network is connected.
iSCSI on its own network switches separated from LAN traffic to access all iSCSI Repositories.
Another iSCSI NIC is added on the same iSCSI Network Switches to access different repository.

iSCSI & CIFS Network;
CIFS is configured on EMC VNXe 3150 and Shared, added to the Veeam Media Server VM (Repository Server).
CIFS is mapped to the Veeam Media Server.

All backup to disk Jobs goes to the CIFS & Veeam backup files to tape from CIFS share to.

Any recommendations...?

Regards,

[skrause]

Are the volumes you are reading from using windows deduplication?

I found that reading backup files which had been deduped reduced my tape performance by a minimum of 50% versus from non-deduped locations and showed the bottleneck as "proxy".

[habibalby]

hi steve,
that's perhaps he cause of the issue. on VNXe CIFS shares is enabled with Deduplications. I'm going to disable that and will see how it goes.
Thanks,

[habibalby]

Hi Steve,
Tried that, but still the same. Writing backup files to CIFS shares via Windows Map drive is unusable speed at all max 11MB/s. iSCSI LUN added to the Veeam VM Media Server (Which also act as a repository server) end up having speed range between 50~87 MB/s. Backing up from Veeam Job from iSCSI LUN to Tape speed range between 60~116 MB/s.

Added Local Disk of R720 Server (Where Tape Library is connected) as Repository to, backup to disk 55~90 MB/s. Backing up Veeam Job from this Repository to tape speed between 80~126 MB/s.

It seems the CIFS shars that accessed via Map Drive on the Repository and this Map Drive added to veeam as Repository, is where the bottleneck arise.

Any thoughts,..??