Discussions specific to tape backups
veremin
Product Manager
Posts: 16778
Liked: 1405 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by veremin » Nov 13, 2014 3:42 pm

Update: Support team has created a KB article regarding this behaviour. Everyone experiencing this issue, please, check it out and download the private fix provided there. Thanks.

Maurice
Service Provider
Posts: 25
Liked: 2 times
Joined: May 26, 2014 7:59 am
Full Name: Maurice Galicic
Location: Münster, Germany
Contact:

Re: Remote tape library. Slow performance

Post by Maurice » Nov 14, 2014 8:00 am 1 person likes this post

The Path in the KB is a bit wrong.
It should be "C:\Program Files\Veeam\Backup and Replication\Backup" right?

Thanks for that fix. It worked and the Workaround with the Scheduling that was sended by the Support Team worked, too.

Great Job for the v8 by the way.
The new Features are great and i hope that the first few bugs are gone in no time. :)

Greetings
Maurice

veremin
Product Manager
Posts: 16778
Liked: 1405 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by veremin » Nov 14, 2014 8:47 am

You're right, indeed. The correct path is C:\Program Files\Veeam\Backup and Replication\Backup. I've asked already support team to update the KB.

Yes, we're trying to addressing any issues raised by users in timely fashion. Thanks be to you for drawing our attention to this or that particular problem; highly appreciated.

nreutemann
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann
Contact:

Re: Remote tape library. Slow performance

Post by nreutemann » Nov 14, 2014 6:27 pm

I create a new job, wich process the backup of only one VM, a new "backup to tape job".
Under "Media Pool", go to "Schedule" into the "media pool for full backups" and set it to "Monthly" on "Last" "Sunday" of "December" (the only month selected).
When I run the job, the throughput is 25Mb/s, the bottleneck is network.

So, I didnt see any change.

I do something wrong?

Best regards!

Our case is ID#00674598

Dima P.
Product Manager
Posts: 10426
Liked: 844 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Remote tape library. Slow performance

Post by Dima P. » Nov 17, 2014 12:44 pm

Nicolas,

Please, continue investigation with Veeam Support team - it could be configuration specific bug and requires deep VBR log and infrastructure analysis for correct troubleshooting. Thank you.

daxb
Lurker
Posts: 1
Liked: 1 time
Joined: Jan 21, 2014 8:59 pm
Full Name: Dax
Contact:

Re: Remote tape library. Slow performance

Post by daxb » Nov 26, 2014 11:06 pm 1 person likes this post

Gotta say I'm pretty disappointed with Veeam support. I logged Case # 00679800 on the 16th of November as tape backups were stuck at 20MB/s following the upgrade and still haven't had anything useful from support. I found this post which appears to have resolved the issue though: http://www.v-strange.de/index.php/2-unc ... o-veeam-v8

So adding the reg key HKLM\Software\Veeam\Veeam Backup and Replication\DisablePublicIPTrafficEncryption = 1 seems to fix my issue.

Worth trying if you are experiencing slow tape transfer following installation of the KB1949 patch.

Would have saved me a lot of time if support had been as responsive as the author of the blog experienced.

Dima P.
Product Manager
Posts: 10426
Liked: 844 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Remote tape library. Slow performance

Post by Dima P. » Nov 27, 2014 2:56 pm

Hello Dax,

I am sorry that you case resolution took longer than expected. The issue you experienced is now fully documented and applied for a fix in upcoming patch, so thank you opening a support case and providing all the troubleshooting information - it helped a lot in finding the root of this problem. Thanks again and sorry for any inconvenience caused.

HHansen
Influencer
Posts: 13
Liked: never
Joined: Jun 03, 2014 10:53 am
Full Name: Henrik Hansen
Contact:

[MERGED] Tape performance, what to expect?

Post by HHansen » Nov 27, 2014 3:31 pm

Hi,

I was wondering what kind of perfomance you other guys have on your tape setup with B&R 8.

I'm currently looking into replacing our HP data protector with Veeam since we already pay for it.

My setup is:

One physical server (2xCPU, 48GB RAM)
2xHP D2600 SAS disk enclosures, with 44TB storage (22 in raid 0+1), directly attached to the physical server
1x MSL 4048 library with 2x LTO5 drives, directly attached to the physical server, also using SAS

All backups are done to disk each night, and kept there for 1 week for fast restore.

Backup to tape is done on a daily basis, by backing up the data on the disks and setting retention.

With HP data protector, i get around 100-120MB/s to tape.

With Veeam i get around 40MB/s.

LTO5 is supposed to run at 140MB/s as far as i know.

Is this normal performance for Veeam or is there something wrong? Veeam says target is bottleneck.

Thanks :wink:

Dima P.
Product Manager
Posts: 10426
Liked: 844 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Remote tape library. Slow performance

Post by Dima P. » Nov 27, 2014 3:58 pm

Henrik,
We have at least two bugs that may affect performance of your existing backup to tape jobs:

- If you using the backup to tape with .vrb files please refer to this KB article Backup to tape jobs fails when the source contains .vrb files
- If not, please check that encryption does not affect your performance by setting the custom regkey: at HKLM\Software\Veeam\Veeam Backup and Replication a DisablePublicIPTrafficEncryption DWORD with value = 1

HHansen
Influencer
Posts: 13
Liked: never
Joined: Jun 03, 2014 10:53 am
Full Name: Henrik Hansen
Contact:

Re: Remote tape library. Slow performance

Post by HHansen » Nov 28, 2014 12:58 pm

d.popov wrote:Henrik,
We have at least two bugs that may affect performance of your existing backup to tape jobs:

- If you using the backup to tape with .vrb files please refer to this KB article Backup to tape jobs fails when the source contains .vrb files
- If not, please check that encryption does not affect your performance by setting the custom regkey: at HKLM\Software\Veeam\Veeam Backup and Replication a DisablePublicIPTrafficEncryption DWORD with value = 1
The files im currently testing with is .bak files, and they are around 20-500GB in size each.

I've tried the regkey setting but it doesnt change anything in relation to performance. What can i expect to get in MB/s on my setup?

vMario156
Novice
Posts: 4
Liked: never
Joined: Nov 28, 2014 3:56 pm
Full Name: Mario Mack
Contact:

Re: Remote tape library. Slow performance

Post by vMario156 » Nov 28, 2014 3:59 pm

d.popov wrote: - If not, please check that encryption does not affect your performance by setting the custom regkey: at HKLM\Software\Veeam\Veeam Backup and Replication a DisablePublicIPTrafficEncryption DWORD with value = 1
Thanks, the registry key increased our backup to tape Speed from 47MB/s to 150MB/s with a direct connected LTO6 SAS Drive (HP MSL 2024).

Cheers,
Mario

HHansen
Influencer
Posts: 13
Liked: never
Joined: Jun 03, 2014 10:53 am
Full Name: Henrik Hansen
Contact:

Re: Remote tape library. Slow performance

Post by HHansen » Nov 28, 2014 5:58 pm

Thanks for your input Mario,

How are your setup? And what type of backup are you running? Files? Which files, sizes etc? :)

tochi
Influencer
Posts: 14
Liked: never
Joined: Jan 26, 2015 10:05 am
Full Name: Renen Quinto
Contact:

Re: Remote tape library. Slow performance

Post by tochi » Jan 26, 2015 10:19 am

Hi Guys,

Just installed V.8 for a customer... veeam server is a VM, tape is connected to a physical machine configured as "tape server" Network speed is 1Gbps (private/dedicated). I'm backing up VBK and VIB files. First testing, throughput is 1MB/s... ouch. VB$R is with latest patch 1. Unchecked hardware compression on backup job option, nothing changes, followed the registry change regarding DisableIPTrafficEncryption. It worked, now its throughput is around 11MB/s... still very slow... I'm using LTO4 SAS standalone drive. Please help.

Regards,

veremin
Product Manager
Posts: 16778
Liked: 1405 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by veremin » Jan 26, 2015 10:59 am

What type of tape job you're using? Backup to tape or file to tape one? What you're archiving to tapes - one big file or bunch of small ones? Just to make sure the device is capable of better performance can you try to perform similar operation a with a native tape vendor tool and see what speed you get? Thanks.

tochi
Influencer
Posts: 14
Liked: never
Joined: Jan 26, 2015 10:05 am
Full Name: Renen Quinto
Contact:

Re: Remote tape library. Slow performance

Post by tochi » Jan 26, 2015 11:35 am

Thanks for quick response..
Job Type: File to Tape
Archiving: Backup Repository Folder: .vbk and .vib (around 15 files, 12GB) took 21 minutes to finish.
Tape device is installed in a W2K8 SP2, I don't think native backup to tape is available.
Backup Summary says Network is the bottleneck 99%

veremin
Product Manager
Posts: 16778
Liked: 1405 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by veremin » Jan 26, 2015 12:02 pm

I was talking about utility provided by your tape device manufacturer. Anyway, what type of repository you're using? And where is it located? Thanks.

tochi
Influencer
Posts: 14
Liked: never
Joined: Jan 26, 2015 10:05 am
Full Name: Renen Quinto
Contact:

Re: Remote tape library. Slow performance

Post by tochi » Jan 27, 2015 12:23 am

Sorry, but we don't have any utility for that... its a Dell PowerVault LTO4 standalone drive. Backup jobs repository is of course directed to a Disk, the one that is having a slow performance is the "file to tape" job. The Veeam server is installed on a VM, the tape is installed on a tape server through a private lan connection running at 1Gbps. The tape drive is used in Sym BE before Veeam, and its throughput is running around 1.5GB/Min or at least 25MB/s, right now we're seeing 10MB/s on veeam file to tape job.

veremin
Product Manager
Posts: 16778
Liked: 1405 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by veremin » Jan 27, 2015 9:27 am

Feel free to open a ticket with our support team and let them confirm your environment. Thanks.

tochi
Influencer
Posts: 14
Liked: never
Joined: Jan 26, 2015 10:05 am
Full Name: Renen Quinto
Contact:

Re: Remote tape library. Slow performance

Post by tochi » Jan 27, 2015 12:02 pm

Thanks,I just did... I'll also update things here when its resolved.

karthik108417
Novice
Posts: 8
Liked: 2 times
Joined: Mar 24, 2013 1:45 pm
Full Name: Karthikeyan Balasubramani
Location: Abu Dhabi
Contact:

Re: Remote tape library. Slow performance

Post by karthik108417 » Feb 02, 2015 7:14 pm

Hi,
I faced the same issue after upgrading to v8
I had my backup server running on win2008 R2 connected to IBM tape library (FC) drive and the backup to tape job was really slow running at 24mb/s

After a long troubleshooting, I upgraded my backup server OS to Win2012 R2 and ran the tape job again and now I can see the speed has drastically increased it's running at 90-120 mb/s and now the job finishes within the time.

Regards
Karthikeyan B

veremin
Product Manager
Posts: 16778
Liked: 1405 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by veremin » Feb 03, 2015 9:37 am

Thank you for taking time and sharing the found resolution. By the way, was upgrade the only thing that helped to increase performance? Did support team suggest it for specific reason or it was rather one of recommended troubleshooting steps? Thanks.

karthik108417
Novice
Posts: 8
Liked: 2 times
Joined: Mar 24, 2013 1:45 pm
Full Name: Karthikeyan Balasubramani
Location: Abu Dhabi
Contact:

Re: Remote tape library. Slow performance

Post by karthik108417 » Feb 09, 2015 6:07 am

Hi,

Yes upgrading the operating system to Win2012 R2 only resolved my issue.

I didn't open a case with support for this issue.

Regards
Karthikeyan B

Dima P.
Product Manager
Posts: 10426
Liked: 844 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Remote tape library. Slow performance

Post by Dima P. » Feb 11, 2015 8:35 am

Karthikeyan,
Was thinking about your resolution – any chance you have noticed the difference in drivers versions for Win 2008R2 and Win 2012R2 operating systems? Or it was the same driver for both?

karthik108417
Novice
Posts: 8
Liked: 2 times
Joined: Mar 24, 2013 1:45 pm
Full Name: Karthikeyan Balasubramani
Location: Abu Dhabi
Contact:

Re: Remote tape library. Slow performance

Post by karthik108417 » Feb 16, 2015 4:49 am

Hello,

Yes I didn't go with Microsoft Drivers for Tape Library, I downloaded IBM Tape library drivers and installed it.

Regards
Karthikeyan B

veremin
Product Manager
Posts: 16778
Liked: 1405 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Remote tape library. Slow performance

Post by veremin » Feb 16, 2015 10:26 am

Hmm, then, as Dmitry's tried to point out, the drivers' replacement (not the OS upgrade) might have been the reason of increased performance. Thanks.

gosnells
Influencer
Posts: 16
Liked: 1 time
Joined: Jul 23, 2012 4:28 am
Full Name: Mike Smith
Contact:

Re: Remote tape library. Slow performance

Post by gosnells » Jun 12, 2015 3:39 pm

finally! Very glad to have found this Veeam reg tweak.

8core Xeon, 32GB mem
2012R2
ML6010 (8Gbps FC)
multi-homed
2GbE LAG to production/mgnt network
10GbE to iSCSI SAN network
10GbE to CIFS backup network

Veeam 8, 8p1 & 8u2
No matter what the source (local Perc array, 16disk iSCSI Target, CIFS dedupe appliance), performance was averaging <35MB/s rendering multiple TB backups unworkable. No joy with consistent acceptable tape library backup performance.

Until now. Reg tweak applied. Decent, consistent performance - 145+MB/s - i.e. what the LTO6 should be capable of.

Thank you.

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Remote tape library. Slow performance

Post by habibalby » Jul 08, 2016 1:15 pm

[SupportID=01844091]

Hi Guys,
My backup structure is layout and backup to tape between 9MB ~ 30MB...

Physical Sever Dell R720
Windows 2012 R2
8 Cores Dual CPUs
16 GB RAM
Dell Tape Library LTO 2000 with two drives connected via SAS.

Physical server is added to Veeam as Remote Tape Library
MUBBACK001 is just configured as Proxy (perhaps to increase the backup performance)


Veeam Media server is a VM with 20 GB Ram & 8 vCPUs Cores.
4 vNICs;
Production where all servers are connected.
vSphere is Where ESXi & vCenter Management network is connected.
iSCSI on its own network switches separated from LAN traffic to access all iSCSI Repositories.
Another iSCSI NIC is added on the same iSCSI Network Switches to access different repository.

iSCSI & CIFS Network;
CIFS is configured on EMC VNXe 3150 and Shared, added to the Veeam Media Server VM (Repository Server).
CIFS is mapped to the Veeam Media Server.

All backup to disk Jobs goes to the CIFS & Veeam backup files to tape from CIFS share to.

Any recommendations...?

Regards,

skrause
Expert
Posts: 431
Liked: 90 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Remote tape library. Slow performance

Post by skrause » Jul 08, 2016 6:35 pm

Are the volumes you are reading from using windows deduplication?

I found that reading backup files which had been deduped reduced my tape performance by a minimum of 50% versus from non-deduped locations and showed the bottleneck as "proxy".
Steve Krause
Veeam Certified Architect

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Remote tape library. Slow performance

Post by habibalby » Jul 08, 2016 9:16 pm

hi steve,
that's perhaps he cause of the issue. on VNXe CIFS shares is enabled with Deduplications. I'm going to disable that and will see how it goes.
Thanks,

habibalby
Expert
Posts: 379
Liked: 23 times
Joined: Jul 18, 2011 9:30 am
Full Name: Hussain Al Sayed
Location: Bahrain
Contact:

Re: Remote tape library. Slow performance

Post by habibalby » Jul 09, 2016 11:27 am

Hi Steve,
Tried that, but still the same. Writing backup files to CIFS shares via Windows Map drive is unusable speed at all max 11MB/s. iSCSI LUN added to the Veeam VM Media Server (Which also act as a repository server) end up having speed range between 50~87 MB/s. Backing up from Veeam Job from iSCSI LUN to Tape speed range between 60~116 MB/s.

Added Local Disk of R720 Server (Where Tape Library is connected) as Repository to, backup to disk 55~90 MB/s. Backing up Veeam Job from this Repository to tape speed between 80~126 MB/s.

It seems the CIFS shars that accessed via Map Drive on the Repository and this Map Drive added to veeam as Repository, is where the bottleneck arise.

Any thoughts,..??

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests