Slow Tape Backup Performance

[BackupTest90]

Hey Guys,
we have a really slow backup tapeperformance in our infrastructure and don't know where is this problem. Hope you have an idea.

Infrastructure:
2 HW-Proxys with (2x8Core, 128GB RAM, 2x8Gbit FC, 2x10Gbit/ Ethernet, w2k19)
1 Tape-proxys (2x8Core, 128GB RAM, 2x8Gbit FC, 2x10Gbit/ Ethernet,w2k19)
* The proxy has access to 3 LTO6-Drives in an HPE MSL6480 Lib.

Productivestorage: 3PAR 7400 with round about 150HDDs (10k/7,2k)
Backupstorage: MSA2040 with round about 90HDDs (only 10k)

The zoning is so that the hw-proxy1 reads the date from the FC-Storagevolume (ReFS) and transfer it to the tape-proxy via 10G-Ethernet. They receive the data and sends it to the tapelib.
When I check the HDD-performancemonitor on proxy 1 then shows there a readperformance for the .vbk files from approx. with 10MB/s - 15MB/s , the MSA Storage shows the same performance. Sometimes it happens that they increase the readperformance to up to 300MB/s.

On the msa are no other backupjobs or somethin running. The MSA is used exclusively by the tape job.

The GFS Job said the same, that the Source is the bottleneck.
Busy: Source 100% > Proxy 11% > Network 0% > Target 0%

Do you have any ideas to optimize the performance?

Regards,

[Dima P.]

Hello Martin,

1. Can you please clarify if you have deduplication enabled on the ReFS repository?
2. Can you run a test and perform a regular file to tape job from the non-ReFS target to compare the results?

Thank you!

[BackupTest90]

thanks for this, i will test it.

UPDATE: Dedup are not as role on the server installed. The second point can we first check, after the tape backup.

[BackupTest90]

short update:
I tested a "File to tape" Job.

File to tape - ReFs Volume - Processing Rate 285MB/s - Bottleneck: Target
File to tape - NTFS Volume - Processing Rate 282MB/s - Bottleneck: Target

These are performancevalues what I expect for and drive.

Do you have any ideas?

[Dima P.]

Thanks Martin! Forgot to ask what Veeam B&R version you've been testing? Cheers!

[BackupTest90]

Version 9.5 U4, we cannot upgrade to Version10 at the moment.. We have a few old host with ESX 5.x that we backup.

[Shmythe]

We have the same thing, did you find the issue?

[Dima P.]

Ryan,

Serval performance improvements were added to v11, so I'd recommend you to test the latest version and let us know if you still have any performance issues. Cheers!

[brajchel]

I'm not happy with Veeam tape speeds. We did notice some improvement after checking all links and ensuring all are 10Gb with MTU 9000. (proxy had MTU set to standard 1500) Once the setting was updated the backup speed went up, but nowhere near what 2 LTO7 tape drives support. We should be getting ~600MB/s (300MB/s per drive) combined instead we are around 288MB/s. Tape is slow when backing up small files, but with large files such as Veeam backups it should be at steaming close to max speed.
My suggestion is to draw a data flow diagram with every NIC, HBA, and Switch listed and check if all ports are set to MAX speed, Jumbo Frame and use Tape/Library tools to check for drive performance.

[Dima P.]

Bart,

I'm not happy with Veeam tape speeds.

Can you please share:

1. Your Veeam B&R version.
2. Your tape drive and library model (I got only that you have LTO7 generation).
3. Your average throughput for backup to tape job (I assume you are using it to process backups from repository).
4. Where tape server (aka tape proxy component is installed)?
5. How tape device is connected to the source repository?
6. What is your source repository?
7. In case the repository is SMB/NFS/DataDomain/HPE StoreOnce where gateway server component is installed.

Thank you in advance!