Very poor Read/Write Performance to an 10Gbit Netapp FAS

[WinstonWolf]

Hello ,
We have an Performance Problem when take an Backup , Merge or create an virtual Full Backup to Tape .

Here is the environment :

-Backup Server ( Proxy ) : W2008R2 Server ( HP DL 380 G7 ) 192 GB RAM , 2 CPUs , 2x300 HDDs RAID 1 , H222 SAS Controller to MSL4048 HP Tape Libary
-10 Gbit Network to the Productiv Storage ( Netapp FAS8020 Metrocluster ) , to the Backup Server and to the Backup Storage ( Netapp FAS 2554 )
-Netapp Backup Storage ( FAS 2554 ) , 2 Aggregates, each with 22x 4TB SATA ( 7200 ) HDDs , on this Aggregats there are on every Aggregat 5 ISCSI Volumes . This Volumes are Drives ( Repositorys ) on the Backup Server.
Netapp Productiv Storage ( FAS 8020 ) , The Veeam Backup Server is connected via 10Gbit Network to the NFS Volumes and uses the Storage Snapshot Copy Funktion to Backup the VMs to the Backup Storage
Tape Libary : HP MSL 4048 with an HP StorageWorks LTO5 Tape Drive , Connected via SAS to the Backup Server

The newest Drivers and Firmwares for Tape Drive , Tape Libary and the Controllers on the Server are installed .


And now our Problem :

A normal Full Backup Job only backup with 144 MB/s to the backup Storage . An Merge Job needs needs very long time and the Creation of an virtual Full to Tape needs for 10TB 41 Hours with 70 MB/s on an LTO5 Tape Drive via SAS.

Here some Screenshots :

[WinstonWolf]

Here some Screenshots :

Here you can see the Screenshot from the last Weekend ( Creation of the virtual Full to Tape )

As you can see , the Tape Drive is only with 24% loaded. ( where i loose my Performance ? ) An LTO 5 should write with 140 MB/s
The speed becomes from Weekend to weekend slower .



Here you can see that the Tape Drive cant be the Problem, because under the Week only the newest .vib File is only copied from the Backup Storage to the Tape Drive and here i have the Full Performance from the LTO ( 140 MB/s )
As you cann see the load of the Target is now 99% . This is ok for me.




Here you can see the Performance of an Active Full Backup from the Productiv Storage to the Backup Storage , means only Write on the Backup Storage



What we do till now :

- Test the Disk Read/Write Performance from the Veeam Backup Server to the Backup Storage with the "Diskspd" Tool , here the Result :

I think the results are very poor .

- We test the Speed for writing a File to the Tape Drive with an Copy to Tape job with an local File to Tape , the Copy Job runs with 140 MB/s , that shows that the Tape Drive is not the Problem .


Have someone the same Problem ?
What can be the Reason for this behavior ?

[WinstonWolf]

Just at the Moment i read an Article about Netapp and Veeam with ISCSI and MPIO .
On my Backup Server MPIO is not installed . Can bring MPIO a better Performance in my Case ?

[Gostev]

It's a possibility.

[WinstonWolf]

Hello again ,

The MPIO brings no success .
I have again very long "Merge" Times for an Job . and also the producing of the virtual Full Backup to Tape needs very long Time .
What can be the Bottleneck ?
Support Case Number : 01801940
For Example :


And here you can see again that the Backup to Disk Job also needs a long Time . The Tape Drive is an LTO 5 , it can be up to 140 MB/s for Backing up speed .
You can see that the Processing Full Backup needs so long Time . What happens there ?


Very Strange