Very Poor Backup to Tape Performance with FAS2554.

[WinstonWolf]

Hello Veeam Technicians . I need your Recommendations .

We have to decide to buy an new Backup Storage for Veeam Backup to Disk followed Backup to Tape , because out Backup Storage has poor Performance.

Our Backup Storage at the Moment is an FAS 2554 from Netapp with 48 x 4TB SATA Harddisks and one Controller with an 4 Core CPU.
This Backup Storage is connected with the Backup Server over 10Gbit Ethernet.
The Tape Libary ( HP MSL4048 ) use 2 x HP LTO7 Tape Drives and is connected via SAS with the Backup Server.

On Weekend there runs two Backup to Tape Jobs which produces on Saturday Morning an synthetic Full Backup to Tape, followed by incrementals over the Week.
The Target is that we have at the End of the Week an complete Backup from the whole Week on Tape ( Synthetic Full + Incrementals ).

Our main Problem is, that the Backup to Tape Job on the Weekend, when the sythetic Full is created, needs very long Time.
We have an Processing Rate from only 80 MB/s for the one Job and 50 MB/s for the second Job.
Veeam told us that the Bottlenek is with 99% the Source. That means for me the FAS2554.
Normaly the LTO 7 Drive should have an write Speed from 250 - 300 Mb/s but as you can see on the Screenshot our Backup to Tape has an Processing Rate from 86MB/s .




When we copy an File direct from the Netapp NFS Share to the Backup Proxy then we have an transfer rate from around 600 MB/s . ( Screenshot )



What does Veeam on the Netapp while processing the Synthetic Full ?!?! Why i have only 86 MB/s .

Is it an Storage Problem or an Veeam Problem ? What can ( should ) we change to become better Performance for the Backup to Tape Job . What will be the maximal Write Speed for Backup to Tape with Veeam ? 300 MB/s ?

Thanks for Help .

Michael

[Didi7]

NetApp storage as source always reminds me of poor performance. Being it the source of VMs in a VMware environment or the source of *.vbk backup files as a backup-2-disk environment.

If you have local disks (DirectAttachedStorage) in your backup server, please save *.vbk files there from a VBR backup job for a test and save them to your tape library LTO7 drive as comparison.

As long as your backup server and your local harddisks are quite modern, you should get far better results, I suppose.

We have several NetApp storage devices in our VMware environments, from entry level FAS2040 or FAS2520 to FAS2552 and enterprise level FAS8020 and they all suffer from performance problems using VBR. Using SAS-drives, throughput from a FAS2040 is 70-100MB/s, from a FAS2520 throughput is around 150MB/s, from a FAS8020 throughput is 270MB/s using SAN-transport. This phenomena seems to be common with NetApp storage using Data ONTAP. As far as I know, E-series NetApp storage is not affected. Backup from NetApp storage snapshot with VBR Enterprise + Edition should speed things up further, but that doesn't help you with your transfer speed from NetApp storage to your tape drives.

We get far better throughput results with HPE or EMC storage, if you ask me, even if the storage is years old.

Regards,
Didi7

[WinstonWolf]

Thank you Didi7 ,

I have feared for this answer But now we have no other Options . We take the Backup Files from an FAS8020 and put it on the Backup Target FAS2554 .
I think if this is true what u say , then i find it very sad that Veeam gives no concret recommendations for Storage Hardware.
Do you think this is an Problem with NFS himself ? Or rather the Hardware ?

Thanks

[Didi7]

Hello WW,

if you use NFS as storage protocol on your FAS8020, I suppose you also use Direct NFS Access in Veeam, otherwise you would only have the option to use NBD as transport protocol, which is very slow using a FAS8020, not more than 100MB/s on SAS and even slower on SATA using 10GBits/s interface for the console.

What are your transfer speeds using VBR (which version you use?), when you backup a big VM (500 to 1000GB) to your FAS2554 backup target and if you do the same to the local disks of your VBR-server?

I am really interested to know the transfer speed, as we have a similar equipment in our datacenter, where we intend to introduce VBR as well.

Regards,
Didi7

[stewsie]

I also get very slow tape performance but using EMC VNX 5300 with CIFS as the source repositories. Most of the tape jobs that use the same tape infrastructure as you run (although connected to the library using FC) at around 30 MB/s hence some of the larger jobs can take around 56 hours to complete. Far from ideal at all.

The source is always shown as the bottleneck. Any suggestions apart from getting rid of CIFS

[Didi7]

Quote: The source is always shown as the bottleneck

Answer: Typical for NetApp storage, in our case VBR backup jobs from VMware Environments (with NetApp storage) to the backup repository (local disks on the backup server)

We have one VNX5300 connected via FC to one of our VMware-Cluster. Using VBR backup jobs, we get more than 200MB/s with HOTADD-proxies, I never configured direct access into the storage. I did this just for comparison. If you only get 30MB/s with CIFS from your VNX5300, do you use 1GBit/s and SATA?

Source bottleneck might not always be bad. We have transfer speed of more than 600MB/s over 10GBit/s iSCSI-network using VADP SAN-transport from HPE MSA storage to backup repositories

As destination repositories for VBR backup Jobs, we only use DirectAttachedStorage, the tape server is on the same server, as the VBR console, so throughput is at maximum and the bottleneck is the Target.

You might have better deduplication in NetApp storage but VBR deduplicates backup data already very good before it's written to the backup repository. We prefer standard DAS with NTFS partitions. In our Environments this is sufficient fast enough for backups to save and fast enough for tape-infrastructure to feed.

[pkelly_sts]

I've been meaning to find time to go over our tape performance (for full backups) as we're also getting jobs running for much longer than I'd expect, knowing our hardware.

For example it's regularly taking 10hrs to complete a 3.5Tb tape synth full from 24-disk JBOD/Direct attach (purchased this year) to LTO-6 SAS on the same physical server but I know I'll need to gather lots of information to make my case. I suspect other jobs (such as copy jobs etc.) might be interfering but I'm not totally convinced that they should be.

[larry]

I have a FA2554 to get full speed for tape backups I use a disk to cache the job.
see tape-f29/tape-clone-possible-t43990.html#p259138

I had other reasons for using the disk cache for this, mainly I need two copies of the tape at times. Founds lots of other good things doing a disk cache, especially speed. When we then switched to LTO 7 speed became even more important. Jobs all run at tape speed. (150 lto 6, 300 lto7 )

I also use Veeam created san snap shots as source and the disk cache allows for other jobs to backup the same vm's at the same time. You just cant overlap the first few seconds that the snapshot takes.

Restores are quick this way.

[Didi7]

Very interesting larry! Might be something I would like to use in a simalar way in our Datacenter soon.

Regards,
Didi7

[mkretzer]

We also suffer bad tape performance. We get ~90 MB/s to a LTO 7 drive. Source was a Hitachi HUS110 at first so we thought it was because of the rather old storage. But now we use a Hitach G200 with 128 4 TB disks which can easily create synthetics at FC speed of up to 1,6 GB/s.
Still, tape goes at ~ 90 MB/s most of the time. Why is there no real multi-streaming??

[chris.childerhose]

I have a FA2554 to get full speed for tape backups I use a disk to cache the job.
see tape-f29/tape-clone-possible-t43990.html#p259138

I had other reasons for using the disk cache for this, mainly I need two copies of the tape at times. Founds lots of other good things doing a disk cache, especially speed. When we then switched to LTO 7 speed became even more important. Jobs all run at tape speed. (150 lto 6, 300 lto7 )

I also use Veeam created san snap shots as source and the disk cache allows for other jobs to backup the same vm's at the same time. You just cant overlap the first few seconds that the snapshot takes.

Restores are quick this way.

How do you go about doing this? Is it a backup copy job to the location and then the tape job reads that? Just curious as I have a Data Domain DD2500 that I am using for my main target using DDBoost but tape backups are still dreadful.

[larry]

I use backupcopy jobs or a straight backup to the cache disk, then files to tape.

[chris.childerhose]

I use backupcopy jobs or a straight backup to the cache disk, then files to tape.

Thanks for that. Something to put in my new design and backup server going in Friday.