Backup to Tape taking too long

[eversys]

We have setup a tape job to backup local file shares to a Dell PowerVault TL2000 tape library (IBM 3572-TL C.30). There is around 900GB of data and this used to take around 6-7 hours. All of a sudden, the job time has increased incredibly to over 50 hours - for exactly the same data size:

Here the job in action:


Any ideas what the problem could be? As a precaution, we have restarted the Veeam server and updated to B&R 8.0.0.917.

Thanks in advance for any feedback.

[Dima P.]

Hello,

There is around 900GB of data and this used to take around 6-7 hours.

Anything changed in the environment (might be throttling or routing rules)? I see the bottleneck is network on the screenshot - can you check the performance by copying files from NAS via regular file browser? In addition, try to disable hardware compression in the file to tape job properties.

[seadave]

I have the same setup. If you are using the iSCSI to SAS bridge the max speed possible is 50MB/s. You are only getting 8! If you are direct connected to the tape server with a HBA, you should be able to get ~100MB/s if the storage can keep up with that rate. If you are using the iSCSI to SAS bridge make sure you have a dedicated NIC on the Veeam server that is directly connected to it and both ends are configured for jumbo frames. If you are going through a switch that has to be enabled for 9000MTU also.

Your 6-7 hour estimate indicates you were running at 50MB/s previously so I would check your networking between the bridge and the Veeam server. If you don't have that, maybe your Veeam server storage is being tasked by some other I/O intensive operation while the tape job is running.

[eversys]

The Veeam server is directly attached to the autoloader with a PCI Express SAS card. Local files on the Veeam server are being directly backed up to the autoloader. The server also has 2 local NICs along with a 4 port NIC, all ports attached to different switches. However, the files are being processed locally and not over the network. We have another job defined, which backs up a seperate VMware host - here of course, the NICs and switches would play a part.

We are not aware of any other processes or I/O intensive operations at time of the backup - starts late evening.

Any further ideas?

[seadave]

The Veeam server is directly attached to the autoloader with a PCI Express SAS card. Local files on the Veeam server are being directly backed up to the autoloader. The server also has 2 local NICs along with a 4 port NIC, all ports attached to different switches. However, the files are being processed locally and not over the network. We have another job defined, which backs up a seperate VMware host - here of course, the NICs and switches would play a part.

We are not aware of any other processes or I/O intensive operations at time of the backup - starts late evening.

Any further ideas?

Maybe create a file copy job as a test? Pick a VBK of sufficient size and see if you can get greater than 8Mb/s. If not then perhaps there is something wrong between the SAS card and the Tape drive. Try different SAS Cables or shut everything down, boot tape drive and after it initializes then boot the server to see if that clears the issue. You might also run a tool such as IOMeter against your storage target to see what kind of IOPS you can get. I get ~150MB/s to 800MB/s with 16 x 4TB drives in RAID6 on a 730XD depending on the block size and IO randomness of the test. Maybe you have a degraded RAID set that is causing the problem? If you are running RAID5 and you lost one of your drives, if that is a 1TB or larger drive it can take a few days to restore parity depending on your hardware config.

Interested to know what is the source of the degradation once you locate it.

[Dima P.]

To get a fair performance output I would initiate a file write operation with a native tape vendor tool (one for a big file and separate for a huge pack of small ones). In addition, I would check the original file to tape with compression off

[eversys]

I don't think it's a hardware problem for the following reason. We have 2 disk2tape jobs defined. We backup our veeam repository, which contains our backed up VMs, to tape. Here we get a processing rate of over 154MB/s:


However, when backing up local files to tape, we get a processing rate of only 9 MB/s and the bottleneck is shown as "Source":


We have also noticed a restore problem with this job. Please see http://forums.veeam.com/tape-f29/files- ... 25457.html. Something is going terribly wrong here