Backup to Tape taking too long

Everything about backing up to tape

Backup to Tape taking too long

Veeam Logoby eversys » Thu Jan 08, 2015 1:34 pm

We have setup a tape job to backup local file shares to a Dell PowerVault TL2000 tape library (IBM 3572-TL C.30). There is around 900GB of data and this used to take around 6-7 hours. All of a sudden, the job time has increased incredibly to over 50 hours - for exactly the same data size:
Image
Here the job in action:
Image
Image
Any ideas what the problem could be? As a precaution, we have restarted the Veeam server and updated to B&R 8.0.0.917.

Thanks in advance for any feedback.
eversys
Enthusiast
 
Posts: 36
Liked: 5 times
Joined: Thu Dec 11, 2014 1:09 pm
Full Name: EVERSYS

Re: Backup to Tape taking too long

Veeam Logoby Dima P. » Thu Jan 08, 2015 7:21 pm

Hello,
There is around 900GB of data and this used to take around 6-7 hours.

Anything changed in the environment (might be throttling or routing rules)? I see the bottleneck is network on the screenshot - can you check the performance by copying files from NAS via regular file browser? In addition, try to disable hardware compression in the file to tape job properties.
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Backup to Tape taking too long

Veeam Logoby seadave » Fri Jan 09, 2015 5:49 am

I have the same setup. If you are using the iSCSI to SAS bridge the max speed possible is 50MB/s. You are only getting 8! If you are direct connected to the tape server with a HBA, you should be able to get ~100MB/s if the storage can keep up with that rate. If you are using the iSCSI to SAS bridge make sure you have a dedicated NIC on the Veeam server that is directly connected to it and both ends are configured for jumbo frames. If you are going through a switch that has to be enabled for 9000MTU also.

Your 6-7 hour estimate indicates you were running at 50MB/s previously so I would check your networking between the bridge and the Veeam server. If you don't have that, maybe your Veeam server storage is being tasked by some other I/O intensive operation while the tape job is running.
seadave
Enthusiast
 
Posts: 35
Liked: 6 times
Joined: Thu Oct 30, 2014 12:43 am

Re: Backup to Tape taking too long

Veeam Logoby eversys » Fri Jan 09, 2015 2:41 pm

The Veeam server is directly attached to the autoloader with a PCI Express SAS card. Local files on the Veeam server are being directly backed up to the autoloader. The server also has 2 local NICs along with a 4 port NIC, all ports attached to different switches. However, the files are being processed locally and not over the network. We have another job defined, which backs up a seperate VMware host - here of course, the NICs and switches would play a part.

We are not aware of any other processes or I/O intensive operations at time of the backup - starts late evening.

Any further ideas?
eversys
Enthusiast
 
Posts: 36
Liked: 5 times
Joined: Thu Dec 11, 2014 1:09 pm
Full Name: EVERSYS

Re: Backup to Tape taking too long

Veeam Logoby seadave » Fri Jan 09, 2015 10:10 pm

eversys wrote:The Veeam server is directly attached to the autoloader with a PCI Express SAS card. Local files on the Veeam server are being directly backed up to the autoloader. The server also has 2 local NICs along with a 4 port NIC, all ports attached to different switches. However, the files are being processed locally and not over the network. We have another job defined, which backs up a seperate VMware host - here of course, the NICs and switches would play a part.

We are not aware of any other processes or I/O intensive operations at time of the backup - starts late evening.

Any further ideas?

Maybe create a file copy job as a test? Pick a VBK of sufficient size and see if you can get greater than 8Mb/s. If not then perhaps there is something wrong between the SAS card and the Tape drive. Try different SAS Cables or shut everything down, boot tape drive and after it initializes then boot the server to see if that clears the issue. You might also run a tool such as IOMeter against your storage target to see what kind of IOPS you can get. I get ~150MB/s to 800MB/s with 16 x 4TB drives in RAID6 on a 730XD depending on the block size and IO randomness of the test. Maybe you have a degraded RAID set that is causing the problem? If you are running RAID5 and you lost one of your drives, if that is a 1TB or larger drive it can take a few days to restore parity depending on your hardware config.

Interested to know what is the source of the degradation once you locate it.
seadave
Enthusiast
 
Posts: 35
Liked: 6 times
Joined: Thu Oct 30, 2014 12:43 am

Re: Backup to Tape taking too long

Veeam Logoby Dima P. » Mon Jan 12, 2015 3:32 pm

To get a fair performance output I would initiate a file write operation with a native tape vendor tool (one for a big file and separate for a huge pack of small ones). In addition, I would check the original file to tape with compression off
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Backup to Tape taking too long

Veeam Logoby eversys » Tue Jan 13, 2015 9:04 am

I don't think it's a hardware problem for the following reason. We have 2 disk2tape jobs defined. We backup our veeam repository, which contains our backed up VMs, to tape. Here we get a processing rate of over 154MB/s:
Image

However, when backing up local files to tape, we get a processing rate of only 9 MB/s and the bottleneck is shown as "Source":
Image

We have also noticed a restore problem with this job. Please see tape-f29/files-are-not-being-correctly-backed-up-t25457.html. Something is going terribly wrong here :?
eversys
Enthusiast
 
Posts: 36
Liked: 5 times
Joined: Thu Dec 11, 2014 1:09 pm
Full Name: EVERSYS


Return to Tape



Who is online

Users browsing this forum: No registered users and 3 guests