Moving vbk to tape via Arcserve very slow

[dekkar]

Hi all..... Just wondering if anyone else is doing something similar.

I backup our main vbk images to tape nightly for offsite storage.....

The problem is it seems arcserve really struggles with these files. We would normally get around 2-4GB/minute when backing up exchange, SQL database or any other type of file.

When arcserve backs up vbk files, they start off fast like the rest, but within a few mins end up below 1GB/minute. Sometimes going as low as 500MB.

The problem with this backing up our 600Gb worth of vbk files ends up taking over 10 hours, when it should take around 5.


I can confirm it isnt the tape drive, network, server, or storage.... As backing up any other file is really fast. It has something to do with arcserve and the files themselves... Possibly the size or compression.

Anyone able to shed any light?

Thanks,
Nathan

[tsightler]

Just a guess, but I's suspect it has something to do with the fact that the VBK file is already compressed, while the other files you are backing up are highly compressible. The actual transfer rate to the tape is likely to be similar but the compression of the data makes the others seems significantly faster. Do you have any other very large, compressed files that you could try. I certainly can't think of anything else that would impact this. Are the files storage locally on the Arcserver system, or some other server? Are you using Arcserve software compression?

[dekkar]

Hi, I was kinda thinking it could be that... I might try and do a veeam backup with no compression or dedupe and see how it backs up... It might be a fair bit bigger, but is worth the extra space if I can back it up at 3 times the speed.

All of our backup files are sitting on a box full of disk..... veeam backups plus the rest of them...... So the target isnt really a problem either...


Compression is set on the tape...... I havent set compression at the arcserve level.

[dekkar]

Hmmmmm I think thats it..... compression.... or dedupe...... Running a backup now of a uncompressed image, and its going at 3.5GB/Min...... which is spectacular speed!

will have to keep testing and see if this flows onto my prod stuff....

[mpozar]

Hi,
We are running CA Arcserve Backup 15 with an IBM TS3100 LTO-5 Tape Library connected via a SAS Interface and I backup my Veeam Backup Images which are created with Compression Level Extreme and I get on Average 5.799 GB/min.

CA ARCserve Backup -- Backup[SubJob]Totals For................... SubJob 1Job No....................... 4Job ID....................... 579Total Session(s)............. 1Total Size (Disk)............ 1.506 TBTotal Size (Media)........... 1.411 TBElapsed Time................. 4h 38m 57sAverage Throughput........... 5.799 GB/minTotal Error(s)/Warning(s).... 1/0SubJob Status................ Finished[Master Job]Totals For................... Master JobJob No....................... 3Job ID....................... 578Total Session(s)............. 1Total Directories............ 5Total File(s)................ 85Total Skip(s)................ 0Total Size (Disk)............ 1.506 TBTotal Size (Media)........... 1.411 TBElapsed Time................. 4h 41m 35sAverage Throughput........... 5.799 GB/min

Have FUN!

Michael

[dellock6]

I can confirm those impressions: disabling tape compression is mandatory when saving to tape, Veeam files cannot be further compressed, and the tape system only wastes time trying to compress them. I'm seeing right now at a customer an average of 2 Gb/min on an LTO4, on a LTO5 you can expect higher numbers.

Luca.

[tsightler]

Hmmmmm I think thats it..... compression.... or dedupe...... Running a backup now of a uncompressed image, and its going at 3.5GB/Min...... which is spectacular speed!

Well, it may be 3.5GB vs 1GB, but the file itself is probably 3x bigger without compressing, so the overall time might be very similar. How is your tape drive connected?

[dekkar]

Unfortunately, it failed. The backup of uncompressed data has been going for almost 17 hrs and currently at under 400MB/Min. It started off well, and stayed fast for around 45 mins.

Back to the drawing board!

I'm going to go to the other extreme and produce a highly compressed job, and see how this backs up.

[ChrisGray]

I'm having a similar situation and wondering if you've found a resolution.

I have two proxy servers, backing up about 30VMs nightly doing the reverse incremental. Everything is 4GB/s FC (Disk based backup for Veeam, Proxy servers and Tape Changer).

Trying to off load about 3TB of these to tape and I'm getting mixed results using CA ARCServe. A backup of one of the proxy servers (.vbk, .vbm and .vbk files only), writing to 1.6TB to LTO6, I get nearly 4GB/Min, then a backup of the second proxy it starts out to get good throughput ~4GB/Min then AS SOON as it hits the .vbk file in the one backup job, throughput drops to 600MB/Min.

Both proxy servers are identically configured, hardware and software. The proxy servers have their Veeam backups on a FC LUN and the performance drop can be attributed to the one large .vbk file.

Problem is that my initial backup run on this job took so long, that the Veeam failed the next night because the file was locked by ARCserve. I'm trying to do my first month end tape for off site storage.

I've just opened up a case with CA and I'm waiting to hear back, but would appreciate any input. I don't think my issue is with network or storage speeds, because for the first 12 minutes or so of the job, through put is acceptable and what we predicted, UNTIL it attempts to back up the one .vbk file, then the speed drops about 5-6 fold.

[Vitaliy S.]

UNTIL it attempts to back up the one .vbk file, then the speed drops about 5-6 fold

So what performance do you get if you disable compression or deduplication for the second backup file? Have you tried doing that?

[veremin]

I'm having a similar situation and wondering if you've found a resolution.

As mentioned above, if you’re using built-in tape compression functionality, there is no wonder why you see such a behavior – highly-compressed and deduplicated .vbk files can’t be compressed any further, it, in its turn, results in decreased writing performance.

Even though you don’t experience any issues with one of .vbk files, it doesn’t necessarily mean something, as backup files, even of similar servers, are completely different and can’t be treated as similar ones.

For the purpose of “pure” testing it stands to reason to perform two completely new full backup, then copy them to tape appliance and see whether there is difference in writing speed.

I have two proxy servers, backing up about 30VMs nightly doing the reverse incremental.

Furthermore, it’s not recommended to utilize reversed incremental mode in case of writing to tape, as each time you’ll end up copying the whole backup.

Hope this helps.
Thanks.

[ChrisGray]

Oddly enough, the problem went away with the next full backup where a updated.vbk file was generated.
I was seeing consistent writing speeds of 4GB/min to tape (.vrb and .vrm files, then once it hit the one .vbk file, the speed dropped to 600MB/min).