Best way to copy to USB HDD's

[PeterPan1234]

Hello,

I have 4 backup jobs, each runs nightly and results in 4x1TB backup files. These are stored on fast local server storage.

I now want to copy these backup files to a USB HDD. This is for offsite storage. I have 10 disks rotated, one per day (over a 2 week cycle)

Option 1, Run a “post” script that copies the files
Problem is that the backup jobs might only take 20 minutes to run (incremental backups), and this will result in lots of jobs copying to the USB HDD at the same time. This will result in HDD thrashing. For every concurrently file I copy, it slows down about %75 of the speed – If I copy 3 files concurrently then im running about half speed – this means I cannot write to the USB’s disks before the end of my work day (so I can take them offsite)

Option 2, Run a “File Copy” Job, linked to the Backup Jobs.
This is the same issues with Option 1. They all run at the same time. I cannot link File Copy jobs to a proxy and limit the concurrent jobs.

Option 3, Backup directly to USB disk, bypassing the local storage.
I have the same issue – lots of jobs trying to write at the same time, and thrashing the disk.
“Rotated drives” is not useful in my case as I have so many USB disks it would be writing a full backup every time.

Option 4, This is what I have in place – and I don’t think it should be needed.
Once a Backup job finished is creates a token in a folder. Another script monitors this folder and when it finds a token, it copies the related file/folder. Once done it loops around looking for another token. Its only every copying one file at once.

What is the best way to copy the Backup files to a USB HDD? Thanks

[HannesK]

Hello,
and welcome to the forums.

Did you already think about the "backup copy job" in combination with a repository marked as "this repository is backed by rotated hard drives"?

Best regards,
Hannes

[PeterPan1234]

Hi Hannes
Thanks for the welcome, and the suggestion.

I have setup Backup Copy Jobs now, initial tests are transferring a bit slower that I would have hoped. In some stages its fast – other stages its slow. I have all my jobs setup and will run it like this for a few days to see if it fits in the time window.

I have a concern that when I swap out my disk at 4pm it will start by copying all data to the new blank disk, BEFORE it has refreshed the backups. The tests I did today show its not really a problem; If there is a conflict then a Backup Copy job will fail and give preference to a Backup Job.

I will provide an update once I smooth out everything.

Thanks Again
Peter

[veremin]

You can check the bottleneck statistics for backup copy job and see what component spends most time working with the data. As to job priority, you are correct in saying that backup job is given preference over backup copy job, so the described situation should not be a problem.

[PeterPan1234]

Thanks All,

This is all working great now - the bottleneck was only adding a few minutes (making a 2 minute job take 5 minutes). I was assuming it was running about half the speed, But that is not the case, its more like its got a few minutes of checking stuff to do before it starts to copy.

Thanks again.