Backup Copy very slow to USB 3 Disk attached to a QNAP NAS

[agrob]

Good Day

At a remote site, we have a new QNAP TS-453A with 4x4TB WD Disk. Created a Share on this Volume. To this share, we backup with VAW a physical Windows Server System. Backup is running fine with good backup performance. So far so good.
The NAS System has USB 3 Ports. on one of this ports is a 2TB WD USB 3 Disk connected. A Share is created on the NAS which points to this USB Disk. Now we use Veeam Backup and replication to offload the backup data once a week to this USB Disk for offload.
The Problem now is, the Copy job is very slow. Speed changes between 5-15MB/s but in generall it is about 6MB/s. But this problem does only appear if we use Veeam to offload the Data. if i stop the copy job and manually copy a VBK file from the Share from the Raid 5 Pool to the Share from the USB Disk, performance is about 80-90MB/s. What we would expect.
Any idea why copy with veeam is that slow and not with constant speed?

Veeam Backup Server is on our Headquater, the nas and the physical server with VAW is on a remote site. Gateway Server for both Backup Repsoitories (the one on the nas and the one on the usb disk) are configured with the local physical server on the remote site..

any idea?

Thanks

[foggy]

Hi Andre, what are the bottleneck stats for this backup copy job?

[agrob]

Hi Foggy

Bottleneck is Target. That would mean the USB 3.0 Disk. But if i manually copy the vbk file from the source repository and "paste" it to the usb 3 disk (over the share from the nas) speed is fine. it just seems a problem when veeam is copy the files... (i guess veeam is handling this copy different, but anyway it should not be that slow)

Best Regards
André

[agrob]

Just a short Update.
With QNAP i can configure the qnap Backup Application which basicaly replicates the folder with the veeam backups to the external attached USB Drive. Statistics of this Job
Folder Pairs : 1
Total File(s) : 5
Total Folder(s) : 2
Skipped Files : 0
Backed-up Files : 5
Total File Size : 360.92 GB
Average Transmit Speed: 116.82 MB/s

As we can see there is for sure not a performance Problem with the NAS or USB Disk.....
I let the veeam Copy job running over the weekend to check how this performs for this data

[Regnor]

Backup Copy is not working file based; it creates "synthetic" restore points on your target repository.
Because of that performance can and will decrease, as it's more I/O intensive, especially with a single SATA disk. On the other hand you get independent, consistent restore points which is the benefit of Backup Copy.

If you want to benchmark synthetic operations you can check the following KB:
https://www.veeam.com/kb2014

[agrob]

that sounds interessting, i'll test that.

i understand that veeam does not just copy the backup files from the repository. But only about 10-20Mb/s i think is a bit slow compared to 90MB/s or more when just manuall moving files.

But i'll do the perf testing....

[agrob]

used the benchmark with 50/50 read/write (diskspd.exe -c1G -b512K -w50 -r4K -Sh -d600 z:\testfile.dat).
these are the results

Total IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 83071860736 | 158447 | 132.04 | 264.08 | z:\testfile.dat (1024MiB)
------------------------------------------------------------------------------
total: 83071860736 | 158447 | 132.04 | 264.08

Read IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 41437102080 | 79035 | 65.86 | 131.73 | z:\testfile.dat (1024MiB)
------------------------------------------------------------------------------
total: 41437102080 | 79035 | 65.86 | 131.73

Write IO
thread | bytes | I/Os | MiB/s | I/O per s | file
------------------------------------------------------------------------------
0 | 41634758656 | 79412 | 66.18 | 132.36 | z:\testfile.dat (1024MiB)
------------------------------------------------------------------------------
total: 41634758656 | 79412 | 66.18 | 132.36

As in the articel described, we have to devide by 2 (or 4 but i guess it is not a transform / rollback operation)
-> So i should have aoubt 33MB/s Troughput... to this USB Disk

[foggy]

Add the fact that data is going through the gateway server, not directly from NAS disk to USB.

[Regnor]

Also there's an overhead and the I/Os are random, plus performance decreases when the disk fills up.
The benchmark can't 100% reproduce the load from synthetic operations, so the value is more an estimate.

[agrob]

thanks for your feedback.
after testing i can say, backup copy jo to a usb 3 Disk connected to the nas directly is not usable. copy job starts with about 50MB/s then it slows down to about 5-10MB/s. Maybe it would be better if the USB Disk is a SDD drive, but with conventional HD not really the way to go.
I now decided to use the Backup functionality the QNAP NAS Provides itself. I scheduled a copy job which copies all the data from the Backup share to the USB Drive once a week. Performance is about 100 MB/s. Notification if job is fine is done via email.

We just need it as an option to offload backup offiste. if we need to restore, we just import the backup files located on the usb disk. that should work fine.

[agrob]

There is just one additional Question.
I use now VAW Server edition. Is there any plan to implement the possibility for multiple schedules / or multiple backup Jobs functionality? because this would solve the problem. then we could do something like

Backup Job1
Mo-Fr to Veeam Repo1
Backup Job 2
Sa to Veeam Repo2 (USB), configure it as full.

Fully integrated with Veeam Backup Server, not scripting etc. (I know there is other solution like this one https://www.virtualtothecore.com/en/sen ... d-connect/ , but then we need scheduled task etc which we also need to monitor if they work corret)

thanks

[Dima P.]

Andre,

I use now VAW Server edition. Is there any plan to implement the possibility for multiple schedules / or multiple backup Jobs functionality? because this would solve the problem. then we could do something like

It's in the list of planned improvements. Thanks!

[rrosiak]

thanks for your feedback.
after testing i can say, backup copy jo to a usb 3 Disk connected to the nas directly is not usable. copy job starts with about 50MB/s then it slows down to about 5-10MB/s.

Hi, I have the same problem with SPX (Shadow Protect), but it on the Qnap side since firmware 4.3.4.0483 Build 20180213 (before was OK).
I've tested few different QNAP and USB HDD and everywhere it's the same