Backup Copy Transformation - Very Slow

[DavidS]

Hi All

We've got a strange issue with our Backup Copy Jobs lately:

We made a Backup Copy Job from our normal Backup Jobs which is now running since a month.
The Size of our Backup is about 1.7TB -> The size of the Backup Copy is about the same.

Five Days ago, the "Transformation" of the Backup Copy Job started (I think this is the merging of the incrementals to a new Full?)
Anyway, it proceeds about 1% Every 4 Hours, now its at 30% (after 5 Days running!) thats horribly slow.

The source of the backup and the backup copy are on the same drive (iScsi LUN provided by our Synology NAS), maybe this make some problems?
In the ressource monitor of the veeam server, I see in the disk performance, that the backup job file (vib/vbk) is continuously in write and read access.
The network and processor utilization however is pretty low, so it looks like he's struggling with reading and writing any faster.

Further do we have the extreme compression activated. Our normal Backup Speed is at approx. 250MB/s (so I think generally its not that bad configured )

Our Environment:

Backup Destination: Synology RS10613xs+
Disks: WD Red
Veeam + Proxy Server: HP DL360 G7 with 60GB RAM and 6 (12 virtual) Cores
Source: 2 x HP Dl580 G8 (But the Backup Copy Jobs don't touch this Servers)


If anyone has an idea, what could cause this slow speed, i'd be glad for any suggestions.

Best Regards
David

[foggy]

Hello David,

The source of the backup and the backup copy are on the same drive (iScsi LUN provided by our Synology NAS), maybe this make some problems?

Definitely, it does. Transformation is a very I/O intensive operation per se, while Synology NAS is not in the top of the fastest backup targets. Making this device the source and the target for the backup copy at the same time literally kills it.

[DavidS]

Hi Foggy

Thank you for your input.

I don't think the Synology is the bottleneck, it should be able to perform around 400'000 IOPS (but you're right, it is more like a middle/low-end storage).

On the other hand, the Western Digital Red Drives we use, have an average performance of 110 IOPS -> In a 9 Bay RAID 5 Array that would sum up to 400 IOPS -> is that too little?

So what would be the Best Practice for our purpose?
Would an extra LUN on the same NAS be enough to handle the requierements or does it have to be a separate device?

Greetings
David

[foggy]

Remember, backup copy job transformation is a heavy random I/O (even though it does 33% less I/O operations than reverse incremental mode transformation). You can try to add a separate LUN, as well as switch to RAID 10 (less I/O penalty) and see whether this helps.

Keep in mind though, that if the NAS fails, you loose both copies of backups, so having copy on another device is definitely recommended.