Backup Copy Job - Very Poor Performance - V8

[mmonroe]

A quick update.. We suffered some really reduced backup times with backup copy jobs after initially going to v8. Things did not improve with v8 update #1. We worked for a couple of months with support with no improvement. Read this whole thread for the details on the issues.

I finally wiped the Veeam server and put v7 update #4 back into server and backup copy went back to their normal time intervals. I left it that way for several months.

I decided to "test" v8 update #3 and see if things had improved any. My testing on separate hardware seem to show time improvements that earlier issues with v8, u1 and u2 did not resolve.

A couple of months ago I decided to put v8 update #3 into production and see how things went. I am happy to report that the backup copy speeds are now better and, for the most part, are improved over v7. This is with the exact same hardware, servers, repositories and whatnot that have been in place for the past year. The only thing having been changes is Veeam versions from v7 to v8 update #3.

It is easy for a vendor to put off speed issues on "the hardware", however, it was pretty easy to document and show where all of the speed issues were related to the Veeam software and had nothing to do with any hardware changes on our end. As of now, the backup copy job speed issues are resolved with update #3 and thankfully I didn't start trashing our backup hardware infrastructure in the process of solving things earlier this year.

I am a bit nervous on updating to v9 since we finally, after three major v8 updates, have v8 running smoothly. I plan to test v9 on test stand-alone hardware this time prior to loading into production. We cant afford to have the backup copy job time windows increases that we saw after loading v8.

Just a friendly update on things..

[rreed]

Interesting you ran into this, and extremely coincidental that you reply today! I've been getting my feet wet w/ Backup Copy jobs recently and in our v8u3 environment (fresh v8 install, not an upgrade from v7) BC jobs are unusably slow at single-digit MB/s over 10Gbps connectivity (SAN and Dell DR4100's). I want to say at one point I got a BC up to around 20-something MB/s but I need to go back through and do some further testing to confirm. Doesn't seem to matter if I BC around from source to a VM's local drive repository (Dell EQL SAN @ 10Gb), DR4100 across 10Gb, anything. I can pull a Windows Explorer copy of files around, robocopy, etc. and speeds match what we would expect but the BC jobs are practically useless. Something isn't adding up here. Even if BC jobs run VM's asynchronusly vs. the Backup Job's syncronus it should still be able to copy a single file much fast than that. I was about to post a question of any BC job improvements in v9 when I stumbled onto this.

[rreed]

Having said that, I'll need to retrace my steps wherein I was getting sub-Fast Ethernet speeds. My currently running test BC job is getting about 105MB/s throughput - reading from one of my old PS6000 SAN's that are 1Gb connected. I believe my previous tests that were slow were coming from dedupe devices - but still when I can do a straight file copy from them at full speed my expectations were the BC should be able to get somewhat close to that. One of my questions was going to be whether Veeam does any file opening/reading or just copy a file. Admittedly, I do need to RTFM on exactly what a Backup Copy does, how it works, etc.

[Amyd]

I am having the same or maybe a similar issue with our Backup Copy Jobs. We went from v7 to v8u3, and after that the backup copy speeds tanked. Backup copies that used to process at 120+ MBps in v7 now creep along at 20-30 MBps, which is obviously pretty painful. In our case the architecture is very simple, both primary backup repositories as well as the secondary backup copy repository are all local drives on the Veeam Backup Server (dumb iSCSI LUNs from Synology units, no dedupe).

I am not sure if it is relevant, but the activity pop-up for a currently running slow backup copy job indicate 2/2/0/24 % as bottlenecks. An older backup copy job which ran at full speed showed 92/4/18/11 %, which in fact sound more plausible.

I haven't (yet) opened a support case, but I was wondering if there are any concrete hints from other users and/or cases of what this problem might be.

[foggy]

I believe my previous tests that were slow were coming from dedupe devices - but still when I can do a straight file copy from them at full speed my expectations were the BC should be able to get somewhat close to that. One of my questions was going to be whether Veeam does any file opening/reading or just copy a file. Admittedly, I do need to RTFM on exactly what a Backup Copy does, how it works, etc.

The main point you should keep in mind is that backup copy job does not copy files at all. It synthetically creates restore points in target location from the changed blocks extracted from the source storage. Synthetic activity requires much more I/O comparing to simple sequential read/write, putting higher load on the storage.

[rreed]

Thanks foggy, that might be those of us who use dedupe devices as end storage are afraid of. I haven't gotten to read into the architecture of Copy jobs yet, but is that what they do? Rather than sequentially write a "new" backup file, it actively reads/writes to the file system? And of course the next question, if so, any change of this in v9 please?

[foggy]

You can review this user guide section, should give you a better understanding of the details.

[Amyd]

I have now opened a support case (ID# 01243090) on the above mentioned problem, let's see if there is something that can be done.

I have also tried in the mean time to see if I can make sure it is not a infrastructure problem, and I ran a few different QuickBackup jobs to the same backup copy repository, they run at speeds up to 200 MBps, even while running in *parallel* to the sloooow backup copy job, which keeps hitting a ceiling of maybe 20 MBps.

The backup copy Job is for the record currently creating a first clean active full backup copy, so as far as I understand the Veeam architecture should be quite similar with a QuickBackup active full in terms of write patterns.

[foggy]

Thanks for opening a case.

Quick Backup performs an incremental job run, so it's speed cannot be compared to the speed of the full run. Moreover, you cannot compare performance of backup and backup copy jobs, since they retrieve data from different sources.

[Amyd]

Sorry, I meant a VeeamZIP backup, not QuickBackup, brainfart there. As far as I understand, the VeeamZIP is basically an Active Full, or not?

Obviously, the source is different (production storage versus the primary backup repositories), but I wanted to make 100% sure that there is no issue with the write speed of the secondary backup repository, since that is reported as the "bottleneck" in the slooow backup copy job.

[foggy]

Right, VeeamZIP is similar to active full. However, this is still not a valid comparison. You can compare write speed to the target storage, but not the overall processing rate reported for VeeamZIP and backup copy job. Anyway, I believe support engineers will be able to verify the setup and review log files to define the actual reason of the slowdown.

[Amyd]

Yes, I understand, but I believe in some way the write speed of the target is the thing that looks so suspicious, that's why I wanted to make sure it works like expected for other type of jobs.

It looks suspicious (to me at least), because watching the current job in progress it mostly seems to oscillate between 10.4 MBps and 20.8 MBps, with the higher value looking like some kind of "hard" ceiling. It's almost like Veeam is trying to throttle the average speed to somewhere in the middle of those two values. But, I don't have any data rate limits active on the repository.