Poor Reverse Incremental Performance

[ejleipold]

Hi Guys,

Another odd problem that's appeared after going from version 6 to 6.1.

I do a full backup (reverse incremental) from a virtualized proxy, to my local NAS and it goes great guns (around 50-60MB/s), awesome throughput with the bottleneck on the source/proxy end.

The next day, the job performs a reverse incremental and the performance is dismal (3MB/s) and the bottleneck is now the target.

No network conditions have changed over night (There is a 2Gb trunk between the proxy and the NAS), no config changes to the job or veeam and my NAS isnt working hard in the slightest.

When it was doing the full backup, the proxy's CPU was maxing out and working hard, but now it's barely doing anything.

Any ideas?

[ejleipold]

After doing a bit more research, I think I've given up on reverse incrementals, they just seem too damn slow. Looking at switching to forward incremental with transformation instead as 3-5MB/s is horrid :/

[J1mbo]

Can you provide some detail on the physical storage?

[ejleipold]

On the target end? It's just a QNAP NAS in RAID5 with SATA disks, nothing flash. I didnt expect the reverse incremental through-put to be less than 10% of the full backup through-put (from what I've read, reverse incrementals use 3 times the IO than a full backup?).

[J1mbo]

How many disks? Does it have any write cache?

[dellock6]

Yes, but this is only about the IO amount. During reverse incremental you do not have anymore sequential writes like forward incremental, but also random access for reads necessary to seek and replace updated blocks. Qnap has low-end processor and low amount of ram. I use also Qnap in some environments with reverse incrementak, it's not a problem itself, you only have to find out how much time it takes to complete the backup with reverse, and decide f you are fine with that amount of time.

Luca.

[ejleipold]

The problem with that is that the QNAP just sits there doing not much (It's a TS-859U+ with 4 cores and a gig of ram), it doesn't even look like it's trying (although I'm trying to get some more disk performance info out of it). I get that there should be a big performance hit, but it just seems a lot bigger since going to 6.1. There's 7 disks in raid 5, write cache is enabled. I've logged a support request with Veeam, but I get the feeling its just that my NAS isnt fast enough for reverse incrementals now. I've switched over to forward incrementals with transformation so I'll see how that goes.

[J1mbo]

Is 6.x reverse increment subject to the same performance bug as the transforms, I wonder.

[ejleipold]

Probably not, I installed 6.1 on a brand new VM and setup a new backup job and still had the same problem.

[J1mbo]

The problem is corrected only with a hot fix at the moment though.

[ejleipold]

Interesting.

[joergr]

There is ONE little chance of the trunk being the main problem, thus, please try to do the next backup with only 1gb ( no 2gb trunk). BUT....

I think it´s the nas box. I think the many random reads (which are necessary) are killing the performance and the nas box cant´t handle it properly. Are you willing to try to backup to the local Partition (maybe you need to resize the c partitition or throw em another vdisk) of your virtual Proxy and check out the next increment run after the main reverse inc run? If yes, please clarify on which storage the virtual proxy runs (local storage of hypervisor or san storage and if san, please give model and vendor and don´t say the "san" is also a nas box )

Best regards,
Joerg