Comprehensive data protection for all workloads
Post Reply
ejleipold
Enthusiast
Posts: 65
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Poor Reverse Incremental Performance

Post by ejleipold »

Hi Guys,

Another odd problem that's appeared after going from version 6 to 6.1.

I do a full backup (reverse incremental) from a virtualized proxy, to my local NAS and it goes great guns (around 50-60MB/s), awesome throughput with the bottleneck on the source/proxy end.

The next day, the job performs a reverse incremental and the performance is dismal (3MB/s) and the bottleneck is now the target.

No network conditions have changed over night (There is a 2Gb trunk between the proxy and the NAS), no config changes to the job or veeam and my NAS isnt working hard in the slightest.

When it was doing the full backup, the proxy's CPU was maxing out and working hard, but now it's barely doing anything.

Any ideas?
ejleipold
Enthusiast
Posts: 65
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Re: Poor Reverse Incremental Performance

Post by ejleipold »

After doing a bit more research, I think I've given up on reverse incrementals, they just seem too damn slow. Looking at switching to forward incremental with transformation instead as 3-5MB/s is horrid :/
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Poor Reverse Incremental Performance

Post by J1mbo »

Can you provide some detail on the physical storage?
ejleipold
Enthusiast
Posts: 65
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Re: Poor Reverse Incremental Performance

Post by ejleipold »

On the target end? It's just a QNAP NAS in RAID5 with SATA disks, nothing flash. I didnt expect the reverse incremental through-put to be less than 10% of the full backup through-put (from what I've read, reverse incrementals use 3 times the IO than a full backup?).
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Poor Reverse Incremental Performance

Post by J1mbo »

How many disks? Does it have any write cache?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Poor Reverse Incremental Performance

Post by dellock6 »

Yes, but this is only about the IO amount. During reverse incremental you do not have anymore sequential writes like forward incremental, but also random access for reads necessary to seek and replace updated blocks. Qnap has low-end processor and low amount of ram. I use also Qnap in some environments with reverse incrementak, it's not a problem itself, you only have to find out how much time it takes to complete the backup with reverse, and decide f you are fine with that amount of time.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
ejleipold
Enthusiast
Posts: 65
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Re: Poor Reverse Incremental Performance

Post by ejleipold »

The problem with that is that the QNAP just sits there doing not much (It's a TS-859U+ with 4 cores and a gig of ram), it doesn't even look like it's trying (although I'm trying to get some more disk performance info out of it). I get that there should be a big performance hit, but it just seems a lot bigger since going to 6.1. There's 7 disks in raid 5, write cache is enabled. I've logged a support request with Veeam, but I get the feeling its just that my NAS isnt fast enough for reverse incrementals now. I've switched over to forward incrementals with transformation so I'll see how that goes.
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Poor Reverse Incremental Performance

Post by J1mbo »

Is 6.x reverse increment subject to the same performance bug as the transforms, I wonder.
ejleipold
Enthusiast
Posts: 65
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Re: Poor Reverse Incremental Performance

Post by ejleipold »

Probably not, I installed 6.1 on a brand new VM and setup a new backup job and still had the same problem.
J1mbo
Veteran
Posts: 261
Liked: 29 times
Joined: May 03, 2011 12:51 pm
Full Name: James Pearce
Contact:

Re: Poor Reverse Incremental Performance

Post by J1mbo »

The problem is corrected only with a hot fix at the moment though.
ejleipold
Enthusiast
Posts: 65
Liked: 9 times
Joined: Oct 19, 2011 6:14 am
Full Name: Evan Leipold
Contact:

Re: Poor Reverse Incremental Performance

Post by ejleipold »

Interesting.
joergr
Veteran
Posts: 391
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: Poor Reverse Incremental Performance

Post by joergr »

There is ONE little chance of the trunk being the main problem, thus, please try to do the next backup with only 1gb ( no 2gb trunk). BUT....

I think it´s the nas box. I think the many random reads (which are necessary) are killing the performance and the nas box cant´t handle it properly. Are you willing to try to backup to the local Partition (maybe you need to resize the c partitition or throw em another vdisk) of your virtual Proxy and check out the next increment run after the main reverse inc run? If yes, please clarify on which storage the virtual proxy runs (local storage of hypervisor or san storage and if san, please give model and vendor and don´t say the "san" is also a nas box ;-) )

Best regards,
Joerg
Post Reply

Who is online

Users browsing this forum: ante_704, Google [Bot] and 198 guests