Veeam 7 gradual speed decrease

[DanielG]

Hi all,

Recently I upgraded our Veeam servers to version 7 (7.0.0.715) and here is what I noticed.
The first time backup job (Full backup) finishes at 60-90MB/sec read speed and 30-40MB/sec transfer speed, which is expected performance in our environment.
The second job shows pretty same numbers, but then read and transfer speed is starting go down gradually and after a week I am getting 9-6MB/sec read speed and 5-3MB/sec transfer speed. With this performance 24 hours isn't enough to finish incremental backups.
We have two different environments with different hardware and it happens in both.

The general configuration and environment:
- Hosts are ESXi 5.x and 4.x running on different HP hardware
- Source storages HP EVA 4000 and MSA 2000 fiber channel
- Target storages are Qnap 1279x iSCSI, Dedicated for backups.1000Gb backup network.
- Veeam installed as VM (proxy and backup together). Transport mode - Network
- No simultaneous jobs.
- Parallel VM processing is enabled.
- Backup mode: Reversed incremental
- Compression: Optimal / Optimized for: Lan target

The average bottleneck status on all jobs is Source: 2013-10-18 3:28:20 AM :: Load: Source 82% > Proxy 4% > Network 22% > Target 76%

Any advice on what can be wrong. Thank you!

[Vitaliy S.]

Hello Daniel,

Did you have the same bottleneck stats prior upgrading to v7? Looks like the reason of such performance degradation might be the target storage, besides reversed incremental jobs are well-known for its IO load on the repository (compared to forward incremental mode).

I wonder what do job logs say on this... BTW, will it possible to switch to forward incremental mode to verify what job performance you will get after this change?

Thank you!

[dellock6]

Another possible cause could be the fragmentation on the VBK file, since reversed incremental always updates the same file. But I never saw it happening so quickly after its first creation, it could happens after some months, and is one of the reason to run an active full periodically, among other reasons.
And if I remember correctly, v7 introduced some enhancements to improve VBK fragmentation too...

Luca.

[Gostev]

That's right, starting v7 reversed incremental should be less prone to fragmentation, and in any case it normally takes weeks before you will start observing noticeable impact.

[DanielG]

Thank you guys for the advices.
I didn't have the same behavior with v6.5 and you're right, it was taken weeks and months before slowing down.
I will do more testing and try forward incremental. I'll post the results.

Daniel

[DanielG]

After a week of different tests it clearly showed that regular incremental solves my issue.
The average Processing rate with Incremental is 35-47MB/sec now. With Reverse Incremental it takes 4-6 runs to get to 5-10MB/sec.

[Vitaliy S.]

Daniel, thanks for the update, however performance degradation still shouldn't be observed after 4-6 runs.