Comprehensive data protection for all workloads
Post Reply
DanielG
Enthusiast
Posts: 25
Liked: 1 time
Joined: Oct 18, 2013 4:14 pm
Full Name: Daniel Guretski
Contact:

Veeam 7 gradual speed decrease

Post by DanielG »

Hi all,

Recently I upgraded our Veeam servers to version 7 (7.0.0.715) and here is what I noticed.
The first time backup job (Full backup) finishes at 60-90MB/sec read speed and 30-40MB/sec transfer speed, which is expected performance in our environment.
The second job shows pretty same numbers, but then read and transfer speed is starting go down gradually and after a week I am getting 9-6MB/sec read speed and 5-3MB/sec transfer speed. With this performance 24 hours isn't enough to finish incremental backups.
We have two different environments with different hardware and it happens in both.

The general configuration and environment:
- Hosts are ESXi 5.x and 4.x running on different HP hardware
- Source storages HP EVA 4000 and MSA 2000 fiber channel
- Target storages are Qnap 1279x iSCSI, Dedicated for backups.1000Gb backup network.
- Veeam installed as VM (proxy and backup together). Transport mode - Network
- No simultaneous jobs.
- Parallel VM processing is enabled.
- Backup mode: Reversed incremental
- Compression: Optimal / Optimized for: Lan target

The average bottleneck status on all jobs is Source: 2013-10-18 3:28:20 AM :: Load: Source 82% > Proxy 4% > Network 22% > Target 76%

Any advice on what can be wrong. Thank you!
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam 7 gradual speed decrease

Post by Vitaliy S. »

Hello Daniel,

Did you have the same bottleneck stats prior upgrading to v7? Looks like the reason of such performance degradation might be the target storage, besides reversed incremental jobs are well-known for its IO load on the repository (compared to forward incremental mode).

I wonder what do job logs say on this... BTW, will it possible to switch to forward incremental mode to verify what job performance you will get after this change?

Thank you!
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Veeam 7 gradual speed decrease

Post by dellock6 »

Another possible cause could be the fragmentation on the VBK file, since reversed incremental always updates the same file. But I never saw it happening so quickly after its first creation, it could happens after some months, and is one of the reason to run an active full periodically, among other reasons.
And if I remember correctly, v7 introduced some enhancements to improve VBK fragmentation too...

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam 7 gradual speed decrease

Post by Gostev »

That's right, starting v7 reversed incremental should be less prone to fragmentation, and in any case it normally takes weeks before you will start observing noticeable impact.
DanielG
Enthusiast
Posts: 25
Liked: 1 time
Joined: Oct 18, 2013 4:14 pm
Full Name: Daniel Guretski
Contact:

Re: Veeam 7 gradual speed decrease

Post by DanielG »

Thank you guys for the advices.
I didn't have the same behavior with v6.5 and you're right, it was taken weeks and months before slowing down.
I will do more testing and try forward incremental. I'll post the results.

Daniel
DanielG
Enthusiast
Posts: 25
Liked: 1 time
Joined: Oct 18, 2013 4:14 pm
Full Name: Daniel Guretski
Contact:

Re: Veeam 7 gradual speed decrease

Post by DanielG »

After a week of different tests it clearly showed that regular incremental solves my issue.
The average Processing rate with Incremental is 35-47MB/sec now. With Reverse Incremental it takes 4-6 runs to get to 5-10MB/sec.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam 7 gradual speed decrease

Post by Vitaliy S. »

Daniel, thanks for the update, however performance degradation still shouldn't be observed after 4-6 runs.
Post Reply

Who is online

Users browsing this forum: divertisity, rweis, Semrush [Bot], ybarrap2003 and 152 guests