Reverse incremental backups getting slower and slower

[mephisto]

Hi guys,

I'm trying to get this clarified a bit as I'm not sure yet where is the bottleneck of my backup jobs. I've a client we are using reverse incrementals so we can keep one full backup as the latest and about 14 snapshots of reverse incrementals behind every day. I've noticed that on the beginning ti was really fast, and they with 2-3 months started to get a bit slow and now 6 months later it is slow as hell. From 100mb/sec down to about 5mb/sec, and testing the storage array I can get normal speeds for example just copying files across the disks normally.

I've been reading a bit and looks like the reverse incremental causes fragmentation inside the .vbk file, and this is the reason the backups are taking such a long time. How can I avoid this happening or resolve this matter now?

I need to keep somehow an eternal full backup like the reverse incremental, it could be the opposite way was well like the oldest being a full backup and normal incremental afterwards, but I believe merging the oldest incremental with the current full backup will end up causing the same problem as the reverse incremental, right?

I don't have storage to create a new full backup, so this is my main problem now.

What are your thoughts about this? Thanks!

[veremin]

It wouldn't help you to save space. Though, it would help to start backup chain anew, increasing backup performance. According to the best practice, you should run full backup every 1-3 months. Thanks.

[mephisto]

So a forward incremental and synthetic full once every week or two won't be any better than a reverse incremental regarding VBK fragmentation, or is there any advantage? Is this really due to fragmentation inside the VBK file just to make clear what is the issue?

[veremin]

Most likely, fragmentation is the major reason of such a slow down in backup performance. Even though, v7 introduced some improvements over reversed incremental mode, running job for 6 months without full backup has eventually lead to the described behavior. Thanks.

[mephisto]

So, if I use a full synthetic backup job to tape for example, the VBK file written to tape will also be "fragmented" right?

[foggy]

Quite expected that VBK file is heavily fragmented after running reverse incremental job daily for 6 months, periodic active fulls are always recommended. Btw, here's a good reading on that.

So, if I use a full synthetic backup job to tape for example, the VBK file written to tape will also be "fragmented" right?

Yes, since we're talking about fragmentation inside the VBK file, not fragmentation of the underlying file system, and the same VBK file that you have in your repository will be copied to tape.

[veremin]

So a forward incremental and synthetic full once every week or two won't be any better than a reverse incremental regarding VBK fragmentation, or is there any advantage?

I wouldn't say that the fragmentation would be the same, as with the synthetic mode full backup updates on less frequent basis. (once a week, not a day). Anyway, whatever mode you choose, you still should run active full backup from time to time. The major advantage of reversed incremental mode is that it's more space conservative.

Thanks.

[mephisto]

Thanks, I'll change my best practices for these backup methods. Much appreciated.

[veremin]

You're welcome. Feel free to ask for additional clarification, if needed. Thanks.

[serverbu]

Hello,

we are using vb&r enterprise to backup vms from an environment with two esxi hosts in a vcenter.
All vms use datastore from san (no local storage of esxi hosts is used).
We installed a dedicated vm on each esxi host to be used as a proxy.
The hosts itself are connected through 4GB multipath fc to storage and normal operation can read/write with appropriate speed.
Backup jops are configured to use reverse incremental with inline dedup, swap file exclusion, optimal compression, local target, usage of cbt and no guest processing.
Backup jobs have a very poor performance on processing rate of 4 to 9 MB/s on the job with bottle neck beeing detected at target (around 73% - 77%) most of the time (sometimes its source with around 65%)
When checking a specific vm in the job we see read rates of around 100KB/s to 3 MB/s.
We checked the hosts CPU, RAM, and NETWORK loads but all were within 30% usage, so much space left on that part.
While the backup is taking place we have heavy impact on our vms. One is a database server and the application working with that db is not usable while backup takes place.
Due to the extreme low performance backup of that maschine takes between 11 and 16 hours for 4 disks with 40GB, 200GB, 300GB, and 20GB.
The Job with this Maschine consists of 7 Linux vms with 2,1TB overall data usage.
We have no clue what is causing this extreme poor performance.

[serverbu]

Nice to be merged in this thread, but we do a active full backup every first sunday of a month, so i don't think fragmantation is the issue in this case.

[PTide]

Reverse incremental mode is very IO intensive on your target storage. Please try to run forward incremental job once and see if there is a difference.

Thank you.

[serverbu]

We completed our tests with forward incremental jobs and experienced disc utilisation of around 63% while with reverse incremental disc utilisation had an average of 97% with several 100% spikes, so it seems our NetApp filer can't handle the I/O produced by reverse incremental transformation process.