Performance in Forever Forward Incremental

[Massimo Vitali]

Hello, I am using Veeam V8 to backup three clusters. Each cluster is backed up by a single job using a dedicated proxy and a dedicated repository. The repository is a virtual disk (VMDK) mounted on the corresponding proxy. The VMs are backed up using hot add. Using this configuration, I expect that all the actual I/O between the hot-added VMDKs and the repository is done entirely by the proxy, without involving the backup server.
A couple of weeks ago I changed the backup method to Forever Forward Incremental (formerly it was Reverse Incremental). On the first schedules, the performance was improved: each job schedule simply deleted the oldest .VRB file and created a new .VIB file (incremental), instead of injecting directly the changes into the full backup and creating the .VRB with the rollback differences.
After 14 days (the retention period), the last .VRB was deleted, and the backup chain consists only of the full backup and the .VIB files. The next schedule creates a new .VIB during the backup of the VMs and deletes the oldest .VIB after injecting it into the full backup. While the actual backup runs as fast as in the previous schedules, the merge of the oldest .VIB into the full backup at the end of the job is dramatically slow. For example, for a job consisting of 34 VMs that are backed up in about 1h 30min, the merge of the oldest .VIB took more than 4 hours. The whole job execution, including both the backup and the merge, was about 3 times slower than the same job using reverse incremental.
My doubt is: in this configuration (repository disk directly attached to the proxy), WHO is actually performing the data transfer between the .VIB and the full backup during the merge? Although during the backup the data transfer between VMDK and repository is entirely done by the proxy (and I think also the injection into the full backup in case of reverse incremental), I suspect that using Forever Forward Incremental, the post-processing phase that merges the .VIB into the full is done by the backup server, involving a data transfer from the proxy to the backup server (for reading the oldest incremental) and another data transfer from the backup server back to the proxy (for writing on the full backup). This could explain why the merge phase is so slow if compared to the merge phase done "on the fly" during the reverse incremental.
Is this correct? I tried to examine the log files both on the backup server and the proxy, but i was not able to understand what actually happens during this phase.
Thank you in advance
Massimo

[foggy]

Massimo, synthetic block relocation activity performed during merge of the latest increment into the full backup is performed locally on the backup repository server (proxy server in your case), similarly to reverse incremental mode, backup server is not involved in it. Technically both methods require similar amount of I/O and provide comparable performance. Any chance Windows deduplication is enabled on the target volume?

As a side note, storing backups on VMFS is not a best practice.

[Massimo Vitali]

Hello Foggy,

yes I am using Windows 2012 R2 native data deduplication on repository disks.

Last night the backup performed better than yesterday. The merge on one repository (full 800 GB, each incremental about 30-40 GB) took 3h 15min (yesterday it required more than 4h). On other repository (full 600 GB, each incremental about 20-30 GB) the merge took 2h 50min (yesterday it took more than 9 hours). Yesterday it was the the first merge after changing the backup method from reverse incremental to forever forward incremental.

Thank you for the links provided in your reply, I will carefully read the related posts.

[foggy]

yes I am using Windows 2012 R2 native data deduplication on repository disks.

That explains. Most likely the full along with the oldest increment were already deduplicated, so the merge process required data rehydration. This does not occur in case of reverse incremental, since the full backup is always the most recent restore point that is never gets deduped (at least, with default 'Duplicate Files Older Than' setting).

[Delo123]

I would argue rehydration is the correct word for this... We observed there is hardly any impact "dehydrating" data from Microsoft dedupe volumes, we even saw better performance due to caching then without dedupe enabled, however as soon as data is deduped access of the data changes from mostly sequential to mostly random. If the repository is mainly hdd based then of course performance drops significantly due to random IO (on top of possible parity raid penalty etc...)

Just my 5 cents based on observations in our environment...