Comprehensive data protection for all workloads
Post Reply
mbreitba
Enthusiast
Posts: 85
Liked: 8 times
Joined: Jun 11, 2012 3:17 pm
Contact:

Forever Forward Incremental consolidation Scheduling?

Post by mbreitba » 1 person likes this post

I have a lot of backup jobs running, and currently it takes around 14 hours for all backups to complete. I'm looking at implementing forever forward-incremental but would like to schedule the backup chain consolidation. Is this something that is possible, or is the consolidation required to be part of the actual backup job? I only ask because I would like to let the backup storage target concentrate on backups, rather than consolidating backup chains during the backup window. I haven't found an option for doing what I'm proposing, so maybe this is a feature request?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Forever Forward Incremental consolidation Scheduling?

Post by Shestakov »

Hello,
Could you provide your jobs bottleneck statistics for the backup jobs? Is the same across all jobs?
The merge process should not take a lot of resources in general.
Thanks.
mbreitba
Enthusiast
Posts: 85
Liked: 8 times
Joined: Jun 11, 2012 3:17 pm
Contact:

Re: Forever Forward Incremental consolidation Scheduling?

Post by mbreitba »

From which ones? I've got 94 jobs covering nearly 600VM's. Bottleneck ranges from source to target, depending on the backup set.

Currently we are running reversed incremental backups, and I'd like to evaluate forever forward incremental, but it seems like I would have the same problem that I have today with reversed incrementals with that many jobs (Mainly, the 2 IOPs per bit of data to read and write the old data into the primary backup file running while snapshots are open).

If I could schedule the consolidations to happen after all of the new backup data was written, it would not impact any running backup jobs. If I cannot schedule it, the merges will be happening at the same time as snapshots are open on running VM's (albeit from a different backup job).
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Forever Forward Incremental consolidation Scheduling?

Post by Shestakov »

The reason why I asked about bottleneck is the fact that additional consolidation IOPs can slow down backups only if the bottleneck is the target.

Comparing Forward Incremental-Forever Backup and Reverse incremental, note that despite the fact they both use 3 IOPS per byte, technically reverse incremental is "slower" during the backup process itself. With reverse incremental all 3 I/Os happen during the backup process itself as each changed block is written to the repository. With forever forward during the backup process itself there is only 1 I/O to the repository, a single write, which means the VM backup will complete much faster, the other 2 I/Os are performed "post-process" during a merge at the end of the job, just like you wanted to.

If it doesn`t work and you want to speed up the process, simple forward incremental with periodic full backups on weekends is a way to go.
Thanks.
mbreitba
Enthusiast
Posts: 85
Liked: 8 times
Joined: Jun 11, 2012 3:17 pm
Contact:

Re: Forever Forward Incremental consolidation Scheduling?

Post by mbreitba »

I understand that it is "after" the backup has been taken, but with 90+ jobs, and 600 VM's, it does cause contention.

Let's say all backup jobs kick off at 5PM. The first 10 jobs kick off, and the other 80 jobs are queued. When the first job completes it's backup, the other 9 may still have open snapshots. It starts consolidation while 9 other backup jobs are still working with open snapshots. The other 80 jobs are still waiting in que. If we could create the new VIB file, and close the backup job, all 90 jobs could complete faster and post-process we could do the consolidation of VIB files during business hours. Does that scenario make sense?
mbreitba
Enthusiast
Posts: 85
Liked: 8 times
Joined: Jun 11, 2012 3:17 pm
Contact:

Re: Forever Forward Incremental consolidation Scheduling?

Post by mbreitba » 1 person likes this post

Any additional thoughts on this? I'd like to submit this as a feature request for forever incremental that the daily transform process be scheduled outside of the backup window to reduce competition with the target storage for in-flight backups.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forever Forward Incremental consolidation Scheduling?

Post by Gostev »

It's a good idea, thanks for your feedback!
Post Reply

Who is online

Users browsing this forum: No registered users and 184 guests