Feature Request - GFS Forever Incremental

[newfirewallman]

For improved IO and diskspace would it be possible that the GFS configuration be used as a forever incremental? Example:
First backup saved as your VMDK, then based on GFS schedule create and incremental job of data that changed since the original vmdk? this would dramatically change the amount of space required to store long term retention on disk and IO should be limited after the original backup.
each new backup time configured via GFS would merge data and rename files per date specified via GFS?

(could call it GFS job vs copy job, and remove the requirement of how to copy (copy interval) as it would know when via GFS schedule and source job. This would also fix the issue where if you have it checking daily so you get the correct GFS date via backup policy your organization might have, but the backups and merge take to long it can't finish the previous before the next day and job occurs.)

[dellock6]

Blake,
there are different reasons why GFS uses full files:
- if you think about the time drift between different GFS retention points, is really dangerous to keep an incremental referring to a full file created for example 6 months before. If anything goes corrupted, you've lost an entire GFS chain, maybe comprising multiple years of data. The actual GFS with full files removes this problem, any restore point is independent
- I'm not so sure your idea would lower the IO requirements. After 6 months again the amount of changes stored in an incremental file would probably be as big as a full file, and the IO required to do the merge would still be 2 per block. Also, when restoring, we should have to read from both files at the same time, 2 IO again. Writing a new full file it's just 1 IO per block.

If GFS merging is taking more than a day, then let's work on the performance issues and solve it, switching to a forever incremental imho it's not going to make things better.

Luca

[veremin]

There has been discussion already about backup copy job storing GFS as full backups; might be worth checking. As Luca's mentioned, the main reason is for each GFS restore point to be completely independent from others. Thanks.

[newfirewallman]

That is understandable for sure.

[newfirewallman]

Although if i have a 8-10TB file server there is the possibility in the long term retention that there isn't going to be much we can do to increase performance, when the goal is long term retention on cheap high density storage. If we didn't do GFS increments then it would be acceptable to only do the copy job portion based on the GFS schedule and not daily or every x number of days like:

"(could call it GFS job vs copy job, and remove the requirement of how to copy (copy interval) as it would know when via GFS schedule and source job. This would also fix the issue where if you have it checking daily so you get the correct GFS date via backup policy your organization might have, but the backups and merge take to long it can't finish the previous before the next day and job occurs.)"

this would give enough time for jobs to complete, get the data on the correct days and get ride of the section of the copy job where it says restore points to keep with a minimum of 2. JUST do the GFS archival portion (saving disk for long term retention)