Trigger Synthetic full via script

[tsightler]

So I've haven't had time to check this myself, but I thought someone out there might already know. Is it possible to trigger a synthetic full transformation from a script? The current scheduling for synthetic fulls is non-ideal for our environment (you can only do it weekly). With "real" full backup we can run them monthly but there doesn't appear to be a way to make synthetic fulls occur only monthly. If you can trigger a synthetic full via a script then we could write a post-script that does the work only for the first day of the month.

[Gostev]

Hi Tom, I can find this out on Monday, but I certainly don't recommend doing synthetic fulls so rarely (we have not done any testing for creating synthetic fulls from VIB chains longer than 6). Not only that, but I also do know for sure (from researching something unrelated) that each additional VIB in chain slows down the next backup slightly. Everything might works just fine, however again, this is completely untested.

[tsightler]

OK, I was just trying to figure out a way to make the incremental backups fit into our retention model while not wasting a lot of space with weekly full backups. Basically the synthetic fulls are killing us from a space perspective since you have to run them every week. For 90 days of retention you end up with 12 full backups. With reverse incrementals it was easy to keep 90 days of retention with out 32TB of storage, now we would need ~90TB to keep the same retention.

[Gostev]

Yep, that was the whole point of reversed incremental backup and the reason why forward incremental backup mode provides optional transform, but I know there are other reasons why you don't want to use this option.

[Bunce]

Did someone say 'tiered retention policies' ..?

Surely a week isn't a huge difference from a month in terms of reliability in the chain if the product is backing up and validating correctly?

[Gostev]

I think I first heard this from Tom couple of years ago

[tsightler]

Yep, that was the whole point of reversed incremental backup and the reason why forward incremental backup mode provides optional transform, but I know there are other reasons why you don't want to use this option.

Well, we might be able to use the transform option. I'm really not sure why the transform option would be THAT much slower than the synthetic full option. When we tested that setup it seemed to be attempting to be too space efficient. It would take the full and the next incremental and appeared to actually roll the changes into the existing full, while creating the new rollback, and then repeat 6 times, taking hours to process each and every incremental.

I would have preferred the process to be a little more "staged". The first stage would be to create the new synthetic full just like it does now. For the second stage it could go back and use the information in the previous full, plus the new fulls, and the incrementals, to create all of the reverse incrementals (at no point during this phase would the original, already known good backups be touched). The final stage, once all the reverse incrementals were created would be to delete the older full backup and incrementals.

Perhaps we will test this option again, but I have my doubts that it's going to work for our relatively large backups.

[Gostev]

Surely a week isn't a huge difference from a month in terms of reliability in the chain if the product is backing up and validating correctly?

Well, I did not say a word about reliability. Reliability of long incremental chain is definitely NOT the issue I am concerned about here.

I would have preferred the process to be a little more "staged". The first stage would be to create the new synthetic full just like it does now. For the second stage it could go back and use the information in the previous full, plus the new fulls, and the incrementals, to create all of the reverse incrementals (at no point during this phase would the original, already known good backups be touched). The final stage, once all the reverse incrementals were created would be to delete the older full backup and incrementals.

Yes, our transform is designed to be space effient. Staged would require enough disk space to keep 2 backup chains. For example 10TB with current implementation, but 20TB with staged approach. Too big difference... and many customer told us they just do not have disk space for second full backup, period.

Reliability wise... I don't see how keeping older full until the end of the process increases it. Current transform implementation essentially perform regular reversed incremental passes one by one (except that all required data is already on disk), so we are the same good old engine we used since version 1... it is designed to make sure that data cannot be lost no matter what happens during synthetic full rebuild.

Anyway, it sounds like best solution would be just to allow monthly synthetic fulls in future release (if testing shows that the engine can handle it okay).