Offsite Incremental to Synthetic Full Utility

[Gostev]

Hi, Anton. There a very slim chance that you can do this through internal data mover command line, unless the process includes some external logic from .NET parts of code. I will be able to give you exact answer next week, when I am in the office and can talk to the devs. Thanks!

[foggy]

I have a set of files from regular backup job (1 vbk and 30 vib incrementals only). I can import them to Veeam and recover VMs from ANY time point (by the way, US support told me that it is impossible without vbm file as incrementals will be unreadable - not true!).

Correct, VBM file is not required to import the backup chain but considerably speeds up this process.

[AntonZ]

Thanks, guys!
At this point I have several sets of such files (vbk+n*vib) and several not working GFS jobs (from which I can't recover anything from incremental points).
I'm running out of space next week (I have 62TB volume with Win2012r2 dedupe, which currently gives me 140TB of backups) and need to do smth very quickly. The idea was to upload everything to amazon, but to do this I need vbk fulls (easier to recover, no need to download the whole chain).
If I need to pay for additional service to help me convert incrementals to fulls (if there is no internal utility or way to do it with available tools), I'm ready to pay.

[Gostev]

If I need to pay for additional service to help me convert incrementals to fulls (if there is no internal utility or way to do it with available tools), I'm ready to pay.

We don't do custom development, so that's off the table unfortunately.

[AntonZ]

Got it. While I'm waiting for an answer from devs, one more question:
is there any difference in backups created by regular job and by backup copy job?
Would it be possible to create a new GFS job, run it once, then delete created backups in repository and move my one full and 30 incrementals to that folder and then trick Veeam somehow (edit vbm file or sql tables; change time on the server to go back), so it'll take my old backups and will start to transform them according to GFS job policy (let's say with 30 recover points)? Sounds like super dirty trick, but who knows...

[dellock6]

not sure the "super trick" could work, but in terms of file format, both backup and backup copy job create a vbk file with the same format.

[Shestakov]

Anton,
What problem are you going to solve with the trick?
You can just start a new job with GFS retention and keep the chain you described with it`s own proper retention. Thanks.

[AntonZ]

The problem I have is that I have a set of files left from one of my GFS job. Vbm file shows 0 bytes and there is no such job in Veeam. So I just have some fulls and many incrementals. I need to convert those incrementals into fulls.

[Shestakov]

Thanks for the answer.
I`m just curious how converting increments into fulls would help you to save the space?
As I wrote above you can just split your historical backups between 2 repositories if you have a lack of space.

[AntonZ]

I'm going to offload data to amazon (I need roughly more than 100TB). Currently I have 13 fulls and 90 incrementals in that folder. I need to have all fulls, so I'll not have a headache in future when I need to recover something from amazon. Obviously, I can't move the whole chain with incrementals to the cloud as when I need to recover smth in the middle, I'll have to get the whole chain.

[Shestakov]

I see your point. Let`s wait for A.Gostev`s answer and see if developers can help with that.

[AntonZ]

Hi, Anton. There a very slim chance that you can do this through internal data mover command line, unless the process includes some external logic from .NET parts of code. I will be able to give you exact answer next week, when I am in the office and can talk to the devs. Thanks!

Hi Gostev!
anything on this?

[cmaier]

Hi,

We use incremental backups for all our jobs and transform the previous chains into rollback on weekend. But this is done job by job so when there are several jobs transforming their chains the others have to wait for start until the first have finished the transformation.

It lasts half a day until some jobs start with their backups in our environment. So it would be nice to finish the incrementals for all jobs first and transform the chains after that.

Thanks
Best regards
Christian

[foggy]

Christian, are you chaining you jobs so that the job started after the previous one has finished?

[ChrisL]

Hi all, don't know if this is even practical, but I occasionally find the need to re-run a job with a transform on a day that's not set to actually run a transform. Perhaps the previous transform failed, for example, but I don't want to wait until the next scheduled transform in a few days time to retry it since by this time the forward-chain is getting a bit long and the transform will take that much longer.

Wonder if it's possible, on the backup job right-click menu in B&R, where there's the option to 'Start' and 'Active Full' to add an option to 'Run with transform' or maybe just 'Transform now'.

Without this, if I want to run a transform job ad-hoc, we need to dig through the job settings to tick the current day in the Transform options, run the job and then remember to return to un-tick that day so that it doesn't go ahead and do it again the following week. All entirely doable, but just a bit messy.

Might be a silly idea, and obviously not expecting you to develop an option that I'm the only person in the world would use (!) but perhaps there's others out there in a similar position..

[Shestakov]

Hi Chris!
Are we talking about synthetic full run?
Thanks for the feedback!

[ChrisL]

Hi, yes that's right, sorry missed that bit out!

Perhaps I should rephrase that as an option to run an ad-hoc Synthetic Full now, in the same way as you can run an Active Full with a right-click. The transform will presumably then just follow automatically.

[futureweb]

For testing purposes (getting errors with synthetic full), it would be very helpful to force a synthetic full.

The problem is,if you have a full today, do incs today, the "synthetic full" is ignored for today, so to be able to test, I have to wait until tomorrow.
Thats why a force would be helpful.

[veremin]

The feature request is noted.

If don't mind using PowerShell, take a look at the script provided in this thread.

Thanks.

[futureweb]

I read this thread already, but it doesn't do any different as if I set days in the GUI.
As it also won't do a synthetic full, if already a full was done on the same day (to note, I tried it with the powershell commands already, just to be sure).

For automatic tasking (as if you don't want weekly synthetic full, the script would work fine), not for testing if you search for an error. We go for weekly synthetic.

Thanks
Patrick

[olafurh]

Hi,

While running backups in full (veeam!) speed over the night, you are putting a lot of write load on your disk arrays, both in terms of IO and throughput. It would be smart, to have the option to defer the merge process on some jobs (the jobs could contain big BI analytic machines with terabytes of changed data every night for example) to another window than the backups are running in. It would put less load on both arrays and repositories servers when they should be busy writing down million dollar data and get a better utilization out your infrastructure investment in a less busy window.

ps, yes, or use refs and utilize fast clone...

olafurh

[nmdange]

The heavy IO on merge happens on the backup repository, not your production array. So if you're following best practices, that would be a completely separate device from your production environment. It would be very bad if your backups were comingled with your production data.

[Poweruser]

anyway the feature could be derived to another idea:
make inc. backups using high priority with max. resources.
and then the later process like merging could be background process low prio only.
so it wont eat up your resources if you plan to do a backup during worktime.

[olafurh]

yes I know that the merge process dose not happen on the same arrays/disk boxes as the production vm data live But to ease up the load on the backup repository spindles/arrays would be nice by be able to defer the merge out side of a backup window.

[Poweruser]

so you can finish backups fast and rebuild reverse incrementals on standby/background

[Reklov]

Hi,
how can I start a synthetic full without or seperate from a Job ?

I' starting a Job by script and setup some resources into a specific "Backupmode". The Incrementaljob runs only 20 Minutes, so my resources can be moved back to non Backupmode, but this will be done at the end of the job ( defined a synthetic full every Saturday ) by a script. The synthetic full takes a lot of time. During this, the backupmode is not nessesary to may Resources ( they dont work in Backupmode ).

So I need to start the creation of a synthetic full separate.

Any possibility ?

Thanks,
Volker

[foggy]

Hi Volker, please review this thread for possible options. Thanks!

[odge]

+1

[Reklov]

@foggy
thanks for this.
Unfortually I'nd found any Possibility to start only a synthetic Full, without starting the according Backupjob ( like in the PS Script ).
The only thing I want to do is to fire out a command to start a synthetic full, based on the incrementals without add a new incremental.

Any idea ?

[foggy]

This is currently not possible.