Something probably isn't right - too many backup files

[Dr.Gerry]

Our Veeam installation has been sitting there, working away quietly for a very long time. It's usually pretty reliable - the 'create synthetic full backups' is working, the 'copy to tape' is working, restores work. It's been so long since it was set up that I have forgotten a lot of the finer points of how it works, and it has been a long time since there was any real human intervention required....

Anyway, this morning I got a message that we are running low on space on our local repository. I checked it out and either something isn't quite right, or in the time since the server last required manual intervention I have just forgotten how Veeam works. I know that sounds pretty silly, but here's my problem.

All our backup jobs are set to retain 14 restore points. Each week each job creates a synthetic full backup, which is then copied to tape. I THOUGHT this should mean that we have 1 'vbk' and 14 'vib' files for each backup job, but that doesn't seem to be the case. In most cases there's 4 vbk files and about 15 vib files for each backup job. I'm guessing this is probably expected behaviour as that's what most of our jobs have (and I plan on getting out the manual and doing a quick refresher immediately after posting this!). However, there's 3 or 4 jobs that have 1 vbk file and 60-odd vib files. I have checked these jobs, and they are all set the same - ie keep 14 restore points and do a weekly synthetic full.

So that's probably why my repository is almost full. My question now is - what do I do about the jobs that have 1 vbk file and 60 vib files? How do I get these jobs to work properly and get rid of some of the extraneous vib files?

[Dr.Gerry]

OK.....this is a bit weird - I think I might be answering my own question.........

There's an awful feeling of Deja Vu about this. Looking a bit more closely at the backup jobs I can see that every job has 'create synthetic full backups periodically' ticked, and every job has 'Saturday' as the nominated day. Unfortunately as our amount of storage has grown over the years it means that some of the backup jobs are still running on Sunday. It used to be that all our jobs were finished by Saturday morning, but now some of them don't run until Sunday......hence a 'create a synthetic full on Saturday' is never going to happen......hence we have 1 vbk file and a million vib files......or at least I assume this is what is happening.

Now for my next dumb question......is there any way to have the 'create synthetic full' step run after another job, rather than nominating a day - in the same way that you can say 'start backup job B after backup job A has finished' can you say 'start the synthetic full backup after this other job has finished'......or should I just tick Sunday as well as Saturday on the 'create synthetic full' option?

[Dr.Gerry]

Did I mention that I really hate Mondays?

[Virtuollie]

In your case, I would reconfigure schedules, so Jobs won‘t be chained and can all run on Saturday.
Checking both days, Saturday and Sunday could lead to two VBKs created on the same Weekend.

[PetrM]

Hi Gerry,

is there any way to have the 'create synthetic full' step run after another job

The only option is to use "after this job" but you will need to specify the day for synthetic anyway.

or should I just tick Sunday as well as Saturday on the 'create synthetic full' option?

Would you be so kind to clarify why do you want to tick the same option twice instead of scheduling synthetic to run once per week?

Thanks!

[Dr.Gerry]

Hi PetrM…...I don't particularly want to tick the box twice......I just want my synthetic full jobs to run once a week as planned. As is probably fairly obvious from my posts, I don't really have a great understanding of how Veeam works.....neither do I want to. I just want backup software where you say "backup these things" and it just does it for you.....something goes wrong and you click Restore This, and it just happens!
That is sort of what happens here......we had the system running pretty well for a number of years.....very little intervention required, but as our number of servers grew, and the amount of data being backed up grew, jobs started taking longer and longer. All our backups, which all had the Synthetic Full option selected for Saturday started to take longer and eventually we reached a point where jobs wouldn't run until Sunday, and hence wouldn't create a Synthetic Full. I just assumed that ticking 'create synthetic full...' would somehow schedule something in Veeam that would kick off a job that creates a synthetic full on the selected days. Somehow this 'magic' behind the scenes seemed to work just fine for quite a while, I just didn't understand the process properly, but it still worked.
To me, the idea of saying 'this job runs at 6 pm, this one runs at 6.30, this one at 7' is really bad. Years ago we used HP Data Protector. You had to specify start times for every backup job you ran. Invariably something would happen that would delay a job, which would have a flow on effect and we would get jobs that didn't run because they had been sitting in the queue waiting to run but the previous jobs took too long to complete, so they timed out. When we got Veeam, and you could just say 'run this job, when it is finished run this next job, and when that is finished run the next job' was a revelation.....we never again missed a backup because it had been sitting in a queue for longer than it's timeout period.
I suppose it's pretty stupid, but I didn't realise that the create synthetic full option was actually part of the backup job itself...….as I said I just assumed that this somehow scheduled something internally in Veeam that would kick off a separate process.....you know, like on a Saturday the "which machines need a synthetic full created" process would be triggered, creating the synthetic full backups.
I guess I'd better sit down with the manual!

[foggy]

This is the typical consequence of job chaining. Instead, just schedule them all one after another with short intervals and make sure you have proper concurrent tasks limits for proxy and repository servers, automatic resource scheduler will do its job. In this case, synthetic fulls will occur regardless of when the VMs in the job actually start to be processed since all the jobs will be started on Saturday anyway, just waiting for the available resources.

[backupquestions]

https://www.veeam.com/blog/advanced-ref ... suite.html

You might look at redoing your repos with REFS and benefit from much faster synthetic full backups as well. You could see several hours go down to 1 hour, or hours to minutes even.

[Dr.Gerry]

Now I'm really confused! When I made that comment earlier about assuming some sort of internal process thing that kicks off a Synthetic Full backup on the scheduled day I thought it sounded pretty silly as I was writing it......but I have just gone back through the manual to try to fill in some of the gaps in my understanding.....and I read this :

"Veeam Backup & Replication automatically triggers a backup job session to create a synthetic full backup, even if a regular backup job session is not scheduled on this day. The job session is started at the same time when the parent backup job is scheduled."

So it actually is some sort of magic thing that kicks off the process for a Synthetic Full on the day nominated.......

In my case, is it not working because I am using job chaining rather than specifying a scheduled start time for each job? ie if I set Job 1 to start at 6 pm Mon-Fri, job 2 to start at 6.10 pm and job 3 to start at 7 pm does that mean that on Saturday (if that was the nominated day) the Synthetic full for job 1 would automatically run at 6 pm, for job 2 at 6.10 pm and job 3 at 7 pm?

As I said previously I have had nothing but bad experience with scheduling backup jobs at set times, so can someone absolutely assure me that if I specify start times for all my backup jobs, rather than chaining them, that they won't sit on the queue and time out?

[Gostev]

I can as this is how almost everyone does it with Veeam, anyway. Chaining jobs was always considered to be a bad practice.

[ITP-Stan]

Just make sure to check your backup repository concurrent tasks limit and adjust it as needed.
For example, on slow backup repositories it might not be a good idea to have 4 concurrent tasks running on it.

Also +1 for ReFS 64K on WIN2016+ for fast synthetic fulls.

[Dr.Gerry]

Given that participating in this discussion has allowed me to see the obvious gaps in my Veeam knowledge, here's another situation I have that probably demonstrates another lacking piece of my knowledge!!
One of my backup jobs hadn't been running it's synthetic full for a while (it was one of those jobs that had 'Saturday' ticked for synthetic full creation, but the job didn't actually run until Sunday, so it never made a synthetic full). It is set to retain 14 restore points, but it currently has 127 restore points. I scheduled a synthetic full last night (which worked successfully) and I thought this would get rid of a lot of those restore points, but it hasn't. For this job there is now a VBK from Aug27 last year, followed by a VIB for every day from then until January 15, another VBK from January 21 (when I initially tried running a synthetic full) and another vbk from last night when I ran a synthetic full again.
This is starting to drive me crazy(er)! How do I get rid of all these restore points and reclaim some disk space in my repository?

[veremin]

The obsolete restore points will be removed on February 4th (January 21 + 14 restore points).

If you want to remove them now, set retention to something low (3, for instance) and run the job two times. This will leave you with 3 restore points.

Thanks!

[Dr.Gerry]

Thanks for your response.

OK, that sounds reasonable. I'll keep an eye on our disk space and if we are going to be able to last until then I'll just let processes take their normal course until Feb 4.

There's still one thing I don't understand though......I ran a successful Synthetic Full of this server on January 20, but that left every vbi file intact.....so we had the vbk file from January 20, and then every previous vbi file stretching back to August 28 last year. Shouldn't most of them have been removed after the successful run on Jan 20?

[foggy]

Hi Gerry, please review this KB for a visual description of how retention works for forward incremental backup method. Should give you the right understanding. Thanks!

[ITP-Stan]

All those VIB's, up until the last one dated Jan 19th are dependent on that VBK from Aug 28th.
They are an incremental chain, so everyone of the files is needed for the chain.
So the last VIB made on Jan 19 needs that VBK from Aug 28th and all those VIB's in between.

Only when your retention settings allow the Jan 19th VIB to be deleted, will that also delete the VBK from Aug 28th and all those VIB's in between.
Because only then they aren't needed anymore.

If you can't wait do as @veremin suggests and lower the retention.

[ITP-Stan]

The Animation that Foggy linked to is for forward incremental. In which case the oldest VIB gets injected in to the full VBK according to the retention setting.
That is not the case when you use weekly (virtual or active) full. In that case my above explained logic applies.

[foggy]

The linked KB actually contains animations for both forever forward incremental and forward incremental methods.