Backup files not deleted according to retention policy

[habibalby]

But remember, synthetic + transform puts a high load on the backup storage, generating 4 I/O for every bit taken from production storage.
Luca.

Hi Luca,

I agree with you, Synthetic + Transform in Forward Incremental for small VMs from 20 GB till 700 GB it was fine with me, I managed to back most of the VMs in that way. But since it hanged for more than 40 Hours just waiting for the job to finish of 1.4TB of one VM, then I changed my mind to stick with Only Forward Incremental, with 7 Days Retention of both Restore Points and VMs Deleted from Disk. I thought with this configuration, I will have full one week backup of 1 VM as I have explained in the above threads. But in reality it's not, it Runs Full on Friday, Sun-Thu incremental, but next week, it repeats the same without deleting the old VBK and VIBs files. This kept me in bad shape with regards of Storage. Good you brought it back to my mind, I will try to run some of the Jobs configured in that way Synthetic+Transform and some of them with Reversed Incremental for the Big VMs.

Thanks,
S.Hussain

[habibalby]

I would also suggest the following enhancement to keep Veeam top Virtualization Backup Product among the rest that available in the market.

1. Multiple job settings for one particular VM. For instant, I can configure Full Backup Job without selecting Incremental which is currently a default setting and must be selected. This should be taken off to have more flexibility of the Backup files.
2. Creating separate Incremental Job / Forward Incremental or Reversed Incremental depends on the customer requirement. This will make sure that files are written to tapes are Full Backup or are Incremental backup. I can bring one example in here which is when I Backup VM in Reversed Incremental Mode, I'm forced to backup only Full Backup without incremental this because the Archive-Bit is resetting from last the job runs. Which force the the product that backing up from Disk to tape to always backup in Full Mode. Agreed for small number of VMs, but when it comes to VMs that have got TBs of Data, I would say Veeam Sorry, I can't backup this VM in Reversed Incremental Mode as it keep changing the original VBK file in every Incremental runs. Thus, the customers can't backup the whole VM and the customer forced to go back to the Legacy Backup application to back his VMs.

Or Veeam.com should do something on how the Reversed Incremental backup works. Reversed Incremental is good and in my onion works better than Forward Incremental specially for those who lacks of disk space to backup to disk, the Retention Policy works perfect, unlike Forward Incremental Backup. If the Reversed Incremental works without changing the main VBK file to keep track of the last Archive Bit when was reset by the Backup to Tape product, this will save us from running full backup from disk of very BIG VMs.

With Forward Incremental, if Veeam.com get rid of doubling the VBK Files during the second week of Full Backup this will keep backing up to Tape is a great choice in selecting this mode of Backup to Disk and then to Tapes.

3. Veeam.com should introduce Back-to-Tape integration so, VMs directly will be processed to tapes, I know it's a bit slower but it's a safest option customer can adopt.

In this scenarios I have only few option:

Sticking to Reversed Incremental Backup with 5 Days Retention. Sat Full Backup and Sun-Thur Incremental. And on Friday, I have to backup the whole backup of Veeam to Tape. This put me at risk of keeping one week backup of data on disk, taking into consideration that if SAN Storage failed, and I didn't do a backup for last 1 week, the whole things would lost. Beside that, I have the option to go back to my legacy Backup Application to backup the VMs which doesn't leave me with no Free Backup Windows. As backing Reversed Incremental Jobs on a weekly basis takes time to finish the job, plus I have to run the legacy backup application to finish some backup jobs.

Veeam.com, please do something regarding the above suggestion to keep the customer using Veeam Backup in more flexible way.

Thanks,

[Vitaliy S.]

But in reality it's not, it Runs Full on Friday, Sun-Thu incremental, but next week, it repeats the same without deleting the old VBK and VIBs files.

The old restore points are not being deleted because you set the job to keep 7 restore points, which in turn requires us to keep whole previous chain (as the last 7 incremental restore points are dependent on previous full and all increments).

In other words the oldest file that have to be deleted (according to the retention policy settings) will be previous VBK, which cannot be removed because all subsequent restore points will become unusable. This will put you out of compliance with your own retention policy, where you said you want to keep previous 7 restore points always available.

Hope this makes sense.

If the Reversed Incremental works without changing the main VBK file to keep track of the last Archive Bit when was reset by the Backup to Tape product, this will save us from running full backup from disk of very BIG VMs.

Not sure how you cannot change VBK file if all recent VM changes have to be rolled into this file during each job run.

[habibalby]

Okay, what is the minimum Retention Policy I can set with Forward Increments? can I keep only 5? this will allow me to do full back on Sat and Sun-Thursday Incremental. On Next Saturday when it runs, it will create a new VBK file and it. On Sun when it runs will delete previous week Sunday.VIB file. will this be achievable? What I'm explaining is will allow me to have 5 Veeam Files, 1 VBK and 4 VIBs. But veeam understands this as 5 Retention period.

The old restore points are not being deleted because you set the job to keep 7 restore points, which in turn requires us to keep whole previous chain (as the last 7 incremental restore points are dependent on previous full and all increments).

In other words the oldest file that have to be deleted (according to the retention policy settings) will be previous VBK, which cannot be removed because all subsequent restore points will become unusable. This will put you out of compliance with your own retention policy, where you said you want to keep previous 7 restore points always available.

Hope this makes sense. Not sure how you cannot change VBK file if all recent VM changes have to be rolled into this file during each job run.

Technically you are right, but there must be a way that Veeam can persevered the changes of when last Full backup runs on the VBK file, then save them somewhere when the Reversed Incremental jobs finish and injected the VRB into VBK again re-commit the state of when last full backup runs on the VBK file. If it works, I belive it will sort out the Reversed Incremental Full Backup and Daily Incremental backup to tapes.

Thanks,

[Vitaliy S.]

You can set retention policy to 1, this will allow you to remove previous backup chain as soon as you create a new full backup (it might be either synthetic or active full backup).

[foggy]

Yes, just be sure you've offloaded it to tape before that. Isn't that what you are looking for?

[habibalby]

You can set retention policy to 1, this will allow you to remove previous backup chain as soon as you create a new full backup (it might be either synthetic or active full backup).

Retention Policy = 1, this apply only on full VBK Files? what about daily VIBs "Incremental" files from Sun-Thur, they will be deleted after 1 Day also?

Thanks,

[Gostev]

Incremental backups (VIB) simply cannot be deleted until full backup (VBK) that they are dependent on is deleted. This is the basics of how incremental backup works... as long as you need the latest restore point in the incremental chain per your retention policy, the whole chain must be kept. When you no longer need the latest restore point in chain, the whole chain is removed at once.

[habibalby]

Yeah, but If I set the Retention Policy to 1 will that mean 1 VBK file an 1 VIB file? Or will always maintain only 1 VBK file?

[Gostev]

This depends on your full backup schedule, of course. Please understand until you have the new VBK, you cannot remove the old VBK, because the following incremental backups are dependent on the old VBK. This is in no way specific to Veeam, this is how incremental backup is designed to work!

[dellock6]

To summarize, retention policy is referred to every retention point, regardless it's a VIB or VBK. VBK must be kept even if it's outdated until the VIB releting to it are into the retention period.

About other posts, the error many customers do is choosing a cheap NAS for backup storage, and then to further save on space, they run reverse incremental on it. The higher requested IO on this backup method often hose the storage, and people think it's Veeam fault. The best solution to keep a low budget is to have a large and slow SATA-based storage, and use forward incremental "without" synthetic nor transform. K.I.S.S. method.

[habibalby]

To summarize, retention policy is referred to every retention point, regardless it's a VIB or VBK. VBK must be kept even if it's outdated until the VIB releting to it are into the retention period.

This point made it clear. Thank you

[arsprod]

so, what's the best way to clean this up? I've switched from forward to reverse incremental, the job's run several times. Do I have to manually delete the old files or will Veeam eventually get rid of them?

[foggy]

Aaron, what are your retention settings and what files do you have in the backups folder? Normally, all unnecessary restore points should be wiped out according to your retention settings.

[arsprod]

I was running incrementals, 14 restore points. After seeing I had up to 35 backups (and full drives) I switched to reverse incremental, same number of restore points. I'm not seeing the number of restore points shrinking (or files disappearing). It's all vm's. What am I missing?

Also, deleted vms data retention period set to 2 days

[Vitaliy S.]

Hi Aaron, how many VRB/VIB/VBK files do you have in the backup repository?

[arsprod]

One of them has 37 vib/vbk files

[Vitaliy S.]

If you have only one VBK file then it explains why the whole previous backup chain has not been removed yet.

You need to wait till the number of available restore points satisfies the configured retention policy settings, after that you will have 14 restore points present on the datastore for jobs using reversed incremental backup mode.

[arsprod]

OK, so then if i set retention days to 1 then when it hits 14 restore points it will delete everything else?

[foggy]

Seems like you're mixing up with the Deleted VMs retention period, which is a separate setting.

[arsprod]

Well, thought they were related. What I'm understanding is that now that it's set for reverse incrementals, when it gets to number of restore points it will delete anything older then 14 + 1 (1 day to retain deleted files). I'm I incorrect?

[Vitaliy S.]

Retention policy determines the number of restore points (not days), so the retention policy will be applied once the number of available restore points reaches 15 (only 14 points will be kept).

[arsprod]

right, sorry - I only run backups once a day. thanks

[dbrewood]

Case ID: 00872102

Looks like I'm joining the club. We are using incremental backup with the system set to keep 6 restore points on disk. However over the weekend we've now grown top 3 x 960gb VBK files as well as the daily VIB files (circa 10gb each) and have now run out of disk space (2.5Tb reserved). Ideally we wish to keep 2 full VBK files on disk and the incrementals. We need to know how to achieve the best configuration.

From what I can tell from the way our installation consultant set the system forward incremental is being used, with no synthetic generation being carried out.

[foggy]

Daron, with forward incremental mode, depending on your schedule, you can have more restore points on disk than you've specified in the job setting.

[dbrewood]

Yeah I've been reading hard and come to that conclusion. I've worked with our consultant and we've given the disk another 1Tb of SAN storage and we'll see if it cleans up tonight. We're working on the basic that VEEAM failed to create the third full backup properly due to lack of disk space and as that failed it did not clear out the initial weeks full backup set (full & incrementals).

It does seem very difficult to achieve what we want which is a weekly full backup on a Friday night, followed by daily incremental backups Monday through to Thursday, that to repeat for a second week and then on the third week the initial backup set be deleted and a fresh one created.

The consultant setting retention to 6 does not make sense and that is what I guess is throwing me. I could see it being 2 (as in 2 full backups) or 10 (2 full sets of 1 full backup and 4 incrementals) - whereas 6 seems somewhat in the middle.

It's not helped that with our old VEEAM system the retention figure was set to 2 and that did work to maintain a weekly full backup on a Friday night, followed by daily incremental backups Monday through to Thursday.

It all seems highly illogical

I've spent all day reading VEEAM KB files and the manual and I'm still no wiser as to what we should set as a configuration to get what we need or how to validate that 6 figure to be correct.

I'm quite happy to share all our config date if anyone can solve the mystery

[foggy]

What is probably confusing is that whether you set it to 2 or 6 the result is still the same, as the chain continues until the new full is created on Friday. This is just how forward incremental mode works (no way to delete old restore points until the number of restore points in the new chain reaches the specified setting). Basically, it is up to you how far back in time you need to be able to recover and Veeam B&R just ensures you always have at least that number of restore points. If you want to have forward chain with fixed number of restore points on disk, consider using forever forward incremental mode.

[dbrewood]

From what you are showing in that link that is how we are set apart from we have it also set to create an active full backup on a Friday night, like this.

[foggy]

Yep, enabling either synthetic or active full, switches Veeam B&R to a forward incremental backup mode.

[dbrewood]

So we are set correctly then, therefore could it have just been that it did not have enough disk space to do the creation of the third full backup file prior to deleting the first backup set?