Backup files not deleted according to retention policy

Availability for the Always-On Enterprise

Re: Backup files not deleted according to retention policy

Veeam Logoby habibalby » Mon Mar 12, 2012 5:34 am

dellock6 wrote:But remember, synthetic + transform puts a high load on the backup storage, generating 4 I/O for every bit taken from production storage.
Luca.


Hi Luca,

I agree with you, Synthetic + Transform in Forward Incremental for small VMs from 20 GB till 700 GB it was fine with me, I managed to back most of the VMs in that way. But since it hanged for more than 40 Hours just waiting for the job to finish of 1.4TB of one VM, then I changed my mind to stick with Only Forward Incremental, with 7 Days Retention of both Restore Points and VMs Deleted from Disk. I thought with this configuration, I will have full one week backup of 1 VM as I have explained in the above threads. But in reality it's not, it Runs Full on Friday, Sun-Thu incremental, but next week, it repeats the same without deleting the old VBK and VIBs files. This kept me in bad shape with regards of Storage. Good you brought it back to my mind, I will try to run some of the Jobs configured in that way Synthetic+Transform and some of them with Reversed Incremental for the Big VMs.

Thanks,
S.Hussain
habibalby
Expert
 
Posts: 350
Liked: 23 times
Joined: Mon Jul 18, 2011 9:30 am
Location: Bahrain
Full Name: Hussain Al Sayed

Re: Backup files not deleted according to retention policy

Veeam Logoby habibalby » Mon Mar 12, 2012 7:30 am

I would also suggest the following enhancement to keep Veeam top Virtualization Backup Product among the rest that available in the market.

1. Multiple job settings for one particular VM. For instant, I can configure Full Backup Job without selecting Incremental which is currently a default setting and must be selected. This should be taken off to have more flexibility of the Backup files.
2. Creating separate Incremental Job / Forward Incremental or Reversed Incremental depends on the customer requirement. This will make sure that files are written to tapes are Full Backup or are Incremental backup. I can bring one example in here which is when I Backup VM in Reversed Incremental Mode, I'm forced to backup only Full Backup without incremental this because the Archive-Bit is resetting from last the job runs. Which force the the product that backing up from Disk to tape to always backup in Full Mode. Agreed for small number of VMs, but when it comes to VMs that have got TBs of Data, I would say Veeam Sorry, I can't backup this VM in Reversed Incremental Mode as it keep changing the original VBK file in every Incremental runs. Thus, the customers can't backup the whole VM and the customer forced to go back to the Legacy Backup application to back his VMs.

Or Veeam.com should do something on how the Reversed Incremental backup works. Reversed Incremental is good and in my onion works better than Forward Incremental specially for those who lacks of disk space to backup to disk, the Retention Policy works perfect, unlike Forward Incremental Backup. If the Reversed Incremental works without changing the main VBK file to keep track of the last Archive Bit when was reset by the Backup to Tape product, this will save us from running full backup from disk of very BIG VMs.

With Forward Incremental, if Veeam.com get rid of doubling the VBK Files during the second week of Full Backup this will keep backing up to Tape is a great choice in selecting this mode of Backup to Disk and then to Tapes.

3. Veeam.com should introduce Back-to-Tape integration so, VMs directly will be processed to tapes, I know it's a bit slower but it's a safest option customer can adopt.

In this scenarios I have only few option:

Sticking to Reversed Incremental Backup with 5 Days Retention. Sat Full Backup and Sun-Thur Incremental. And on Friday, I have to backup the whole backup of Veeam to Tape. This put me at risk of keeping one week backup of data on disk, taking into consideration that if SAN Storage failed, and I didn't do a backup for last 1 week, the whole things would lost. Beside that, I have the option to go back to my legacy Backup Application to backup the VMs which doesn't leave me with no Free Backup Windows. As backing Reversed Incremental Jobs on a weekly basis takes time to finish the job, plus I have to run the legacy backup application to finish some backup jobs.

Veeam.com, please do something regarding the above suggestion to keep the customer using Veeam Backup in more flexible way.

Thanks,
habibalby
Expert
 
Posts: 350
Liked: 23 times
Joined: Mon Jul 18, 2011 9:30 am
Location: Bahrain
Full Name: Hussain Al Sayed

Re: Backup files not deleted according to retention policy

Veeam Logoby Vitaliy S. » Mon Mar 12, 2012 8:21 am

habibalby wrote:But in reality it's not, it Runs Full on Friday, Sun-Thu incremental, but next week, it repeats the same without deleting the old VBK and VIBs files.

The old restore points are not being deleted because you set the job to keep 7 restore points, which in turn requires us to keep whole previous chain (as the last 7 incremental restore points are dependent on previous full and all increments).

In other words the oldest file that have to be deleted (according to the retention policy settings) will be previous VBK, which cannot be removed because all subsequent restore points will become unusable. This will put you out of compliance with your own retention policy, where you said you want to keep previous 7 restore points always available.

Hope this makes sense.
habibalby wrote:If the Reversed Incremental works without changing the main VBK file to keep track of the last Archive Bit when was reset by the Backup to Tape product, this will save us from running full backup from disk of very BIG VMs.

Not sure how you cannot change VBK file if all recent VM changes have to be rolled into this file during each job run.
Vitaliy S.
Veeam Software
 
Posts: 19960
Liked: 1145 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup files not deleted according to retention policy

Veeam Logoby habibalby » Mon Mar 12, 2012 8:46 am

Okay, what is the minimum Retention Policy I can set with Forward Increments? can I keep only 5? this will allow me to do full back on Sat and Sun-Thursday Incremental. On Next Saturday when it runs, it will create a new VBK file and it. On Sun when it runs will delete previous week Sunday.VIB file. will this be achievable? What I'm explaining is will allow me to have 5 Veeam Files, 1 VBK and 4 VIBs. But veeam understands this as 5 Retention period.

Vitaliy S. wrote:
habibalby wrote:But in reality it's not, it Runs Full on Friday, Sun-Thu incremental, but next week, it repeats the same without deleting the old VBK and VIBs files.

The old restore points are not being deleted because you set the job to keep 7 restore points, which in turn requires us to keep whole previous chain (as the last 7 incremental restore points are dependent on previous full and all increments).

In other words the oldest file that have to be deleted (according to the retention policy settings) will be previous VBK, which cannot be removed because all subsequent restore points will become unusable. This will put you out of compliance with your own retention policy, where you said you want to keep previous 7 restore points always available.

Hope this makes sense.
habibalby wrote:If the Reversed Incremental works without changing the main VBK file to keep track of the last Archive Bit when was reset by the Backup to Tape product, this will save us from running full backup from disk of very BIG VMs.

Not sure how you cannot change VBK file if all recent VM changes have to be rolled into this file during each job run.


Technically you are right, but there must be a way that Veeam can persevered the changes of when last Full backup runs on the VBK file, then save them somewhere when the Reversed Incremental jobs finish and injected the VRB into VBK again re-commit the state of when last full backup runs on the VBK file. If it works, I belive it will sort out the Reversed Incremental Full Backup and Daily Incremental backup to tapes. :)

Thanks,
habibalby
Expert
 
Posts: 350
Liked: 23 times
Joined: Mon Jul 18, 2011 9:30 am
Location: Bahrain
Full Name: Hussain Al Sayed

Re: Backup files not deleted according to retention policy

Veeam Logoby Vitaliy S. » Mon Mar 12, 2012 9:52 am

You can set retention policy to 1, this will allow you to remove previous backup chain as soon as you create a new full backup (it might be either synthetic or active full backup).
Vitaliy S.
Veeam Software
 
Posts: 19960
Liked: 1145 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup files not deleted according to retention policy

Veeam Logoby foggy » Mon Mar 12, 2012 9:56 am

Yes, just be sure you've offloaded it to tape before that. Isn't that what you are looking for?
foggy
Veeam Software
 
Posts: 15289
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backup files not deleted according to retention policy

Veeam Logoby habibalby » Mon Mar 12, 2012 9:56 am

Vitaliy S. wrote:You can set retention policy to 1, this will allow you to remove previous backup chain as soon as you create a new full backup (it might be either synthetic or active full backup).


Retention Policy = 1, this apply only on full VBK Files? what about daily VIBs "Incremental" files from Sun-Thur, they will be deleted after 1 Day also?

Thanks,
habibalby
Expert
 
Posts: 350
Liked: 23 times
Joined: Mon Jul 18, 2011 9:30 am
Location: Bahrain
Full Name: Hussain Al Sayed

Re: Backup files not deleted according to retention policy

Veeam Logoby Gostev » Mon Mar 12, 2012 10:01 am

Incremental backups (VIB) simply cannot be deleted until full backup (VBK) that they are dependent on is deleted. This is the basics of how incremental backup works... as long as you need the latest restore point in the incremental chain per your retention policy, the whole chain must be kept. When you no longer need the latest restore point in chain, the whole chain is removed at once.
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Backup files not deleted according to retention policy

Veeam Logoby habibalby » Mon Mar 12, 2012 10:04 am

Yeah, but If I set the Retention Policy to 1 will that mean 1 VBK file an 1 VIB file? Or will always maintain only 1 VBK file?
habibalby
Expert
 
Posts: 350
Liked: 23 times
Joined: Mon Jul 18, 2011 9:30 am
Location: Bahrain
Full Name: Hussain Al Sayed

Re: Backup files not deleted according to retention policy

Veeam Logoby Gostev » Mon Mar 12, 2012 10:09 am

This depends on your full backup schedule, of course. Please understand until you have the new VBK, you cannot remove the old VBK, because the following incremental backups are dependent on the old VBK. This is in no way specific to Veeam, this is how incremental backup is designed to work!
Gostev
Veeam Software
 
Posts: 21611
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Backup files not deleted according to retention policy

Veeam Logoby dellock6 » Mon Mar 12, 2012 3:41 pm

To summarize, retention policy is referred to every retention point, regardless it's a VIB or VBK. VBK must be kept even if it's outdated until the VIB releting to it are into the retention period.

About other posts, the error many customers do is choosing a cheap NAS for backup storage, and then to further save on space, they run reverse incremental on it. The higher requested IO on this backup method often hose the storage, and people think it's Veeam fault. The best solution to keep a low budget is to have a large and slow SATA-based storage, and use forward incremental "without" synthetic nor transform. K.I.S.S. method.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5138
Liked: 1381 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Backup files not deleted according to retention policy

Veeam Logoby habibalby » Sun Mar 18, 2012 12:17 pm

dellock6 wrote:To summarize, retention policy is referred to every retention point, regardless it's a VIB or VBK. VBK must be kept even if it's outdated until the VIB releting to it are into the retention period.

This point made it clear. Thank you :)
habibalby
Expert
 
Posts: 350
Liked: 23 times
Joined: Mon Jul 18, 2011 9:30 am
Location: Bahrain
Full Name: Hussain Al Sayed

Re: Backup files not deleted according to retention policy

Veeam Logoby arsprod » Tue May 15, 2012 3:32 pm

so, what's the best way to clean this up? I've switched from forward to reverse incremental, the job's run several times. Do I have to manually delete the old files or will Veeam eventually get rid of them?
arsprod
Enthusiast
 
Posts: 86
Liked: never
Joined: Fri Jan 21, 2011 6:09 pm
Full Name: Aaron Spiegel

Re: Backup files not deleted according to retention policy

Veeam Logoby foggy » Tue May 15, 2012 4:15 pm

Aaron, what are your retention settings and what files do you have in the backups folder? Normally, all unnecessary restore points should be wiped out according to your retention settings.
foggy
Veeam Software
 
Posts: 15289
Liked: 1133 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backup files not deleted according to retention policy

Veeam Logoby arsprod » Tue May 15, 2012 8:06 pm

I was running incrementals, 14 restore points. After seeing I had up to 35 backups (and full drives) I switched to reverse incremental, same number of restore points. I'm not seeing the number of restore points shrinking (or files disappearing). It's all vm's. What am I missing?

Also, deleted vms data retention period set to 2 days
arsprod
Enthusiast
 
Posts: 86
Liked: never
Joined: Fri Jan 21, 2011 6:09 pm
Full Name: Aaron Spiegel

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 1 guest