Son-Father-Grandfather rotation suggestion

[bbricker]

They say a picture is worth a thousand words, and I got to thinking about this screen back from when I was playing with VMware VDR. This is the kind of thing (although maybe with more options, such as a box for "daily" instead of "recent") that would be great to see in Veeam, at least as an optional alternative to the standard scheduling:

[oki7536]

Hi, I want to choose forward incremental backup and then every week backup the whole chain (first full backup + vib's) to tape.
When I use retention policy of '1', it should delete the whole chain (vbk + vib's) when the first synthetic full is performed, right? My questions now, does veeam delete the chain before the synthetic full is performed or after?
This makes a huge difference in storage.
On every Sunday I do a full synthetic (VBK file)
on Mon till Friday I run incremental (forward). (5 vib files)
I set retention to 1
I backup to tape on Saturday (1 vbk + 5 vib files)
On Sunday again, the full synthetic will run BUT will have no space enough on the target till the previous weekly chain has been deleted.

So does this retention policy work for me or should I manually (or via script) delete the chain in order to free-up space on the target for the next synthetic full ?

[foggy]

When I use retention policy of '1', it should delete the whole chain (vbk + vib's) when the first synthetic full is performed, right?

Correct.

My questions now, does veeam delete the chain before the synthetic full is performed or after?
...
So does this retention policy work for me or should I manually (or via script) delete the chain in order to free-up space on the target for the next synthetic full ?

Retention settings are applied after synthetic full is created so you always could have the specified amount of good restore points available.
Note, that if you remove the vbk file before the job is started, it will fail (unless you have the ForceCreateMissingVBK key set to 1).

[andrew.proctor]

[merged]

Just wondering whether something like the following is possible:

For one of our critical servers I'd like to take multiple backups during the day (perhaps one every hour between 9-5pm), but then at the end of the day, roll these up into a single restore point so that I don't have hundreds, so that I only keep one per day.

At the end of a month therefore I'd have 30 restore points, (not over 200).

I guess these intra-day restore points are more like recovery points and are something that you'd normally do on the SAN itself with snapshots, however I'm wondering whether I could use Veeam to do it as my Veeam backups go offsite.

Is this something I could do with 2 jobs? If I were to use 2 jobs, would I have to backup to separate locations? Perhaps this is not the most sensible thing to do, I'd be interested in what folk think

Cheers
Andrew

[dellock6]

Uhm, IF your storage architecture can stand this I/O stress, one possible idea could be:
- run a reverse incremental backup every hour, from 9 to 5 (using maybe powershell and task scheduler to run it). A possible alternative could be forward incremental during the day to lower the stress on storage, and do synthetic full at the end of the day
- after the last execution of the day, you will end up both ways with a full VBK file. Instead of running another backup job, simply copy/move the file to an alternate location and delete all the files in the backup chain, so it will be empty and ready for another day.

This idea came to me in 2 minutes reading at your post, for sure you will need to further investigate and test if and how it can be done!

[antspants77]

Hi,

After reading these threads, the one with the screenshot that speaks a thousand words is exactly what we need for all our customers.
We have already found holes with just having x number of days worth of backups. We had a couple of requests for older data beyond the 90 days we already had.
4 dailies, 5 weeklies, 12 monthlies, 7 yearlies = 28 restore points for 7 years as opposed to the 90 restore points for the last 90 days. The former is much more useful and if you give clients the choice, they will choose this every time.

VMware and other backup products simply have a console asking how many days, weeks ,months , years you want to keep. Type in some numbers, done.

The issues with some of the multi-job suggestions are;

1. Keeping the same "Full" data multiple times (4 times more in the suggestions that will meet our requirements)
2. 4 times more jobs to manage and maintain. With tens of terrabytes and dozens of VMs, web already have 5 jobs that will turn into 20, just for backup)
3. Jobs are independent of each other, so running an end of month job, will inevitably conflict with a daily or weekly job and have 2 running on the same day.
Same goes for the Yearly job.

Can I suggest this as an urgent product request. If not here, then can you suggest how we get this pushed up the chain.
Veeam is an awesome product and this is the only thing, which is causing us grief across all our clients.
The selling point is that it gives your product instant "Compliance".


Cheers

Tony

[Bunce]

I can't see it happening any time soon. I suggested this more than 2 years ago after benefiting from it in previous products.

The storage format Veeam uses doesn't lend itself well to this type of feature - I'm sure if it were doable they would have implemented by now given how often its requested and how it lends itself to actual real-world requirements that businesses currently require. Veeam are generally quite responsive to feature requests.

[Gostev]

The truth is that it was not requested as often as some other features, but generally speaking, this was not a high priority because most of the customers who need this sort retention also have a requirement to store these backups off-site, so such rotation is achieved outside of our product (full backups are copied on tapes and preserved as long as needed). I do still see the use for this feature when your backup target is a disk based storage (specifically, deduplicating storage), and we do have plans to implement this functionality down the road as resources and priorities will allow.

[nixdorf]

[merged]

Hello,

is it possible to create a generation principle with veeam?

Incremental backup every Day
Full Backup every Week, avalible for 8 days
Full Backup every month, avalible for 6 month

Greetings Nixdorf

[Jhem]

[merged]

Hello everyone

I will admit that I am rather new in working with backup policies and I have some questions regarding the retention policy. I am about to setup a incremental backup with a full synthetic backup applied once a week. I want to keep 6 restore points as I need a keep a daily backup for a week (6 days incremental and 1 day with a full synth). But for how long will the full synthetic backup be stored?

My idea is following

Daily x 6 incremental (monday to saturday)
Weekly x 4 full (each sunday)
Monthly x 11 full (01/xx/xxxx)
Yearly x 1 full (01/01/xxxx)

Do i need to make a job for each period as stated above and leave the full synthetic backup disabled?

Best regards
Jesper

[cparker4486]

[merged]

Hello,

As I understand it B&R still does not handle tiered retention schedules. However, I'm wondering if now that v6.1 has VeeamZIP if Tiered Retention is not more easily accomplished.

There are many topics in the forum on Tiered Retention but I haven't found (not saying it doesn't exist) a rundown of how to actually accomplish it.

Are there documents out there (Veeam or community written) that describe a method to achieve Tiered Retention? I want dailies, weeklies, monthlies, and yearly backups.

[tsightler]

Can you share exactly what you want to do, i.e. are you backing up to a dedupe target? Are you wanting to copy older files to a new location? Configuring retention with Veeam today basically involves some scripting based on the platform. For example, if using a dedupe appliance like DataDomain a simple script that leverages the DD "fastcopy" functionality is what I use. If you want to keep the W/M/Y backups on the same storage volume, a simple PowerShell/bash script (based on if your repo is Windows or Linux) that leverages hard-links is all that is really needed. If you want to copy W/M/Y backups to an alternate location the same basic script that uses file copies instead of hard-links.

[cparker4486]

Currently I am doing Synthetic Backups with no periodic fulls. I have about 21 restore points per job (VMs are grouped into multiple jobs based on OS). Restore points are 24 hours apart. The backup files all reside on the SAN (something I'm in the planning stages of improving).

Are you saying my script should, periodically, on my schedule, copy the VBK files to another location? What about the meta-data?

[tsightler]

That's pretty much it, just copy the VBK files (you can grab the VBM files too, as they might be useful, but they are not required for recovery). The VBK files are the only thing you need, then simply age out the files as time passes. I can share some Linux bash files that I've written that do this for DataDomain using FastCopy, they at least give the concept. I could probably create some for PowerShell. Perhaps it would make a good blog post/whitepaper, at least until Veeam has something built in.

In your case you would have to copy the files because the VBK is changed every night. The FastCopy/hardlink method works only when using Forward Incremental since the files are not touched after they are created.

[cparker4486]

I could probably create some for PowerShell. Perhaps it would make a good blog post/whitepaper, at least until Veeam has something built in.

I think that would be a valuable source to a lot of people and I'd be interested in seeing how you do it. But I do have my own script that does a similar thing with SQL backups that I could probably adapt for this.

In fact... my script is pretty short so here it is. I'd be interested to get your feedback about how I handle the tiered process. In this specific case this script runs every hour of every day and processes unprocessed files.

Code: Select all

##################
## the goal of this script is to manage daily, weekly, monthly, and yearly
## SQL backup files (.bak & .trn). this will be accomplished by keeping a
## running history of daily, weekly, monthly, and yearly backups.
## .trn files are kept alongside the daily backups only.
##################

$working_dir  = 's:\backups\'
$backup_dir   = '\\<server>\sql\'
$todays_date  = get-date
$dailies      = 7 * -1       # 7
$weeklies     = 5 * 7 * -1   # 5
$monthlies    = 12 * 31 * -1 # 12

function compress_encrypt_file($file)
{
	c:\powershell\sevenzip.exe a -mx9 ($file+'.7z') $file '-p<passphrase>'
}

# do stuff with .bak file(s)
foreach($file in ls -r $working_dir -Include '*.bak')
{
	compress_encrypt_file($file.fullname)

	# always copy daily backup to \daily.
	# daily backups will be cleaned up at the end of this script.
	cp ($file.fullname+'.7z') ($backup_dir+'\daily')

	# because multiple rules could be true on the same day the
	# least frequently true storage type goes first because it has
	# a higher priority.
	if($todays_date.dayofyear -eq 1)
	{ # yearly backup
		cp ($file.fullname+'.7z') ($backup_dir+'\yearly')
	}
	elseif($todays_date.day -eq 1)
	{ # monthly backup
		cp ($file.fullname+'.7z') ($backup_dir+'\monthly')
	}
	elseif($todays_date.dayofweek -eq 'Saturday')
	{ # weekly backup
		cp ($file.fullname+'.7z') ($backup_dir+'\weekly')
	}
	
	# clean up files
	del $file.fullname         # original (uncompressed)
}

# do stuff with .trn file(s)
foreach($file in ls -r $working_dir -Include '*.trn')
{
	compress_encrypt_file($file.fullname)
	cp ($file.fullname+'.7z') ($backup_dir+'\daily')

	# clean up files
	del $file.fullname         # original (uncompressed)
}

## cleanup files outside of history thresholds
## we don't run this on yearlies because we want to keep all of them.
ls -r ($backup_dir+'\daily')   -Include '*.trn.7z' | Where {$_.LastWriteTime -le (Get-Date).AddDays($dailies)}   | del
ls -r ($backup_dir+'\daily')   -Include '*.bak.7z' | Where {$_.LastWriteTime -le (Get-Date).AddDays($dailies)}   | del
ls -r ($backup_dir+'\weekly')  -Include '*.bak.7z' | Where {$_.LastWriteTime -le (Get-Date).AddDays($weeklies)}  | del
ls -r ($backup_dir+'\monthly') -Include '*.bak.7z' | Where {$_.LastWriteTime -le (Get-Date).AddDays($monthlies)} | del

[Yuki]

[merged]

Hey all,
So we would like to store 30 days of backups and then 18 month of backups. It would have been ideal if Veeam had a mechanism to achieve this through it's own schedule and cleanup engine, but i'm not seeing how to do it right now. I assume we need to break up the jobs in two - backup that does 30 daily backups and another one that does 18 monthly backups.

1 - The question is - how would the backups affect each other if we want to do reverse incremental on both (only latest backup in each schedule is full, the rest are reverse incremental). The two would not interfere with each other?

2 - How will the de-duplication affect such scenario? Won't one set be not really "full" if de-duplication is involved? Since there are two separate jobs sharing common data, seems that only a single copy of the data would be stored if de-duplication is used, but how and which set of files will contain truly full set and which will be the "de-dupped" one?

[dellock6]

If you are going to run two different jobs on the same VMs, they will act independently:
1 - no
2 - deduplication is made at the job level, two jobs will not have any saved data in common. If you need further deduplication, look at dedup appliances.

And also, if you are going to use reverse incremental, why don't you simply save the VBK in a different location once per month? In this way you are going to hit with IO activities your production storage only once.

Luca.

[Yuki]

Hi Luca, thanks for commenting!

When you say to savve VBK in a different location once per month you mean to use a script or other mechanism to copy the newly created VBK (full backup files) to an alternate location?
Well, we could do it, but then we would have 18 really large VBKs which would eat up a lot of storage capacity (and cost us money as a result).

We would prefer to have 18 monthly backups but still have them as reverse incrementals (latest month is full and the rest are incrementals). That would save us a lot of space, especially since this data is going off-site and remote storage provider may charge us for extra space and/or transfers (Depending on who we go with).

[Vitaliy S.]

When you say to savve VBK in a different location once per month you mean to use a script or other mechanism to copy the newly created VBK (full backup files) to an alternate location?

Yes, this is one of the most commonly used approaches to offload your backup files to the 3rd party locations.

[Yuki]

Privet!

As i've noted above - if we do that, we will have 18 FULL backups... Our single full backup is around 1TB with incrementals being around 10GB. So if we go with this approach, instead of having 1.2TB backup set we will have 18TB backup set. That's a lot of data to move off-site.

[dellock6]

I understand your doubt, but having a job with monthly schedule will surely lead to a huge CBT amount of data: how many blocks will be changed in 1 month timeframe? If they are say 90% of the full VM, it's better to create full backups and avoid reverse incremental that only leads to higher IO without any real benefit.
I wrote an article about this you maye want to look at:

Veeam Backup: check the daily CBT before choosing reverse incremental

Luca.

[J1mbo]

We would prefer to have 18 monthly backups but still have them as reverse incrementals (latest month is full and the rest are incrementals). That would save us a lot of space, especially since this data is going off-site and remote storage provider may charge us for extra space and/or transfers (Depending on who we go with).

Obviously I don't know the specifics of the requirements, but it might be worth considering the purpose of the offsite backups. Traditionally for archive, so on-site is just as good, with the offsite need only really being the most recent, in many cases.

[Yuki]

Guys ( and gals)
We are not really choosing this on our own - we are told what the client wants (in this case 18 month of data off-site for their financial data). We are trying to make Veeam work for them, but they haven't invested into it yet. Acronis is a pile, so it's is out of the question too. We can go with another solution if you have suggestions. These guys got 6 physical machines with about 20 VMs and are ready to spend the money on the right solution.

[J1mbo]

Given the dataset is so small, why not just use 2TB USB drives and send them to either the bank or a different branch, at each month end?

[Yuki]

It is an option we are looking at. Just wish Veeam would listen to the customers (we have a dozen other clients using Veeam based on our spec, but requirements there are slightly different) and implement a more advanced scheduling system.

For us this has been the biggest issue as far as software capabilities go, otherwise the package is pretty stellar.

[hongman]

[merged]

Hi

We are quoting for a customer which requires data to be retained for up to 6 years.

Obviously we dont want to keep all restore points for 6 years as the storage requirement would be too big.

What is the best way to do this? We are thinking something like:

Every day for the past 14-28 days
1 per quarter
1 per year

Does this just mean separate backup jobs with different schedules and retention policies?

We will also be seeding backups to offsite location, so essentially 2 of each as well?

[dellock6]

Well, this is first of all a customer requirement, so you need to check with them the retention frequency. If they only need 1 copy per year of the oldest years, your idea can be good, but I think it's up to them to choose. As consultant, explain them in details pros and cons of the different retentions (consumed space vs number of retention points). After all, if they want more frequent retention points, they are going to buy bigger storare systems for them and so they are going to pay you more

Luca.

[DataAssure]

[merged]

If I want to set a backup schedule like the following, how to accomplish that in Veeam or combination of Task Scheduler w/ Powershell scripts?
Last day of the Year backup Keep 10 years.
Last day of the Month backup Keep 3 years
Saturday Weekly backup Keep 1 months

I also read online stating Veeam will maitain 993 restore points on disk but actual number depends on full backup schedule. Could somenoe explain what does it mean?

TIA

[foggy]

I also read online stating Veeam will maitain 993 restore points on disk but actual number depends on full backup schedule. Could somenoe explain what does it mean?

I believe this is referred to the fact that with forward incremental mode the number of restore points actually kept on disk varies from day to day depending on the backup schedule. While you set the minimum number of restore points to be retained, the actual number varies, as the whole previous chain is required for restore from the particular increment. If you want to keep the exact number of restore points you can use reverse incremental mode.

[zak2011]

Regarding the retension policy..
as Vitaliy has mentioned "each job uses its own backup files, so if you create two jobs, you will have two chains of VBK + VRB/VIB files."
So if i have three jobs backing up the same VMs with different rentension policies..i end up having three chains of VBK + VRB/VIB files.

I would specify one job to run daily..
one to run monthly
one to run yearly..
but is there a way not to have the same VIBs/ VRBs to come in the monthly and yearly jobs i run, since i already have them in the daily.

Thanks!