Removing old VMs from backup sets

[cbeilman]

I've found that removing stale VMs from backup sets is a slow and tedious process. I've created this Powershell Script that will find VM backups older than 30 days and delete them. I've commented out the $_.Delete() line so you can see what it will delete without deleting it.

Code: Select all

$keepForDays = 30

$oldestBackupDate = (Get-Date) - (New-TimeSpan -Days $keepForDays)
$backups = Get-VBRBackup 
foreach ($backup in $backups) {
	$oldBackups = $backup.GetOibs() | Where {$_.CreationTime -le $oldestBackupDate}
	if ($oldBackups -ne $null) {
		$oldBackups | % {
			Write-Host ([String]::Format("Removing {0} from {1} ({2})", $_.VmName, $backup.JobName, $_.CreationTime)) 
			#$_.Delete()
		}
	}
}

You can also modify the third line to something like this to prune backups ending with Daily:

Code: Select all

$backups = Get-VBRBackup | Where {$_.JobName -like "*Daily"}

[Gostev]

Thanks for sharing this, Clint.

Just wanted to note here that starting from version 5.0.2, "stale" (deleted/removed/unreachable) VMs will be removed from backup automatically according to set retention policy in number of days configurable in the job properties. Thus, those backup files will no longer be "locked" and job's main retention policy will process this normally.

You may want to keep this setting set for at least a few days, because sometimes VM can appear "deleted" just because vCenter server does not return it as present due to API call timeout or some other issue for example. And you do not want VM to be dropped from your backup file immediately just because of that single glitch.

[cbeilman]

That's great. Will the retention setting be changing from session to days? Also, do you know if there are any plans for having retention rates which would allow me to do something like save one backup per month, one per week and say 7 daily jobs? I was thinking of trying to make a Powershell script to do something like that.

[Bunce]

I've requested realistic tiered retention policies ( to match that of competing products such as esxpress / vdr etc) in the past, but the focus for veeam does not seem to be in this area.The current options available for retention in veeam are pretty much pointless for us as it forces us to periodically run fulls (offload to tape), which we don't need or want to do.

We'd much rather replicate our current backup storage offsite using modern technologies without having to continually run fulls and offload every few months because storage gets wasted.

We're still utilising a competing product at one of our remote sites, and haven't had to touch it for 24 months. It maintains a daily backup for the past 2 weeks, a weekly backup for the past 6 weeks, 2 monthly backups for each of the past 14 months, and 2 yearly backups each year.

Removal of old backups is automatic, and overall storage space remains almost constant.

We Rsync the storage to external HD each week (rotating between 3 drives) for disconnected/offsite requirements, which also helps to alleviate any 'corruption' concerns although the backup target runs 'self-healing' checks periodically as it is.

http://www.veeam.com/forums/viewtopic.php?f=2&t=6390
http://www.veeam.com/forums/viewtopic.p ... ion#p21473
http://www.veeam.com/forums/viewtopic.p ... 378#p13772
http://www.veeam.com/forums/viewtopic.p ... ion#p15687

[Gostev]

That's great. Will the retention setting be changing from session to days?

For staled VMs, retention setting is in days.

Also, do you know if there are any plans for having retention rates which would allow me to do something like save one backup per month, one per week and say 7 daily jobs? I was thinking of trying to make a Powershell script to do something like that.

Yes, the plans are there, but this is not a short-term feature.

[cparker4486]

[merged]

Hello,

(First of all I hope I don't reveal too much of a lack of understanding of B&R's capabilities and/or backup strategy in general with this suggestion. If this idea is possible now or with a simple workaround I'd be glad to find out about it.)

I'd like to propose something I'm going to call "decaying synthetic backups." My idea is that B&R should reflect the relative importance of a VM's backup over time by merging old backups as they get older. In general as a backup gets older it becomes less "important" B&R currently treats all synthetic backups as equally important by simply deleting the oldest backups to make room for the new ones. This is an inefficient use of disk space as it prevents a long history of backups to be kept.

The idea is simple when said plainly: The period between available backups becomes greater as you move further back in time.

Example:
1 VM is to be backed up daily with 45 backup points. This will give is 45 days of backups. For the sake of this example let's also say that the .VBK is 100GB and each .VRB is 10GB. This would require 550GB of storage. What if it were possible to stretch that 550GB of storage into 200 days of history or more?

How likely is it that you would need to go exactly 26 or 39 days into the past to do a restore? In all my 10 years of experience it's been very unlikely. Generally I need to restore something from 1 to 15 days in the past and that's about it. So why keep all those 10GB .VRB files?

Why not instead progressively merge the older synthetic files so that you end up with fewer (albeit some amount larger) individual .VRB files that covered a greater period of time?

* 1 backup per day for 14 days (14 .VRB files)
* 1 backup per 2 days starting on day 15 for 12 days (6 .VRB files; the oldest day's data is available)
* 1 backup per 4 days starting on day 28 for 48 days (12 .VRB files; the oldest day's data is available)
* etc.

Over time you'd end up with a much longer history and slightly fewer/larger .VRB files.

This is a very similar idea to tape backup strategy. Daily, weekly, monthly, and yearly tapes. Instead in this case it would be a rolling history.

I made an Excel file to help illustrate the idea but it doesn't appear the board allows uploading attachments.

[Bunce]

Search for tiered retention. Has been requested quite often..

[cparker4486]

I see it's mentioned several times but are we close to getting it or what? Do you have any idea?

[Bunce]

I'll leave that to the guru's but don't believe it is possible using the current Veeam architecture.

The products we've used that offer this functionality use a storage 'appliance' as a target - Veeam uses a file based 'chain' so if an older (unwanted) backup is removed, all backups since would become invalid since it breaks the chain (using Incremental mode anyway).

May be able to setup something using reverse incremental though. I've also heard mention of a 'forever incremental' feature in the next release but unaware of the specifics..

We've grappelled with this for some time, and have relented to using Monthly backup jobs (that we retain) in combination with Daily's (with a shorter retention period). Still a waste of space, but the only option we've come up with to achieve similar functionality.

<enter Anton with spiel on how evil storage appliance's are>

[cparker4486]

Seems like it should be possible though I can imagine it could be very processor and disk intensive. Here's to hoping someone at Veeam is working on it.