Disk Space Management

[cardendave]

Hi All

I am using Veeam across multiple clients. I wondered how everyone else was managing disk space on the backup destination? We have backups that run out of space, we then have to delete the oldest restore points and run again manually. What we want to do is for Veeam to delete older backups as it runs out of space, is this possible? We are trialling reverse incremental, is this a better way to not run out of space?

Thanks

Dave

[Gostev]

Hi, Dave. Veeam has no such functionality, as deleting backups that are still under retention policy will result us in compromising data retention policy required by the customer as per backup job settings. The recommended way to manage and monitor backup repository disk space is through Veeam ONE capacity planning capabilities. Thanks!

[foggy]

There're also built-in space notifications that allow to track free space on the repository. And yes, reverse incremental is more space-efficient, however puts higher I/O load on repository.

[kkuszek]

Hello,
Veeam has the ability to warn me when backup storage is at or below a certain percentage, but I think we should be able to set a critical space termination point. I.E. when the drive reaches x% or xgb stop using it as a backup repository.

I understand space management should be better but since you manipulate the number of recovery points there is a black art to management if you find yourself running a few out of band full backups. Since the drive could be used as a target for other types of backups I think we should be able to set veeam to stop when it has 200gb free instead of 200kb free as another safety.

[Vitaliy S.]

Hi Kurt,

Thanks for your feature request. As a workaround for now, you may want to use Veeam ONE monitoring capabilities. When there is an alert about reaching the certain thresholds as a post alarm action you can configure a script to terminate the backup job pointed to this repository.

Thanks!

[ATIhelpdesk]

Actually, I would like an option to set the minimum freespace (as bytes or percentage) and, have it make that the retention setting. That is, keep retaining backups until the minimum available freespace is 5% (or the store is 95% full).

[veremin]

How would you deal with backup jobs that aren't using forever modes, like the ones with periodic full backups. Providing how retention policy works for such jobs, it won't be possible to purge the oldest restore points. Thanks.

[kkuszek]

Hi Vitaly,
Is this a common script I can find somewhere and easily do? I suppose that could potentially work although I see complexity & failure modes there.

Let me expand this feature concept as well to include the option to allow delete oldest incremental or chain to make space for a new backup (set by repository/job). This way backups can continue without manually having to clean up files and make a mess of the sql side of veeam. I would rather tonight's backup actually happen with a warning that it had to remove my oldest backup to make space. I archive my restore points to tape so I need to HAVE backups more than I need to have more retention points, it's just nice to have when there is space.

[kkuszek]

Vitaly I think an "allow delete oldest|second oldest|never" retention point option could satisfy ATIhelpdesk's request as well. At that point if it deleted jobs based on space and not just number of retention points he could effectively just set his number of points to keep to infinity.

If someone wants to slice out x amount of space for y & z backup jobs they could just make subfolders and make them different repositories.

This is something I struggle with myself. I have 10tb of dedupe space to work with, and I just want to ensure I can fit as many restore points as possible on there. I don't think I should have to tweak that number based on free space and monitor closely. I feel like i am flying close to the sun here with 1-300 gb incrementals and 3.5tb fulls. I just want Veeam to deal with it and I have however many restore points I have for the space and data turnover in that time period.

Yes it changes the fundamental nature of how backups are managed but it allows people to more effectively utilize their resources. I want veeam to get xyz space and use it all for as many points as I can get away with, so why not?

[Gostev]

Valid issue, but wrong solution.

I just want Veeam to deal with it and I have however many restore points I have for the space and data turnover in that time period... it allows people to more effectively utilize their resources.

I would argue that flooding your storage with backups until it fills up can be classified as "effective utilization of their resources" because you are essentially spending money to store the data your business no longer needs.

Yes it changes the fundamental nature of how backups are managed

That is absolutely correct. RTO, RPO and Retention Policy is determined by the business needs. You don't keep backups longer than you really need (because you are paying for this one way or the other), and you don't delete backups still under retention to free up space for new.

I do agree this feature can be great for a consumer product and specifically to simplify (remove) retention configuration, in fact this is exactly how Shadow Copies work in Windows (oldest are deleted automatically). But, at Veeam we are not in a consumer backup business with B&R. We should probably consider this feature for local targets in our Veeam Endpoint Backup FREE though.

For an enterprise backup product though, even putting costs factor aside, such approach adds reliability issues, because both the storage itself, as well as multiple protected workloads are very dynamic and much less predictable. It's hard to guess how many older backups need to be deleted to fit the new one - especially with multiple jobs and other processes using the same storage. As a result, following this paradigm will lead to many additional backup job failures, and those jobs potentially leaking out into production hours with retries.

That said, we do recognize the individual backup storage device space management issue, and we are solving this in v9 with the Scale-out Backup Repository. With that, you will no longer have to worry that a particular backup storage device is running out of disk space, as the job is no longer tied to a particular storage device.

[kkuszek]

Hi Gostev,
You do raise some valid points, although I would counter that business requirements are typically "at least" and not "at most". They might budget to say we should fit at least xyz number of recovery days on xyz budget (not always recovery points). You buy in budget to meet those points. Nobody ever says they don't want more for their money. Maybe another viewpoint to approach with is allowing more (if space allows) than the recovery point setting which is deemed a minimum. As long as your maximum retention doesn't exceed your global retention policy for compliance/legal having "too many" backups is a sysadmin dream.

Backups are semi predictable but not always. A few big days and you could be well over. I shouldn't need to double my space just in case or set my recovery points artificially low, I should utilize it more effectively. I'm also on standard in the SMB space using the resources I have at hand and the target market for standard edition. Scale out is great and could help buffer overflow but probably better suits enterprise with a mass of resources. I want as much as I can get from what I bought with the budget they gave me.

You do have an effective point in that B&R would have issues predicting just how much to clear up for the next job. It could easily result in successive failures. If the last job failed due to lack of space delete another backup and retry. you could have 2 failures before you get a success but it could still self recover. You could also monitor free space as the job continues and when reaching xyz pause the current backup, delete another old backup , then continue the current backup.
Enterprise very well may not use this as an option, but that's the beauty of a checkbox defaulted unchecked.

[Gostev]

One thing great about scale-out repository is specifically the fact that it allows you to use the resources you have at hand. Any chunk of free disk storage you have available, on any server or NAS - you can add it into your scale-out repository, no matter of the type of storage resource. Everything is 99% used? Just add a few hard drives into the existing physical server - and there you go, another extent of a few TB in size for just a few hundreds dollars/euro in costs.

My biggest concern about your idea is reliability outside of the smallest environments with dedicated backup storage - but there is no way for me to restrict people using this feature in the environments of any size, and very few will understand the implications. I am guilty of adding a few features like that to the product before, and now have to live with them causing tons of support cases, yet unable to discontinue them. So, I am trying not to repeat such errors any more, and instead prefer universally reliable solutions (such as scale-out repository).

[kkuszek]

well even if we call this a 2 part, minimum free space definition in a repository (or even a repository space definition such as defining one as 10tb vs just until it runs out) should hold no other implications. Combined with allowing backup to exceed restore points if space allows and you can get it from the other side where you have a guaranteed minimum recovery point retention. You could even require that option to have a repository space definition to prevent target infinite run over.

[cardendave]

Dear Veeam

I have a regular (weekly) case where I run out of space on my backup media. NAS, USB, Cloud, any of them. Veeam is obviously an advanced piece of software. As we know I have to select the amount of restore points I want to keep.

Windows Server Backup is basic and free, however when I setup my backup media in Windows Backup it manages its space for me! I never have it run out of space, as long as my drive is bigger than the partition.

I would like to put forward a feature request. I install my backup repository, it looks how much space there is and then starts to backup on it. As it runs out of space it automatically removes older restore points. Then in the results email it can tell me how many restore points are currently in the repository. With this feature IT admins will get the most out of their repo space, without having to go in and manually adjust it all of the time. The existing option can remain where you can choose the restore points if you want, or tick "automatically manage this for me"

The problem with the system at the moment is peoples data grows. So when I buy a new server and set my restore points, fine. But as my data grows the amount of restore points I can fit on my media declines. One day I run out of space and I have to adjust it, but I can't restore the individual restore points, so I have to delete the entire backup.

The email alerts regarding backups can include the amount of restore points available on the backup media for IT admins to monitor it.

Dave

[Vitaliy S.]

Hi Dave, thank you for your FR, please review Anton's reply above for more info on why this might not be the best approach to address a lack of free space on the repository issue.

[cardendave]

Guys

The theory is simple. I have a 2TB drive, my server is 300GB in data usage. I have the option to choose my retention OR set to "Allow Veeam to Manage this for me". If I choose to have Veeam to manage this for me, as the space starts to get to less than the size of my data, lets say it is 400GB, older backups in the chain start to be removed.

Anyone doubting this idea let me tell you this. I am an MSP and have purchased Veeam for 15 of my clients. I currently have 3 support tickets open and I would say I have weekly issues where a drive has run out of space. I then have to call the client and ask them to plug the previous drive in so I can then delete backups. Very time consuming. I then cannot reduce the amount of restore points and run the backup again as it deletes restore points after the backup, so I have to manually delete files OR the whole backup from within the B&R console.

Enterprise product or not, this is basic functionality that is even in Windows Server Backup and as someone mentioned Microsoft Shadow Copies.

Please consider this seriously as it would be life changing for our support desk and business.

Dave

*EDIT - I actually have to run Windows Server backup on all of my servers alongside Veeam onto different media, in case I have to delete my Veeam backups - not ideal!

*EDIT2 - All my Windows Server Backups work! Lots of my Veeam ones fail. ALWAYS drive space issues.

[kkuszek]

I still disagree with Veeam's stance quite a bit. I think data turnover is dynamic and space usage in turn is also dynamic while disk space is fixed. The only way to meet your rpo is to overbuy on storage and leave plenty of buffer. Someone moves a bunch of data and you have a big backup night or you have a request for off scheduled backups and you play the lottery.

This is wasted space and a gamble in your backup strategy requiring more frequent attention. Scale out repo helps quite a bit but it just fragments your backup if you have poor strategy or a highly dynamic month. It also isn't much better than having a single big datastore. If you have reliable first level backups and scale out to include a cheap expansion or other available storage then Veeam will put your newest backup on the scale out instead of relocating the oldest. This could actually be seen as a negative.

This is not different because veeam has an enterprise target market. The difference with an enterprise and smb is they can afford to throw more budget at their storage pool and often have more mature retention requirement policies. Enterprise sys admins are never going to say they are upset about "too many backups".

A smaller business likely backs up what they can afford to backup more often than what they could use.

Unless someone tells me otherwise I want every backup on disk I can fit, period. I probably don't need the older ones, and I can't justify buying more storage to expand... but that's not the point.

In the end I think allowing for minimum recovery point objectives while utilization of up to x space as defined for a datastore is the smartest, most adaptive way for veeam to accommodate all scales. The user could configure minimum recovery points to either be a hard failure or a soft alarm/warning.

This would reduce service calls to your msp's, allow dynamic and changing months to happen, and be extremely easy to implement.

You don't have to know the backup size to plan. You backup until you hit the space limit, then you delete the oldest backup, and then you continue. If you run out of space again you just... do it again.

This could extend backup windows due to the delete time pause, and individuals who want no more than x recovery points may take issue... but that problem is easily solved by having a checkbox next to recovery points for "strict".

It drives me batty that I had to greatly underprovision my recovery points so that veeam didn't make my life hell. I don't want to micromanage my backups, so I underutilize my target to avoid the nuisance veeam mandated.

Business Backup strategies are defined as what is the minimum you can get away with or afford. Not around how many backups are too much. So why do we inherently design around wasted resources?

[cardendave]

I still disagree with Veeam's stance quite a bit. I think data turnover is dynamic and space usage in turn is also dynamic while disk space is fixed. The only way to meet your rpo is to overbuy on storage and leave plenty of buffer. Someone moves a bunch of data and you have a big backup night or you have a request for off scheduled backups and you play the lottery.

Yes people are listening to me!

Windows Backup does this and is as basic as it gets. Give us guys the option.

It drives me batty that I had to greatly underprovision my recovery points so that veeam didn't make my life hell. I don't want to micromanage my backups, so I underutilize my target to avoid the nuisance veeam mandated.

My Veeam life is hell and I am micromanaging daily. Some times I have to delete entire backup chains, risking clients data.

Please add this feature!

Amount of restore points on the media in the email is also a great idea, gives us the chance to see it quickly and easily.

[Gostev]

All, just to set the right expectations and stop wasting everyone's time in this argument, I wanted to repeat that this feature is not planned for inclusion for the reasons I've mentioned earlier, and particularly due to reliability concerns of this approach.

I understand you disagree with this decision, but I still wanted to be open about our plans, as I understand that some of you may make a choice to change your backup vendor based on these plans.

One of the main reasons why people prefer us to (free) Windows Backup specifically is reliability, and to achieve this we need to make certain architecture choices that ensure predictable behavior. So I am confident that this is the correct decision in the bigger picture - even if it forces some of you to change our backup product to a competing one (as much as I hate to see this).

Thanks a lot for your feedback though, and especially for your time spent in posting one! It will certainly be considered regardless, for example if we end up redesigning our backup storage format in the future.

[kkuszek]

Thanks Gostev.
I don't know that it's something I would call a deal breaker for me personally, but as the customer I wanted to advocate for my needs in the product and explain my pain points. I understand all of these requests don't always equal implementation and we won't just win you over if you don't want to do it.
I don't understand your viewpoint as to how this could impact the product reliability implemented as I described at all.
It's what some of us would like to see, but I guess we have to agree to disagree. Thanks for taking the time to address, and maybe things will change if enough people come forward with similar feelings. I guess we will see.

[cardendave]

I have arrived at work this morning and one of my clients NAS drives is full.

I cannot delete backups. I have to delete the entire backup.

This is while I have a support ticket open because the other backup this client has isn't working.

I have no choice but to leave my client with NO backups.

This could have been avoided, looking at the backup job I have 17 restore points. Veeam could have deleted some and then told me how many I had in the results email.

I am not saying remove the option to select X amount of restore points, but give us the option to override it and let Veeam manage it for us.

Like someone else said, I cannot micromanage this. Someone at the office copied a lot of data to the server 3 days ago. Veeam One will not predict that.

Luckily I have Windows Backup running (a non enterprise product)

Not great Mr Veeam.

[Gostev]

I cannot delete backups. I have to delete the entire backup.

This is actually something that we do have planned (ability to delete certain backups from within UI). We even have similar functionality already available in v9, but the corresponding menu options only appear on inconsistent or corrupted backup chains. I have the CR logged to make it available also on "good" backup chains so that you could cleanly delete certain restore points - those with no dependencies.

In any case, please keep in mind that the easiest and recommended way to resolve the issue is to turn your existing repository into a scale-out backup repository, and add additional piece of storage to the pool (which can be anything). This will allow backups to continue normally.

Also, be sure you don't disable backup repository disk space warning (and don't ignore them once they start appearing - at 10% free disk space by default).

[cardendave]

Arrived at the office today.

Another client, another Veeam, another beeping NAS out of space.

[Gostev]

Well, may be this is an indication that you do need to start doing proper disk space planning for your clients in the end sorry to say this, but with so many issues your clients are experiencing, it seems that none of your Veeam implementations actually included much planning... and while it is easy to put blame on the product, let's be fair here and admit there's another side of this issue as well.

There's even a tool that makes such planning super easy - you simply ask your client for their required data retention period (remember that in many cases, there is a company policy or even a legal requirement on data retention they MUST meet) and then fill in this form http://rps.dewin.me/

But of course, on-going monitoring is also the key here as your clients continue to grow - definitely not something to be overlooked. You do need to switch from using fire extinguishers to using smoke alarms in your approach to managing your clients, so to speak.

Also, do keep in mind that scale-out backup repository makes it super easy to expand the repository without making any changes to your job setup. And I recommend you leverage it preemptively, because there's additional performance benefits of using one (significant backup window reduction).

[ATIhelpdesk]

Well, may be this is an indication that you do need to start doing proper disk space planning for your clients in the end ...

Spoken by a person who doesn't live in the "real world" of IT consulting. Clients don't deem in necessary to tell us when they download a bootleg copy of last season's Game or Thrones. We do plan storage space based on the client needs at-the-time! Then they change something and we find out with an email containing lots of the colour red.

The most annoying part of Gostev's argument is that the bulk of what's needed is already in Veeam. Right now it deletes old restore points (RPs) based on a count of retained RPs. It is almost trivial to use available space instead. Yes, I understand backups may fail and have to retry after older RPs are removed. This could be handled by allowing the current backup to temporarily "invade" a reserved overflow space. For example: let's say I tell Veeam to retain backups until the freespace is 15%. It is understood Veeam can use that freespace while a job is currently active. This is almost what I do manually. I have a NAS device for my backups on which I have reserved about 200 GB. When backups run out of space, I add space to that backup's store from the reserved space, finish the backup job, reduce the retention by a day, wait for Veeam to delete enough RPs, then reduce the size of the backup store. Because, I don't have enough to do so I can mess with this.

Many enterprises I deal with do have a maximum retention days policy ( usually set by their legal department ). There should be a way to set maximum retention by days. Such as 1095 days ( 3 years ).

Finally, the idea that organizations never need to restore old data flies in to face of my experience. While it does not happen often, I have been asked to recover a file or folder a user hasn't looked at since the previous quarter.

[kkuszek]

@atihelpdesk

Gostev is really pretty good with things here, he's not the bad guy.
I personally set a 21 point retention on my jobs but could fit 30-33 days on my storage unit if it wouldn't fail out. It's what we have to live with in the current environment. Throw deduplicating storage into the mix (I use windows 2k12 dedupe) and you're begging for dynamic storage needs. He's not wrong that you need to plan around all the typical variations a specific business unit might see. It sucks to waste aaaaaalllllllll that expensive space for a maybe buffer but that's Veeam as it is. You could try netwrix file server monitor or write some scripts to watch servers at risk so you get automatic notifications when there is a large amount of changed files/free space. Then you get a warning in case a backup *could* fail.


I actually also don't like the way you proposed. That leaves free space an unknown, and letting it run over could interfere with other things you want out of that storage. I.E. I have some scheduled backups outside of veeam that share the same repository space that would fail out if it got too low and fragmentation gets a lot harder to manage.
A min/max retention point policy per job would solve for retention minimum and legal's maximum requirements, and a minimum free space (or maximum repository size) per target should be a hard limit.

I much prefer the method I proposed and while I do not understand where the technical hurdles for it lie, it's the programmers prerogative. A job can very simply run out of specified space, pause, and delete older than minimum retention points on the spot to make enough room to complete instead of failing out. It would extend job times, but if that's a problem you plan around it or don't set a higher maximum retention point than the minimum for that job - or a job prep task can be scheduled that looks at when a specific job runs next, calculates mean job size, and deletes retention points to meet the freespace.

Also,
When I was saying I probably don't need the older ones I meant the backups that there was room for beyond the minimum retention point policy. We all need to restore old backups at times and those are archived by however means.

The only complexity/difficulty and possible argument I can see in the way would be balancing multiple jobs and overflow retention priority i.e. will a job be allowed to delete only it's own restore points or from any older restore points? how would you determine job A can have 35 points on a 21 point retention minimum and job B can fit 30 on a 21 point retention?
A min/max retention point per job setting might be easier, and a checkbox for "Allow other jobs to delete restore points above minimum to satisfy free space" could say whether or not it gets priority.

Again we're a couple of dreamers I think because it seems they are not interested in implementing. I do give Veeam credit for typically listening and taking feature requests to heart compared to other software vendors I've worked with. For whatever reason this one doesn't fit their vision.

[fredywenger]

Hi all

We are in evaluation for veeam for windows (to possible replace Acronis).
I can confirm, that this feature is strongly missing (Acronis has it, b.t.w.).
We have machines, that have to be backup to a local USB drive.

It should be possible to set up the backup as "set and forget":
- Configure the (local connected) USB drive as backup media
- Checkbox "Automatically delete oldest backup, if disk full"
That's it...
=> It should not be a problem, to calculate the needed space before backup is done (at least as long only ONE machine does a backup to a local USB disk.
Maybe, it would make sense to add a additional checkbox to the notification email "Notify, if backup was deleted of cause disk was full".

So.. I can follow the arguments in this thread, but nevertheless, such a function should be enhanced!
And.. the customers are able to decide, if the activation of this function does make sense for their use case oder not... trust me..
Fredy

[Vitaliy S.]

Hi Fredy,

Deleting backups automatically without honoring the retention policy setting might be dangerous, but thanks for your FR!

[Gostev]

@Fredy, we're solving this whole issue in a bit different way in U4, by allowing to offload older backups to an object storage. I still don't like the "destructive" way of fixing the issue (by deleting backups completely instead), as I can see some scenario where this approach may eventually result in customers left with NO backups at all. Thanks!