Discussions related to using object storage as a backup target.
Post Reply
matt-631
Novice
Posts: 6
Liked: never
Joined: Apr 30, 2021 1:27 pm
Full Name: Matt Gosnell
Contact:

Does Veeam clear up expired immutable files?

Post by matt-631 »

Very basic setup to dip my toes into the SOBR world.

- Backup job (Forever Forward incremental) 7 day retention pointing at a SOBR no synthetic Fulls or GFS setup.
- SOBR consisting of local extent and s3 (backblaze) capacity tier
- SOBR configuration is set to COPY all backups to object storage as soon as they are created
- S3 capacity tier has 7 day immutability set on it. (which I now understand is 17 days because of block chain generation)

(Forever incremental job past the initial 7 day retention window) - When the incremental job runs each day, (now that we are past the 7 day retention) the oldest incremental is pushed/merged into the full backup on the local extent. Q: What happens to the incremental blocks on the S3 copy?
I assume the the associated blocks for said incremental are now remarked with a new immutable date because they now exist as part of the FULL and as such are needed for the full and the other incremental's which exist forward in the chain?

So when do blocks get deleted from s3 in this type of setup. Incremental's will be forever smashed into the full each day but how does data fall out of the full backup when its no-longer needed? and more importantly how does the immutability of the files (17 days) play a part in when Veeam will consider deleting something.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Does Veeam clear up expired immutable files?

Post by HannesK »

Hello,
and welcome to the forums.

They are re-marked every 10 days (see FAQ about immutable costs post338749.html#p338749), but yes, your general thinking sounds right.

Retention gets applied similar to on-prem: if objects are not needed anymore, they will be cleaned up. For "not needed", the retention must and immutable time must be over. The concept is the same like for the few vbk / vib files. Just many small of them. The concept is the same.

Best regards,
Hannes
matt-631
Novice
Posts: 6
Liked: never
Joined: Apr 30, 2021 1:27 pm
Full Name: Matt Gosnell
Contact:

Re: Does Veeam clear up expired immutable files?

Post by matt-631 »

Thanks Hannes

A follow up question if I may.

In my example- With the performance extent only having a 7 day retention on day 8 the first incremental is overlaid and absorbed by the full. Within this process the full backup on the performance extent disregards some blocks which are no longer needed. The capacity tier can do none of the above actions as it’s locked by immutability.

So what process and when do the surplus blocks drop out of the capacity tier? I don’t see anything in the job logs to show anything been removed from the capacity tier?.. even after 17 days (7 days + 10 for block generation)

Is there a way to verify that data is being removed from the capacity extent when no longer needed?
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Does Veeam clear up expired immutable files?

Post by HannesK »

Hello,
correct, nothing is shown in the job logs. We are discussing improvements.

When I did tests some time ago, I could see it in the AWS console once retention was applied (usage dropped)

Best regards,
Hannes
matt-631
Novice
Posts: 6
Liked: never
Joined: Apr 30, 2021 1:27 pm
Full Name: Matt Gosnell
Contact:

Re: Does Veeam clear up expired immutable files?

Post by matt-631 »

How about in the Veeam backend logs?

My capacity usage seems to be going up and up (almost like there is no removal happening). How can I validate?
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Does Veeam clear up expired immutable files?

Post by HannesK »

when did you start offloading? The used capacity goes down every 10 days (if all backups were offloaded at the same day). If you uploaded multiple jobs / VMs on different days, then the 10 days period overlaps and it's probably not visible directly in the used capacity.
How can I validate?
Does BackBlaze have invoicing any other monitoring with API usage that shows deletion? I'm not sure how you would want to validate each single object. I mean yes, there are logs in c:\programdata\veeam\backup\SOBR_Offload_<your-repository> - but not sure how this really helps.
matt-631
Novice
Posts: 6
Liked: never
Joined: Apr 30, 2021 1:27 pm
Full Name: Matt Gosnell
Contact:

Re: Does Veeam clear up expired immutable files?

Post by matt-631 »

when did you start offloading?
Day 1. (Monday 19th April) We started uploading 1 Job, Daily Backup with 7 day job retention and 7 day immutability.

Day 8 (Tuesday 27th April) We see in the logs the local extent behaving as you would expect and merging the oldest increment.

Both the local extent and object storage extent continue to only show 7 backups so we assumed that the backups at the object level were being merged as well HOWEVER the capacity tier continued to grow. We then read and realize that the capacity data has the extra 10 days immutability added on. So we wait until day 17..

Day 17 (Thursday 6th May) Expected to see something different today as immutability will have ended for some files. - We do see a very small drop in storage capacity usage. So what disappeared here? I assume the only thing that could (x1 days worth of useless blocks which use to belong in the full which are no longer needed.)?

Day 18 We expected storage to remain fairly stable from here on in. As x1 days worth of incremental come into the capacity tier, we expected to see x1 days worth of data fall out of the full. But NO :-(

Day 19 , 20 ect Data rate continues to grow.
The used capacity goes down every 10 days
Are you saying that on Day 27 we would expect to see the next drop? or day 20?

Today is currently day 22 (11th May)
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Does Veeam clear up expired immutable files?

Post by HannesK »

continue to only show 7 backups
correct. The UI only shows backups that fit to the configured backup retention. 7 in your case.

I have to correct myself about the "every 10 days". It was every "10 days + immutable time" for my test with reverse incremental. Developers told me, that the cleanup algorithm for immutable object storage is complex to be cost optimized with the forever incremental approach (nothing to be documented in public). The cleanup happens when the data is not needed anymore. My recommendation is to just give it more time and eventually you will see that data is cleaned up.
matt-631
Novice
Posts: 6
Liked: never
Joined: Apr 30, 2021 1:27 pm
Full Name: Matt Gosnell
Contact:

Re: Does Veeam clear up expired immutable files?

Post by matt-631 »

Thanks Hannes, I appreciate your help on trying to get the bottom of this. This whole thread has been about running a proof of concept using a small subset of data and using this to predict a cost of putting backups onto immutable storage. With the API calls aside, predicting how many GB/TB will be in the cloud using my real world data helps me to decided if this would work at scale.

At the scale I'm testing a few GB left behind doesn't matter however if I was to scale this project we'd be talking hundreds of TB's. Therefore its important to understand when we would next expect to see a clear up if they are indeed staged and run on set time frames.

Here the table of my experiment so far.
https://imgur.com/a/VXUCYM3
I have to correct myself about the "every 10 days". It was every "10 days + immutable time"
Are you saying that it clears up on "10 days + immutable time" and then will wont do any further clear up for another "10 days + immutable time" So in my case day 36.. (10 + 7 + 10 + 7)?
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Does Veeam clear up expired immutable files?

Post by HannesK »

more like every 17 days (17+17+17+17). It works differently for different backup methods and retentions.

Testing the impact with your configuration makes sense, yes. But you need to wait longer before you can see the full impact of it.
matt-631
Novice
Posts: 6
Liked: never
Joined: Apr 30, 2021 1:27 pm
Full Name: Matt Gosnell
Contact:

Re: Does Veeam clear up expired immutable files?

Post by matt-631 »

Thanks. Ill wait for day 36 and come back to this thread with my findings.

I'm sure there is logic somewhere, but for 7 days worth of backups in object storage with immutability your indicating that we'll be storing 35 days worth of backups at the extreme case before the day 36 clear up (17+17) correct?

If my immutability date was 355 days + the 10 days, your then saying that id only get a cleanup every 365 days (365+365+365) so the problem is exponentially worse because I'm now storing 729 days before the clean up on day 730.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Does Veeam clear up expired immutable files?

Post by HannesK »

you probably want to avoid 355 days immutability :-)

I have seen customers configuring 90 days immutability with only 14 days backup retention... also a bad idea...

keep in mind that some objects / blocks you backed up on day one can still valid after a year if they never changed. So that object will become immutable again and again until it changes and retention applies.
ortec-RW
Enthusiast
Posts: 41
Liked: 7 times
Joined: May 04, 2016 1:39 pm
Full Name: Richard Willkomm
Contact:

Re: Does Veeam clear up expired immutable files?

Post by ortec-RW »

Interesting read.
I was looking at exactly the same. I just could not get me head around it. Still processing it.

I understand how the local extent behaves when the oldest increment is merged with the 1st full in the chain, to adhere to retention and make room for a new increment.
But that involves a change (block add/replace) of the 1st full. You can actually see the filedate on the full change. It basically always has the same date at the newest increment .vib file (the last time of your job run).

That changed full also has to offload to the capacity tier. But this is immutable, so I assume blocks representing the new full on the local extent only get added to the capacity pile. Not replacing/removing any blocks, since that would violate immutability.
So far so good.

So, I was thinking.
If I were to raw import data from only my S3 bucket into a new clean Veeam. Would that actually show and give me additional restore points due to immutability ? In other words, would I be able to use restore points from inside the bucket as a result of immutability, that would otherwise not be shown in Veeam since they already fall out of the retention period of the local extent (marked as not-needed) ?
That would explain why the SOBR only shows restore points inside the retention window. You never see any more, but they are still in the bucket due to immutability. Right ?

I'm about to set up a new test Veeam, and connect it to the S3 to see what it will show. And also to simulate what we need to do when stuff hits the fan.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Does Veeam clear up expired immutable files?

Post by veremin » 2 people like this post

Your understanding is correct and to see those restore points you will need to use PowerShell cmdlet. Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 12 guests