Does Veeam clear up expired immutable files?

[matt-631]

Very basic setup to dip my toes into the SOBR world.

- Backup job (Forever Forward incremental) 7 day retention pointing at a SOBR no synthetic Fulls or GFS setup.
- SOBR consisting of local extent and s3 (backblaze) capacity tier
- SOBR configuration is set to COPY all backups to object storage as soon as they are created
- S3 capacity tier has 7 day immutability set on it. (which I now understand is 17 days because of block chain generation)

(Forever incremental job past the initial 7 day retention window) - When the incremental job runs each day, (now that we are past the 7 day retention) the oldest incremental is pushed/merged into the full backup on the local extent. Q: What happens to the incremental blocks on the S3 copy?
I assume the the associated blocks for said incremental are now remarked with a new immutable date because they now exist as part of the FULL and as such are needed for the full and the other incremental's which exist forward in the chain?

So when do blocks get deleted from s3 in this type of setup. Incremental's will be forever smashed into the full each day but how does data fall out of the full backup when its no-longer needed? and more importantly how does the immutability of the files (17 days) play a part in when Veeam will consider deleting something.

[HannesK]

Hello,
and welcome to the forums.

They are re-marked every 10 days (see FAQ about immutable costs post338749.html#p338749), but yes, your general thinking sounds right.

Retention gets applied similar to on-prem: if objects are not needed anymore, they will be cleaned up. For "not needed", the retention must and immutable time must be over. The concept is the same like for the few vbk / vib files. Just many small of them. The concept is the same.

Best regards,
Hannes

[matt-631]

Thanks Hannes

A follow up question if I may.

In my example- With the performance extent only having a 7 day retention on day 8 the first incremental is overlaid and absorbed by the full. Within this process the full backup on the performance extent disregards some blocks which are no longer needed. The capacity tier can do none of the above actions as it’s locked by immutability.

So what process and when do the surplus blocks drop out of the capacity tier? I don’t see anything in the job logs to show anything been removed from the capacity tier?.. even after 17 days (7 days + 10 for block generation)

Is there a way to verify that data is being removed from the capacity extent when no longer needed?

[HannesK]

Hello,
correct, nothing is shown in the job logs. We are discussing improvements.

When I did tests some time ago, I could see it in the AWS console once retention was applied (usage dropped)

Best regards,
Hannes

[matt-631]

How about in the Veeam backend logs?

My capacity usage seems to be going up and up (almost like there is no removal happening). How can I validate?

[HannesK]

when did you start offloading? The used capacity goes down every 10 days (if all backups were offloaded at the same day). If you uploaded multiple jobs / VMs on different days, then the 10 days period overlaps and it's probably not visible directly in the used capacity.

How can I validate?

Does BackBlaze have invoicing any other monitoring with API usage that shows deletion? I'm not sure how you would want to validate each single object. I mean yes, there are logs in c:\programdata\veeam\backup\SOBR_Offload_<your-repository> - but not sure how this really helps.

[matt-631]

when did you start offloading?

Day 1. (Monday 19th April) We started uploading 1 Job, Daily Backup with 7 day job retention and 7 day immutability.

Day 8 (Tuesday 27th April) We see in the logs the local extent behaving as you would expect and merging the oldest increment.

Both the local extent and object storage extent continue to only show 7 backups so we assumed that the backups at the object level were being merged as well HOWEVER the capacity tier continued to grow. We then read and realize that the capacity data has the extra 10 days immutability added on. So we wait until day 17..

Day 17 (Thursday 6th May) Expected to see something different today as immutability will have ended for some files. - We do see a very small drop in storage capacity usage. So what disappeared here? I assume the only thing that could (x1 days worth of useless blocks which use to belong in the full which are no longer needed.)?

Day 18 We expected storage to remain fairly stable from here on in. As x1 days worth of incremental come into the capacity tier, we expected to see x1 days worth of data fall out of the full. But NO

Day 19 , 20 ect Data rate continues to grow.

The used capacity goes down every 10 days

Are you saying that on Day 27 we would expect to see the next drop? or day 20?

Today is currently day 22 (11th May)

[HannesK]

continue to only show 7 backups

correct. The UI only shows backups that fit to the configured backup retention. 7 in your case.

I have to correct myself about the "every 10 days". It was every "10 days + immutable time" for my test with reverse incremental. Developers told me, that the cleanup algorithm for immutable object storage is complex to be cost optimized with the forever incremental approach (nothing to be documented in public). The cleanup happens when the data is not needed anymore. My recommendation is to just give it more time and eventually you will see that data is cleaned up.

[matt-631]

Thanks Hannes, I appreciate your help on trying to get the bottom of this. This whole thread has been about running a proof of concept using a small subset of data and using this to predict a cost of putting backups onto immutable storage. With the API calls aside, predicting how many GB/TB will be in the cloud using my real world data helps me to decided if this would work at scale.

At the scale I'm testing a few GB left behind doesn't matter however if I was to scale this project we'd be talking hundreds of TB's. Therefore its important to understand when we would next expect to see a clear up if they are indeed staged and run on set time frames.

Here the table of my experiment so far.
https://imgur.com/a/VXUCYM3

I have to correct myself about the "every 10 days". It was every "10 days + immutable time"

Are you saying that it clears up on "10 days + immutable time" and then will wont do any further clear up for another "10 days + immutable time" So in my case day 36.. (10 + 7 + 10 + 7)?

[HannesK]

more like every 17 days (17+17+17+17). It works differently for different backup methods and retentions.

Testing the impact with your configuration makes sense, yes. But you need to wait longer before you can see the full impact of it.

[matt-631]

Thanks. Ill wait for day 36 and come back to this thread with my findings.

I'm sure there is logic somewhere, but for 7 days worth of backups in object storage with immutability your indicating that we'll be storing 35 days worth of backups at the extreme case before the day 36 clear up (17+17) correct?

If my immutability date was 355 days + the 10 days, your then saying that id only get a cleanup every 365 days (365+365+365) so the problem is exponentially worse because I'm now storing 729 days before the clean up on day 730.

[HannesK]

you probably want to avoid 355 days immutability

I have seen customers configuring 90 days immutability with only 14 days backup retention... also a bad idea...

keep in mind that some objects / blocks you backed up on day one can still valid after a year if they never changed. So that object will become immutable again and again until it changes and retention applies.

[ortec-RW]

Interesting read.
I was looking at exactly the same. I just could not get me head around it. Still processing it.

I understand how the local extent behaves when the oldest increment is merged with the 1st full in the chain, to adhere to retention and make room for a new increment.
But that involves a change (block add/replace) of the 1st full. You can actually see the filedate on the full change. It basically always has the same date at the newest increment .vib file (the last time of your job run).

That changed full also has to offload to the capacity tier. But this is immutable, so I assume blocks representing the new full on the local extent only get added to the capacity pile. Not replacing/removing any blocks, since that would violate immutability.
So far so good.

So, I was thinking.
If I were to raw import data from only my S3 bucket into a new clean Veeam. Would that actually show and give me additional restore points due to immutability ? In other words, would I be able to use restore points from inside the bucket as a result of immutability, that would otherwise not be shown in Veeam since they already fall out of the retention period of the local extent (marked as not-needed) ?
That would explain why the SOBR only shows restore points inside the retention window. You never see any more, but they are still in the bucket due to immutability. Right ?

I'm about to set up a new test Veeam, and connect it to the S3 to see what it will show. And also to simulate what we need to do when stuff hits the fan.

[veremin]

Your understanding is correct and to see those restore points you will need to use PowerShell cmdlet. Thanks!