Object Storage and Immutability principles

[Samuraj]

Hello,
I'm trying to understand the details and principles of how storing backups to object storage repository (I describe my understanding on the blog, the last one https://www.samuraj-cz.com/en/article/v ... epository/). I have read the official documentation, I found more details here on the forum (Sorry if I missed any answers). However, I am not sure if I understood some things correctly and I have questions that I have not found the answer to. I have a lot of questions (sorry) and I appreciate every answer.

1) In several discussions I found information that there is no merged on the object storage. And a description of the principle which I wrote below, but I don't know if I understood correctly?

Object storage doesn't work with files, but with objects. Veeam divides backup files (Restore Points) into blocks of a certain size (1 MB by default), compresses them and transfers them as objects to object storage. Each restore point references the required objects. When an incremental backup occurs, objects with new (changed) data are transferred. The new restore point again references all necessary objects to enable full recovery. For this reason, we don't need to perform a synthetic full backup and space is saved similarly to Fast Clone.

But in my Backup Copy Job log there is an action "Full backup file merge completed successfully" and It takes some time. So does Veeam do merges or not?

2) Maybe the above is not accurate and the term checkpoint is important. I haven't been able to find any explanation of it. Apparently the checkpoint contains metadata for backups and allows me to go back in time. And this is somehow related to creating a Full Backup and extending immutability for active blocks. And thanks to Block Generation, the extension is only done once every 10 days?

3) My assumption - Block Generation 10 days, daily backup. Retention Policy should be >= Repository Immutability period + Block Generation. Always 10 backups will have the same Immutability date. The next 10 will be 10 days longer. Normally we have forever forward incremental. After the specified Retention days have passed, Retention Policy deletes the oldest Restore point every day.

I don't understand "Actual retention = job retention policy + immutability period + Block Generation period" https://helpcenter.veeam.com/docs/backu ... ml?ver=120. If the objects are not immutable, the retention policy deletes them and I cannot restore them. Or not? In various tests, I manually deleted data after immutability ended. Retention Policy deletes Restore Points at least from the configuration. But on object storage, it waits for an immutability period + Block Generation longer?

4) For testing I set a shorter Retention Policy than Immutability Period. This situation has been discussed several times on the forum. I understand that Retention Policy will remove old Restore Points from the metadata (and also the configuration DB?), but the objects in the repository will remain.

Is it correct, that objects will be deleted automatically when Immutability ends? https://helpcenter.veeam.com/docs/backu ... n_job.html "The background retention waits until the immutability and retention periods end for these files."

I haven't figured out when is used job retention and when background retention. I found the log for background retention (History – System – Background retention). But I don't see information about deleting restore points in the job log (not even for Windows Repository). Is there any log? And in the case of an immutable backup, Veeam will not attempt to delete it and therefore no error will occur? (I expected an error to be logged somewhere)

5) Another question is about the data structure in object storage. Backups for a single object (VM) are probably stored in a specific (virtual) folder.

Buckets / [bucket name] / Veeam / Backup / [Veeam repository folder name] / Clients / [Client ID] / [Backup ID]

Is the folder name really the backup ID? When I list this ID using PowerShell Get-VBRBackupObject, it's not the same. I was only able to find out what backups are in a given "folder" based on the information in Metadata / Checkpoint.

Best regards,
Petr

[veremin]

But in my Backup Copy Job log there is an action "Full backup file merge completed successfully" and It takes some time. So does Veeam do merges or not?

The "merge" operation in this context refers to the process of updating metadata. Specifically, at this point, are we creating a new checkpoint, recording information about the restore points it consists of , and updating the references for these restore points (indicating that the oldest full now consists of specific blocks, etc.).

And thanks to Block Generation, the extension is only done once every 10 days?

Correct, the immutability prolongation occurs once in generation period, once in 10 days by default.

If the objects are not immutable, the retention policy deletes them and I cannot restore them.

Exactly. If objects are not protected by immutability, you won't be able to restore from them once they are deleted. However, with immutability in place, even after restore points are removed from the UI according to the retention period, you can still revert to the previous backup chain state and restore from them during the immutability and generation periods. This is where the concept of actual retention comes into play.

In various tests, I manually deleted data after immutability ended. Retention Policy deletes Restore Points at least from the configuration. But on object storage, it waits for an immutability period + Block Generation longer?

Correct.

Is it correct, that objects will be deleted automatically when Immutability ends?

Correct.

And in the case of an immutable backup, Veeam will not attempt to delete it and therefore no error will occur?

Immutable restore points will be removed from the UI once their retention period expires, but they will remain in the object storage until the end of the immutability and generation periods.

Is the folder name really the backup ID?

It is a cloud_backup_id that can be obtained from "backup.model.backups.directbackupinfos" table.

Thanks!

[Samuraj]

Thank you very much. Again, I see that checkpoints play an important role. Is there any documentation that describes what exactly they are and how they work?

[veremin]

A brief description of checkpoints can be found in our User Guide here.

In general, a checkpoint is a logical entity that describes the state of the backup chain at a specific point in time at the metadata level. It details which restore points are currently part of the chain and the blocks they consist of.

Thanks!

[Samuraj]

Thank you. Maybe I'm starting to understand it .
It's probably a shame that throughout the VMware User Guide the term checkpoint is only used in "Background Checkpoint Removal Job". I found a brief description in two other Guides. https://helpcenter.veeam.com/docs/backu ... ity-period, https://helpcenter.veeam.com/docs/backu ... heckpoints

[veremin]

We tried to avoid overcomplicating an already complex concept, so we limited the description of checkpoints to a couple of sentences by just mentioning that they represent a state of the backup chain at a specific point in time. Thanks!

[Samuraj]

I spent the whole weekend reading other discussions here on the forum and re-reading the explanations in the documentation. I think I understood Checkpoints. But I still wasn't clear about Actual retention. I guess I misunderstood how Immutability is extended (I thought it was only during merge operation, but sometimes the entire Backup Chain wouldn't be protected).

Finally, I tried to draw it. Are my following assumptions correct? (the description doesn't have the exact terminology)

A new Checkpoint is created for each backup. Restore Points belonging to the Backup Chain from the previous Checkpoint are used in it and a new Restore Point is added. If the objects (Restore Points) from the previous Checkpoint do not have the same Immutability as the new Restore Point, their Immutability is extended.

Another thing is the application of the Retention Policy.

• Restore Point is created, Immutability is set (Immutability Period + Block Generation Period)

• Immutability is extended to previous Restore Points (is it done now or after merging?)

• if retention is exceeded, then the Full Backup is merged with the previous Incremental Backup (within Checkpoint)

For the example I used the values:

• Immutability Period 10 days

• Block Generation 10 days

• Retention Policy 20 days

If it works like this, then a lot of things make sense to me. In Veeam I see (after a certain time) 20 Restore Points. There are 40 on the object storage and I can go back to them (Powershell).

I tried to draw the course of different days schematically. They are individual generations below each other, where the Immutability Date is always the same (even extended).



Thanks for all the advice.

[mjr.epicfail]

Im also a little bit fuzzy on the subject, there are some lightboard sessions on YouTube (https://www.youtube.com/watch?v=jrSg5oXCmXU) with for example explanation about failovers etc. That was very clear after the video, not so much with the documentation.
Maybe Veeam can create some more lightboard video's starting with this topic

[Samuraj]

So I described how I understood this whole area. https://www.samuraj-cz.com/en/article/h ... utability/
I would appreciate any corrections, additions, or clarifications. Thank you.

[veremin]

I briefly reviewed your materials on our immutability model, and in my opinion, you have grasped the principles of its operation quite well, which is impressive, as it's a complex topic and not many people can boast such a level of understanding.

The only clarification I'd like to make is regarding the extension of immutability. It occurs once per generation period. That is, when a initial checkpoint is written, all the blocks it contains receive immutability equal to "immutability period + the generation period". If a new checkpoint is created the next day, which includes both the old blocks from the previous checkpoint and new blocks, the immutability of the old blocks is not extended, while the new blocks are given immutability equal to "immutability period + (generation period - 1 day)".

Hope this makes sense.

Thanks!

[Samuraj]

Thank you very much for your time. I understood extending immutability as you describe it. But I guess I didn't describe it clearly. I'll try to rewrite it.

I described it in the article:
If objects from previous Restore Points don't have the same Immutability as new data blocks, it gets extended.
Thanks to using Block Generation, new blocks have the same Immutability date for 10 days. Therefore, extension is done once every 10 days and blocks from previous Restore Points are moved forward by 10 days.

[veremin]

I must have missed or overlooked this part in your article. In that case, you've described the extension of immutability along with the concept of the generation period correctly. Thanks!

[roland]

This a effectively complex to understand between retention period, immutability period, block generation, but i believe it's mandatory to make the good choices and anticipate what will be the storage consumption and costs, what we can restore normally, what we can restore in case of malicious action on veeam backup server. (by powershell https://helpcenter.veeam.com/docs/backu ... ml?ver=120)
It could be great to have the effective date of block deletion and more concrete example with different period (for example immutability 7, retention 14, block generation 10) to understand the overall block process on a complete cycle of more than 31 days 7 +14 +10 days...
Maybe veeam could write a kind of best practice for this ?
Thx

[veremin]

We generally tried to describe in relatively simple terms what needs to be kept in mind when working with our immutability model. The formula that describes the actual retention is specifically provided so that you can estimate the necessary space. By adding the retention period to the immutability period and the generation period, you get the value of the actual retention, and based on that, you can estimate the required space.

That is, by including 7 days of immutability on backups with a 14-day retention, you will find yourself in a situation where your backups will remain for an additional 17 days (immutability + generation), and consequently, you'll need additional space for this.

You can use the materials above if you want to understand the situation in more detail. For our part, we haven't planned to release any additional descriptions of the current immutability scheme since we want to redesign it in version 13 to reduce the requirements for additional space. Then, I believe it would be the appropriate time to release the accompanying materials.

Thanks!

[Samuraj]

What helped me the most was to draw the course of retention and immutability (like on a calendar), see my previous picture. There could be more emphasis on data that is out of retention and remains in storage. And add a longer period.
Thanks for the info about the possible change in version 13.