download/restore VBK from archive tier does not lookup for existent blocks on local extents

[mcz]

Hi everybody,

I've done some restores/downloads from our archive tier (object storage) the last time and realized that it would always download the whole vbk. It wouldn't surprise me if I hadn't got something in my mind that veeam first looks if there are some blocks already on some local extents (which should have been the case because we have held some fresher backups regarding that vm)...

Has the logic changed in some of the latest updates?

I also guess that this functionality is limited to extents in the same SOBR, probably this will be extended in the future? It might be that you're holding an older backup of a particular vm on a different SOBR or repository and so you might have more hits because less data has changed...

Thanks!

[Mildur]

Hi Michael

I've done some restores/downloads from our archive tier (object storage) the last time and realized that it would always download the whole vbk. Has the logic changed in some of the latest updates?

Is your question about our Veeam Capacity Tier or our Veeam Archive Tier?
We changed the logic for the capacity tier with V11a. If you restore from the capacity tier tree in the vbr console, Veeam will restore anything from the capacity tier even there are existing blocks on the performance tier.
If your question is about the Veeam Archive tier, I need to check it first.

I also guess that this functionality is limited to extents in the same SOBR, probably this will be extended in the future? It might be that you're holding an older backup of a particular vm on a different SOBR or repository and so you might have more hits because less data has changed...

A machine has it's own backup chain and own dedicated backup job for each repository. We cannot merged blocks from different backup chains on different repositories. The blocks in the backup chains are not connected to each other.

[mcz]

Is your question about our Veeam Capacity Tier or our Veeam Archive Tier?

capacity tier, sorry for the confusion

We changed the logic for the capacity tier with V11a. If you restore from the capacity tier tree in the vbr console, Veeam will restore anything from the capacity tier even there are existing blocks on the performance tier.

Why was that changed? I found it very useful to only download blocks that are not on the local side already.

[Mildur]

Thanks.
We added it, so that our users can do restore tests directly from the object storage. This was not possible before without any manual interaction (Setting Maintenance Mode on Performance Tier) who leads to backup jobs which can not run.

It was documented in the release notes:
https://www.veeam.com/kb4215

Restore directly from Capacity Tier: when a restore process is initiated from backups stored in the Capacity Tier (using the Backups > Object Storage node), we will no longer read matching data blocks from the Performance Tier and instead use solely the data from object storage. This capability can be useful for performing periodic non-disruptive restore tests directly from the cloud without putting your Performance Tier into maintenance mode.

[mcz]

Thanks Fabian, makes sense from this point of view... But still, there might be the requirement to download only the deltas for time/cost savings. Is it possible to trigger the old mechanism?

[veremin]

The old mechanism is present in the general backup node - if you initiate restore from general backup node (instead of object storage one) and select a restore point located in Capacity Tier, the product should try to reuse blocks from Performance Tier.

However, while retesting it today, we discovered a product bug that prevents usage of local data - currently all blocks are taken from Capacity Tier regardless of what node the restore is executed from. It will be addressed in the later product versions.

Thanks!

[mcz]

Thanks Vladimir, regarding the fix in future versions: Will it include the download from capacity tier? Because you can only trigger it directly from the object storage backups...

[veremin]

You can trigger restore from Capacity Tier in several ways:

- Initiate restore from Object Storage node
- Initiate restore from Backups node and select restore point present in Capacity Tier

In the former case the blocks from Performance Tier will not be reused, in the later case - they will be.

Unfortunately, the later case is currently broken and reusage mechanism is not utilized - however, we treat this as unexpected behaviour that will be addressed in the future product versions.

Thanks!

[mcz]

If you say "restore", I guess it includes all possible kinds of restore (FLR, IR, etc.)? Is there a chance in the future to only download the deltas if you for instance wanna pull down a vbk for some archiving reasons (backup to tape for instance)?

[veremin]

As soon as bug is fixed, the second option I've described will give you exactly that. Or am I missing the point here and you want something different?

[mcz]

Well, I do not understand a restore as the same as a VBK download from the object storage... If I select the restore point in the backup properties and select "copy to performance tier", that's not the same action as if I'd do a FLR for instance...

[veremin]

Ah, my bad - I thought for some reason that we were still talking about restore.

During backup download from Capacity Tier, product automatically checks whether blocks to copy are already present on Performance Tier and re-use them in case of positive answer. So only deltas are downloaded.

Also, I apologize for providing a bit misleading information: the bug I mentioned exists only in v12 builds, not in 11. We are currently working hard on delivering second v12 beta, so whenever the question about bug or product inconsistency arises among different departments, everyone automatically assume it's v12 that is being discussed.

Surely the bug will be fixed by the time v12 goes GA.

Thanks!

[mcz]

Thanks Vladimir, very useful information. But still, I think I've got an issue here, so please let me ask the question different: Would I see it in the UI (v11) when I'm downloading a VBK from the object storage that there are several bytes pulled from the local extents?

[veremin]

Not directly, but in the download session you will see the size of backup to download and how much data is read from object storage. So if subtract one from another, the resulting value will be size of data used from local storage. We might think about better logging mechanism in one of the next product releases. Thanks!

[AlexHeylin]

Can I suggest it would be helpful if it was very clear to an idiot operator who did not RTFM and now needs to do an emergency restore, before the restore is started if the restore will be combined (only pulls some blocks from object) or remote-only (pulls all block from remote) - and how to make it do the opposite of the one being done now? Confirmation of this on the final screen with the GO button might be a good way to ensure this. The user should not be expected to have kept up with every little nuance of the UI in every release.

[veremin]

Agree with the last statement that user indeed does not need to familiarize him or herself with each and every UI nuance, thus UI tries not to bug user with additional questions and typically selects the best option automatically (like reusing local block instead of downloading everything from object storage).

But I get the idea, it's been noted.

Thanks again for sharing the feedback; much appreciated!

[mcz]

Vladimir, I checked some real-time scenarios (v11) and I don't see any "local pulls" for SOBR downloads. Can you please confirm the following:

- go to backups -> object storage node
- right click on a backup => properties
- select a vm where there's at least one restore point on the performance tier
- select a restore point which is solely on the capacity tier and then click on "copy to performance tier"

Now on your side the download would only pull the deltas, correct?

In my case I was downloading a 20 GB (vbk-size) vm (windows server 2019 DC) and it always downloaded around 19.7 GB which is way too much if it only has pulled the deltas...

[veremin]

What happens, if you try the same steps, but from Backups (not Object Storage) node?

[mcz]

Hi Vladimir, I'm currently doing a download, but if I choose the backups node, I will at least see the hint "x GB will be copied from performance tier", so it looks like that the logic here does get applied...

[veremin]

Correct, as mentioned above there is a different logic used depending on the place from which download or restore is executed. If executed from Backups node, download and restore process re-utilize the local data. If executed from Object Storage node, then only data in object storage is used. Thanks!

[mcz]

Hi Vladimir, this makes totally sense - as long as you know it. As you can see I wasn't aware of this differentiation, so I would be happy to get a hint somewhere in the restore-process like "downloading whole data from object storage, to re-utilize local data use backups-node", that would be very helpful. Thanks!

[veremin]

Might be a good idea, will keep it in mind for future product versions. Thanks, Michael!