object storage integrity check

[mcz]

Hi everybody,

cloud object storage is much more resilent than self-managed on prem storage, I've already read an article about that. But nevertheless, it would be good to be able to run some kind of integrity checks aka SureBackup / block-validation-tool to make sure that your data on object storage is valid and usable in case if you need it.

I've already had the case that a bug in veeam-logic has silently wiped out needed/existing blocks from object storage and I only recognized it due to a FLR - otherwise I would not have noticed that. Of course, there is this new immutability feature but maybe you wouldn't "seal" all blocks for months/years and even then if there is a corruption on storage side, you wouldn't notice it.

So, are there any plans for the short- or medium-term to be able to do some kind of block-validation or even SureBackup on an object storage?

Thanks for letting me know!

[dalbertson]

Possibly, but one side effect of this would be API calls. In on prem instances of object storage this may not be an issue, but with cloud based object adding daily, weekly, monthly health checks of each object will have a big effect on costing.

[Gostev]

SureBackup on an object storage

You mean it does not work today? This is a bit unexpected to me. Perhaps it was blocked for no good reason, because we do support instant recovery from object storage - and that's all that is needed to make SureBackup work.

[mcz]

But one side effect of this would be API calls.

Of course. But if your provider wouldn't charge for that (for instance wasabi), then it wouldn't matter

[mcz]

You mean it does not work today?

Yeah I've tried to add an VM to an existing application group and there I wasn't able to use a vm from the object storage. Maybe I made something wrong but that was my approach.

[Gostev]

@Dima P. can you please check with SureBackup developers on this limitation? Thanks.

[mcz]

Strange, just wannted to double-check and it looks like that I can choose some of my offloaded backups but not all of them?? Looks like the "normal" backups are available but not the those which were created by a BCJ...

[Dima P.]

Michael,

Checking with RnD folks, stay tuned!

[Dima P.]

Michael,

We've checked in the lab and found no issues with the simple setup. Can you please raise a support case and share the ID with us, then I'll ask QA team to review the case details. Cheers!

[mcz]

Thanks Dmitry for the update. Just to make sure that it is a bug on our side:

Can you please confirm that in your lab you are able to add a vm to an application group from a copy job? I can only choose from "normal" backup jobs.

[Dima P.]

Michael,

Update: looks like a limitation of application groups - you cannot use vm from backup file created by backup copy job in application groups. Cheers!

[mcz]

Oh deer, could have checked it on my own... Anyway, now we're at the beginning: How can I now validate these contents on an object storage?

[Gostev]

Well, Dima already confirmed above there are no issues with using SureBackup against backups residing on object storage. So, at the very least you should be able to perform validation by importing those backups to another Veeam Backup & Replication install.

[mcz]

Yeah but does that really change the situation? I mean if I import those backups within another veeam instance, they are still offloaded BCJ's. Can I then use them for SureBackup? BTW: Will I need a second veeam license then because SureBackup isn't available in the community edition? What is the reason that BCJ's cannot be used in combination with SureBackup/Application groups?

And what happens when I start the backup validator from the commandline, will it then check the offloaded blocks or only the metadata?

Thanks!

[Gostev]

Yes, you should be able to use them for SureBackup. I believe the current limitation around BCJ is pretty "artificial" anyway, to help avoid a bunch of bugs with Backup Copy and SureBackup jobs overlapping with one another.

Yes, you will need to install a license in the second Veeam backup server as well.

Good question on Backup Validator. This needs to be tested, but there's a good chance that it will be checking the offloaded blocks too, because blocks location is transparent for the data mover.

[mcz]

Yes, you will need to install a license in the second Veeam backup server as well.

Can I use the same license or will I have to purchase one?

[Gostev]

Of course, you can use the same license. Our licensing policy allows for multiple installs with the same license file as long as the total instance/socket usage is not exceeded across all installs. Besides, you cannot purchase a zero instance/socket license anyway which would be required here considering this second installs does not need to back anything up.

[mcz]

Ok, that's great, thanks Anton. Now since there's an answer for my question, I'd like to ask the famous question: Will SureBackup or block validation via GUI / schedule / automation from a BCJ on an object storage ever be considered and therefore be found in future releases?

[Gostev]

As always, it will depend on the number of requests, as we have to prioritize requests that are in high demand over those that are only requested by a few customers. As it stands right now, this is the only such request in almost 1.5 years since we shipped object storage support. And frankly speaking, I don't expect too many customers doing this with object storage just because of egress and transaction costs.

[mcz]

Yeah, I understand. I think there's a good explanation why I'm the first in 1.5 years who has this request: Unfortunately I had multiple issues in combination with object storage just because the feature itself is/was quite new. I also lost backups because veeam did somehow purge them due to another issue. Within the console, everything looked normal but when I did a restore it failed and showed me something like "block missing".

Now when you never do validation checks, you would never realize it until you need it... I'm sure that all these issues will get sorted out and that we will have a very stable integration just as we are used to the other integrations. But issues can always occur and as everyone knows: Facing them during restore is the worst moment ever.

[mcz]

Just wannted to update this thread. We're copying all data to (cloud) object storage with immutability - I love it. Every now and then I import the bucket on a fresh veeam-installation and try to do a FLR to make sure that the restore would basically work. It always worked - until now: Error for every FLR on every vm. I'll create a case to see whats going on here, but it just underlines the need for an integrity check!

Are there any plans meanwhile or am I still the only one?

Thanks!

[Gostev]

Indeed, as you can see - still the only one in over 3 years

Nevertheless, the good news is this feature is becoming essential regardless. Simply because V12 will open flood gates of using on-prem object storage as primary backup repositories, requiring us to deliver all the same capabilities for them as are available today with regular backup repositories.

[mcz]

Thanks Anton!

[dave_lub]

I've experienced similar issue with backups being unrecoverable from object storage which shows the need for some sort of health check. Will this be available in V12?
2 separate issues (cases 05649650, 05658257)

[Mildur]

Hi Dave

Yes. Health check for object storage is planned for V12.
Just as a note, a health check alone will not make sure, that the machine or application will work after a restore.
Periodic restore tests should also be performed.

Thanks
Fabian

[dave_lub]

Hi Fabian,

Thanks for the confirmation if the health check for object storage works the same as the validator tool or backup copy health check, then it would have detected both of our issues.