Discussions related to using object storage as a backup target.
Post Reply
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

object storage integrity check

Post by mcz »

Hi everybody,

cloud object storage is much more resilent than self-managed on prem storage, I've already read an article about that. But nevertheless, it would be good to be able to run some kind of integrity checks aka SureBackup / block-validation-tool to make sure that your data on object storage is valid and usable in case if you need it.

I've already had the case that a bug in veeam-logic has silently wiped out needed/existing blocks from object storage and I only recognized it due to a FLR - otherwise I would not have noticed that. Of course, there is this new immutability feature but maybe you wouldn't "seal" all blocks for months/years and even then if there is a corruption on storage side, you wouldn't notice it.

So, are there any plans for the short- or medium-term to be able to do some kind of block-validation or even SureBackup on an object storage?

Thanks for letting me know!
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: object storage integrity check

Post by dalbertson »

Possibly, but one side effect of this would be API calls. In on prem instances of object storage this may not be an issue, but with cloud based object adding daily, weekly, monthly health checks of each object will have a big effect on costing.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: object storage integrity check

Post by Gostev »

mcz wrote: Jun 10, 2020 1:40 pmSureBackup on an object storage
You mean it does not work today? This is a bit unexpected to me. Perhaps it was blocked for no good reason, because we do support instant recovery from object storage - and that's all that is needed to make SureBackup work.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz »

dalbertson wrote: Jun 10, 2020 3:09 pm But one side effect of this would be API calls.
Of course. But if your provider wouldn't charge for that (for instance wasabi), then it wouldn't matter
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz »

Gostev wrote: Jun 10, 2020 3:54 pm You mean it does not work today?
Yeah I've tried to add an VM to an existing application group and there I wasn't able to use a vm from the object storage. Maybe I made something wrong but that was my approach.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: object storage integrity check

Post by Gostev »

@Dima P. can you please check with SureBackup developers on this limitation? Thanks.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz »

Strange, just wannted to double-check and it looks like that I can choose some of my offloaded backups but not all of them?? Looks like the "normal" backups are available but not the those which were created by a BCJ...
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: object storage integrity check

Post by Dima P. »

Michael,

Checking with RnD folks, stay tuned!
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: object storage integrity check

Post by Dima P. »

Michael,

We've checked in the lab and found no issues with the simple setup. Can you please raise a support case and share the ID with us, then I'll ask QA team to review the case details. Cheers!
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz »

Thanks Dmitry for the update. Just to make sure that it is a bug on our side:

Can you please confirm that in your lab you are able to add a vm to an application group from a copy job? I can only choose from "normal" backup jobs.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: object storage integrity check

Post by Dima P. »

Michael,

Update: looks like a limitation of application groups - you cannot use vm from backup file created by backup copy job in application groups. Cheers!
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz »

Oh deer, could have checked it on my own... Anyway, now we're at the beginning: How can I now validate these contents on an object storage?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: object storage integrity check

Post by Gostev »

Well, Dima already confirmed above there are no issues with using SureBackup against backups residing on object storage. So, at the very least you should be able to perform validation by importing those backups to another Veeam Backup & Replication install.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz »

Yeah but does that really change the situation? I mean if I import those backups within another veeam instance, they are still offloaded BCJ's. Can I then use them for SureBackup? BTW: Will I need a second veeam license then because SureBackup isn't available in the community edition? What is the reason that BCJ's cannot be used in combination with SureBackup/Application groups?

And what happens when I start the backup validator from the commandline, will it then check the offloaded blocks or only the metadata?

Thanks!
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: object storage integrity check

Post by Gostev »

Yes, you should be able to use them for SureBackup. I believe the current limitation around BCJ is pretty "artificial" anyway, to help avoid a bunch of bugs with Backup Copy and SureBackup jobs overlapping with one another.

Yes, you will need to install a license in the second Veeam backup server as well.

Good question on Backup Validator. This needs to be tested, but there's a good chance that it will be checking the offloaded blocks too, because blocks location is transparent for the data mover.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz »

Yes, you will need to install a license in the second Veeam backup server as well.
Can I use the same license or will I have to purchase one?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: object storage integrity check

Post by Gostev »

Of course, you can use the same license. Our licensing policy allows for multiple installs with the same license file as long as the total instance/socket usage is not exceeded across all installs. Besides, you cannot purchase a zero instance/socket license anyway :D which would be required here considering this second installs does not need to back anything up.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz »

Ok, that's great, thanks Anton. Now since there's an answer for my question, I'd like to ask the famous question: Will SureBackup or block validation via GUI / schedule / automation from a BCJ on an object storage ever be considered and therefore be found in future releases?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: object storage integrity check

Post by Gostev »

As always, it will depend on the number of requests, as we have to prioritize requests that are in high demand over those that are only requested by a few customers. As it stands right now, this is the only such request in almost 1.5 years since we shipped object storage support. And frankly speaking, I don't expect too many customers doing this with object storage just because of egress and transaction costs.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz » 1 person likes this post

Yeah, I understand. I think there's a good explanation why I'm the first in 1.5 years who has this request: Unfortunately I had multiple issues in combination with object storage just because the feature itself is/was quite new. I also lost backups because veeam did somehow purge them due to another issue. Within the console, everything looked normal but when I did a restore it failed and showed me something like "block missing".

Now when you never do validation checks, you would never realize it until you need it... I'm sure that all these issues will get sorted out and that we will have a very stable integration just as we are used to the other integrations. But issues can always occur and as everyone knows: Facing them during restore is the worst moment ever.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz »

Just wannted to update this thread. We're copying all data to (cloud) object storage with immutability - I love it. Every now and then I import the bucket on a fresh veeam-installation and try to do a FLR to make sure that the restore would basically work. It always worked - until now: Error for every FLR on every vm. I'll create a case to see whats going on here, but it just underlines the need for an integrity check!

Are there any plans meanwhile or am I still the only one? :(

Thanks!
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: object storage integrity check

Post by Gostev » 1 person likes this post

Indeed, as you can see - still the only one in over 3 years :)

Nevertheless, the good news is this feature is becoming essential regardless. Simply because V12 will open flood gates of using on-prem object storage as primary backup repositories, requiring us to deliver all the same capabilities for them as are available today with regular backup repositories.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: object storage integrity check

Post by mcz »

Thanks Anton!
dave_lub
Novice
Posts: 9
Liked: never
Joined: Jun 14, 2012 10:36 am
Contact:

Re: object storage integrity check

Post by dave_lub »

I've experienced similar issue with backups being unrecoverable from object storage which shows the need for some sort of health check. Will this be available in V12?
2 separate issues (cases 05649650, 05658257)
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: object storage integrity check

Post by Mildur »

Hi Dave

Yes. Health check for object storage is planned for V12.
Just as a note, a health check alone will not make sure, that the machine or application will work after a restore.
Periodic restore tests should also be performed.

Thanks
Fabian
Product Management Analyst @ Veeam Software
dave_lub
Novice
Posts: 9
Liked: never
Joined: Jun 14, 2012 10:36 am
Contact:

Re: object storage integrity check

Post by dave_lub »

Hi Fabian,

Thanks for the confirmation if the health check for object storage works the same as the validator tool or backup copy health check, then it would have detected both of our issues.
Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests