Discussions related to using object storage as a backup target.
Post Reply
mcz
Veeam Legend
Posts: 893
Liked: 198 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

wasted a lot of time with data corruption & feature request

Post by mcz » 1 person likes this post

Hi everyone,

I'm talking about the case # 06269372.

It started after my vacation when I had to restore the yearly restorepoint from the object-storage (part of a SOBR). This restore failed with a common message pointing to a data corruption. Now I was concerned, data corruption on an object storage when the integrity should have been checked automatically, how could that had happened?

Created cases on the veeam-side and on the object storage provider side. The OS-provider asked for the affected objects as they couldn't find anything suspicious on the bucket. Veeam struggled to give me that information. We tried a new tool, which didn't give us this information and we increased logging levels. After a while and another test for restoring the affected disk only, I realized that the restore was successful, when this was totally not expected (we thought it would fail due to the data-corruption). So what is going on here?

After doing all the verifications, the following happened:

obviously the full restore point was downloaded to the sobr weeks ago and then something bad happened on the storage-side, that's why a certain block became corrupted. Now when I restored from the OBJECT STORAGE, veeam checked which blocks it found on the SOBR and then it did the restore from the VBK with the corrupted data. But it NEVER gave me the hint, that the corruption was on the local extend and not on the object storage. Even the support team didn't notice that and so we were hunting a corruption on a storage, that did never exist there. I only had the idea about the downloaded data when I found the hint "cannot find backup files in performance tier" when trying to restore a different item from the object storage.

So, all in all, my summary and feature requests are:
  • if you're restoring from the object storage and veeam hits a corruption, it should exactly tell you, WHERE that corruption is (local vbk on the extents or on the object storage - don't forget to mention the affected objects)
  • doing a restore should always give you a hint about the source of the data - if it's mixed (the source) it should also be displayed
  • obviously a VM file restore from object storage doesn't use already downloaded VBK on the extents, which is strange! is this a bug?
  • when hitting a corruption on a vbk on local extens (like in my case), veeam should probably give you a warning and then try to restore that block from the object storage. I don't see much benefit in aborting the restore when the data is fine on the "secondary location".
Do you agree? Happy waiting for the answers - thanks! :)
veremin
Product Manager
Posts: 20290
Liked: 2259 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: wasted a lot of time with data corruption & feature request

Post by veremin »

Let us verify the case outcome and a few other things on our side and post back. Thanks!
mcz
Veeam Legend
Posts: 893
Liked: 198 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: wasted a lot of time with data corruption & feature request

Post by mcz »

Thanks Vladimir! Basically I'm done with the case but I'll come back once it's closed.

edit: the case is now closed
veremin
Product Manager
Posts: 20290
Liked: 2259 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: wasted a lot of time with data corruption & feature request

Post by veremin »

Sorry for a delay, but we are still confirming specific questions internally. Once everything is clarified, I will update the thread. Thanks!
veremin
Product Manager
Posts: 20290
Liked: 2259 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: wasted a lot of time with data corruption & feature request

Post by veremin »

Now when I restored from the OBJECT STORAGE, veeam checked which blocks it found on the SOBR and then it did the restore from the VBK with the corrupted data.
Upon further investigation, we found a product bug that leads to performance tier data still being re-used during restores executed from the capacity tier node. If not for a bug, you wouldn't have come across this backup chain issue. The bug has been noted and will be addressed in the next product version.
doing a restore should always give you a hint about the source of the data - if it's mixed (the source) it should also be displayed
This information is already provided within the debug logs, but we might add it to the restore sessions as well.
obviously a VM file restore from object storage doesn't use already downloaded VBK on the extents, which is strange! is this a bug?
Just to be sure - by VM file you mean .vmdk, .vmx and similar (not guest OS files), correct? If so, was the restore executed from the capacity tier node?

Thanks!
mcz
Veeam Legend
Posts: 893
Liked: 198 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: wasted a lot of time with data corruption & feature request

Post by mcz »

The bug has been noted and will be addressed in the next product version.
Perfect, thank you.
but we might add it to the restore sessions as well.
appreciate it, thanks.
by VM file you mean .vmdk, .vmx and similar (not guest OS files), correct?
yes. If I understand the direction you're pointing at, it should not use the performance tier when restore is initiated from capacity tier. So if I got you right, this behaviour is expected. I just didn't understand the difference between this method and the first point, but as we now know that this was due to a bug, I do understand now. thanks.
veremin
Product Manager
Posts: 20290
Liked: 2259 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: wasted a lot of time with data corruption & feature request

Post by veremin »

Your understanding is correct, restores executed directly from the Capacity Tier node should not re-use the blocks residing on the Performance Tier. Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests