I'm talking about the case # 06269372.
It started after my vacation when I had to restore the yearly restorepoint from the object-storage (part of a SOBR). This restore failed with a common message pointing to a data corruption. Now I was concerned, data corruption on an object storage when the integrity should have been checked automatically, how could that had happened?
Created cases on the veeam-side and on the object storage provider side. The OS-provider asked for the affected objects as they couldn't find anything suspicious on the bucket. Veeam struggled to give me that information. We tried a new tool, which didn't give us this information and we increased logging levels. After a while and another test for restoring the affected disk only, I realized that the restore was successful, when this was totally not expected (we thought it would fail due to the data-corruption). So what is going on here?
After doing all the verifications, the following happened:
obviously the full restore point was downloaded to the sobr weeks ago and then something bad happened on the storage-side, that's why a certain block became corrupted. Now when I restored from the OBJECT STORAGE, veeam checked which blocks it found on the SOBR and then it did the restore from the VBK with the corrupted data. But it NEVER gave me the hint, that the corruption was on the local extend and not on the object storage. Even the support team didn't notice that and so we were hunting a corruption on a storage, that did never exist there. I only had the idea about the downloaded data when I found the hint "cannot find backup files in performance tier" when trying to restore a different item from the object storage.
So, all in all, my summary and feature requests are:
- if you're restoring from the object storage and veeam hits a corruption, it should exactly tell you, WHERE that corruption is (local vbk on the extents or on the object storage - don't forget to mention the affected objects)
- doing a restore should always give you a hint about the source of the data - if it's mixed (the source) it should also be displayed
- obviously a VM file restore from object storage doesn't use already downloaded VBK on the extents, which is strange! is this a bug?
- when hitting a corruption on a vbk on local extens (like in my case), veeam should probably give you a warning and then try to restore that block from the object storage. I don't see much benefit in aborting the restore when the data is fine on the "secondary location".
