BCJ / offload destroyed by cancelling BCJ?

[mcz]

Hello,

since quite a while I'm having issues and finding bugs in combination with offload/object storage and last week, I've "found" another issue, which is maybe due to the new BCJ copy mode. My support engineer (T2) wrote me the following:

Also, I tried to quickly recreate the chain of events that lead to the error, and here's what I noticed:
1. On 03.06 you disabled the job in the middle of merge:
[03.06.2020 17:51:50] <01> Error Transform step failed
[03.06.2020 17:51:50] <01> Error Job has been disabled by user VEEAM\admin (Veeam.Backup.Common.CDisableSessionException)
2. Since then it began to fail with the same error on every run:
[03.06.2020 18:02:26] <01> Error Unable to create a restore point: some backup files backing previous restore points are offloaded, please run an active full backup (System.Exception)
3. Until you forced active full on 05.06, after which it failed with the DB error:
[05.06.2020 17:26:13] <01> Info Violation of UNIQUE KEY constraint 'UQ_CryptoKeyRecoveryRecs_EncryptedKeyId_RecoveryKeyId'. Cannot insert duplicate key in object 'dbo.CryptoKeysRecoveryRecs'. The duplicate key value is (221bc60a-56bc-4584-862b-8d30e7a434c4, 83898d5a-41b0-44aa-ae15-d921dd25f8f6).
[05.06.2020 17:26:13] <01> Info The statement has been terminated.]
[05.06.2020 17:26:13] <01> Error Microsoft SQL server hosting the configuration database is currently unavailable. Possible reasons are heavy load, networking issue, server reboot, or hot backup.

So it looks like it was the cancelled merge that caused the error. Moreover, it might be that it caused the other error too.

Now I needed to create a new BCJ. Now there's no need to blame anyone for that, the only thing that always annoys me is the fact that due to a new BCJ, you loose the whole deduplication-benefit on s3 and unfortunately it's not the first time that I had to start with a new chain/BCJ due to issues.

Just wanted to post this case here so that the product can be improved.

Thanks!

[veremin]

Can you share the ticket ID with us? Thanks!

[mcz]

Vladimir, the ticket # is 04107801, but many other issues in combination with s3 have been handled there if I'm not mistaken. So you might not find what you're looking for if you start reading the case from the beginning

[veremin]

I've reached QA team. They will verify the described scenario. I will update the topic, once I have more information. Thanks!

[mcz]

Thanks Vladimir. Just another thought: I know that veeam warns you when you abort a BCJ saying that it might lead to data corruption. But when I confirmed, I would have only expected that the currently merged/created restore point could get corrupted so that you would rebuild it in the worst case - or better said veeam would do that for you. I never expected issues like that the whole BCJ could get unusable, which is a bit more severe IMHO. So if that issue was "expected", please adopt the warnings - thanks.

[veremin]

Honestly, the issue does not look expected. So, we will check the issue on our side to see what might be the root cause. Thanks!

[sykerzner]

Is there anything new on this?

We did some much needed HW upgrades on our VCC servers, which probably involved cancelling a few Copy Jobs. Some backups in some SOBRs are failing with the same errors in the same sequence.

Is the only fix really to create new jobs?
In one case, I made the client VBR forget the cloud repo and then re-added it. Mapping didn't prevent a new Active Full and the old ones only shows up as Imported.

[mcz]

be aware that the used veeam version was way different then what we have today...