Encryption state is invalid on scale-out repository with global encryption

[eider]

== Case ID: #04891930 ==

I have B&R server set up with two repositories. One is for servers (`pve`), another is for computers. The one used for computers (`pc`) has forced encryption password set via `Access permissions...` menu as it is used by non-managed agents to store backups. The other one is used purely by jobs managed by B&R and does not require this configuration (instead each backup job has encryption password specified separately). I also perform monthly backups to tape from both of these repositories.





Recently I have migrated these repositories to scale-out ones. Ever since then, backup to tape fails for the `pc` repository with an error message: Encryption state is invalid (Veeam.Tape.Core.TapeMetaGenerationException)

The backups that are stored on `pve` repository (which is identical scale-out repository configuration on same bucket, just using different folder) work perfectly. Since the only difference between these two repositories is selection of encryption password repository-wise (both repositories are set to additionally encrypt data offloaded to object storage) I am inclided to believe that B&R is visibly confused about existence of two separate encryption keys for single backup (one that is set up repository-wide for all backups that are stored on performance tier, and another that is used for backups offloaded to capacity tier). This is clearly not an issue when backups are encrypted per-job basis, but seems to cause issue when they are encrypted repository-wide.

This is possibly because encryption password is set globally on `pc` repository which encompasses both capacity tier as well as performance tier, while capacity tier having its own encryption key using same password.

--Removed by moderator--

[PetrM]

Hello,

Thanks for sharing a support case ID! However, I recommend to avoid posting log snippets on forum as they just worsen readability and we don't troubleshoot technical issues over the forum posts. Let's wait for what our support engineers can figure out based on debug logs analysis.

Thanks!

[eider]

Hi @PetrM, I do understand that however I believe this particular snippet was very important for people looking for this thread in future as it includes important information as to why B&R thinks encryption state is invalid (namely, that it found 2 user keys for backup when it expected one).

[PetrM]

Hello,

Your willingness to contribute to the Community is highly appreciated! The provided description is full and comprehensive, it's totally enough for people who will find the topic in future.

Thanks!

[eider]

Unfortunately it seems like the support case is on its way to be closed due to lack of active support subscription from my side. That is of course fine and I understand this, however I would still like for someone to note of this issue internally as this is very much most likely a logical issue in the process of requesting user encryption key from database and should be fixed in new GA patch release.

I have not included this information before as to not muddle the waters for support team analyzing the logs, however this is second month this has happened so far. Previously I have successfully worked around this by destroying scale-out repositories and recreating them anew then starting the tape job BEFORE they had a chance to offload their data to remote end (so there was no second encryption key for capacity tier yet in the database for the backup jobs) and this has worked successfully.

If no response will be given by the time next month rolls out I will most likely attempt to recreate this specific repository by removing it and recreating with the 'Encrypt data uploaded to object storage' option disabled (as it is redundant according to Veeam documentation when both encryption phases use same password and all backups are already encrypted once repository-wide) and get back with information if this fixed the issue. This will however take time as for non-managed agents Veeam refuses to offload existing backups until new one is created by agent.

Another point of this issue is that I have left the tape job to finish and despite it failing, I can confirm that B&R sees these backups as performed and can browse them. If you wish, I can attempt to restore one of them and see what happens.

[PetrM]

Hello,

Yes, I agree. I'll ask our support team to review the case. I'd suggest to keep on hold all tests while we're waiting for the reply.

Thanks!

[eider]

Coming here again to report that the case was silently closed with no reply due to lack of subscription; I will hence attempt to resolve this in my own capacity as stated above and report back.

[Dima P.]

Hello eider,

Thank you for the update and sorry to hear that your case was closed. I've asked QA team to take a quick look at the logs attached to the case. Will update this thread based on results.

[Dima P.]

eider,

We've reopened the case. Seems that old logs were removed due to the support policy, can you please upload new log to continue the investigation? Thank you in advance!

[lyapkost]

eider,

Thank your for uploading new potion of the debug logs. We've noted the bug (#337169) and will continue the investigation on our end.