Case # 04554115 Error: Invalid encrypted PEK blob length

[patricknh]

So after testing Veeam backup and recovery for months in our test environment as as AD backup and recovery solution that could restore accounts and passwords, we deploy to production and on the first test of the product below we get following error, saying it cannot restore the password due the the registry hive not being able to be loaded. Below is snippet from the log, and the only thing that stands out is the encrypted PEK blob length. Otherwise the setup is the same as in test. The registry hive is there in its usual place and the backups are of the entire DC.

Code: Select all

12/18/2020 2:14:07 PM    9 (10700)   Closed successfully
12/18/2020 2:14:07 PM    1 (9580) Error: Invalid encrypted PEK blob length
12/18/2020 2:14:07 PM    1 (9580) Type: System.Exception
12/18/2020 2:14:07 PM    1 (9580) Stack:
12/18/2020 2:14:07 PM    1 (9580)    at Veeam.ActiveDirectory.Restore.PEKDecryptor.OldDecrypt(Byte[] encrypted)
   at Veeam.ActiveDirectory.Restore.PEKDecryptor.Decrypt(Byte[] encrypted)
   at Veeam.ActiveDirectory.Restore.DomainPasswordManager.AddDomains(LoadedHiveReader hiveReader, IADStore store)
   at Veeam.ActiveDirectory.Restore.RegistryHiveLoader.Load(IDomainPasswordManager domainManager, IADStore adStore, IVBRMountedData mount, String hiveFileName, String storeFile)
   at Veeam.ActiveDirectory.Explorer.Tasks.AsyncMounter.ProcessStore(AdStoreConfig currentConfig, String hiveFileName, IVBRMount mount)
   at Veeam.ActiveDirectory.Explorer.Tasks.AsyncMounter.Run(IProcessObserver observer, CancellationToken ct)
   at Veeam.Presentation.Async.VisualAsyncTask.Execute(IProcessObserver observer)

Anyone else seen this in their AD environment? Thoughts?

[Natalia Lupacheva]

Hi Patrick,

First, thank you for sharing your case id!
I see you have quite an active discussion with Support.
You case does not seem to be a common issue, so please keep in touch with Support.

I've checked if we've faced this before but it looks like this can be a propagated error, so the logs analysis is required here.

Thanks!

[patricknh]

Out of curiosity, do you have any cases similar to this after RC4 encryption was disabled in and AD environment?

[PetrM]

Hi Patrick,

I've managed to find just one similar case in our support ticket tracking system but this one has been closed without provided solution as there was no possibility to reach the end customer and to perform the necessary tests. Apparently, the issue is not trivial so our support engineers need to have some time in order to figure out the possible causes.

Please keep in mind that you always can talk to a manager to escalate the case to a higher level.

Thanks!

[patricknh]

2 months in and still no success. To me what appears to be happening is that veeam cannot decrypt piece of either the .dit or the registry hive in order to access the same database. Any chance you can share which company had this issue? I have also opened a case with microsoft for their assistance. Also Veeam.ActiveDirectory.Restore.PEKDecryptor.Decrypt(Byte[] encrypted)
at Veeam.ActiveDirectory.Restore.DomainPasswordManager.AddDomains(LoadedHiveReader hiveReader, IADStore store) appear to to be functions written by veeam, is there not a way to get more granular debugging around that point in the call?

[patricknh]

So in continuing to figure out what is going on here. Quick question, since to decrypt the system hive the bootkey from the dc being backed up is required- is it trying to pull that bootkey from the backup itself in the veeamflr mounts, or does it need to reach out across the network to get it?

[PetrM]

Hi Patrick,

As you've correctly noticed, the mentioned functions represent a part of our code however I would let our support team to move on with a more precise debugging. As far as I see, the case is already escalated and RCA is still in progress. Let's wait for a while, I'm pretty sure that our support engineers will be able to figure out the cause of this behavior.

Thanks!

[patricknh]

Just a follow-up on this in case anyone comes across that error again. This was due to the DC's in question being promoted using IFM. This results in two PEKs which the VEEAM tool didn't know which to chose (more of less) Veeam fixed this in the 11.x release and it should only be an issue if you were trying to restore an AD account and password from a DC that was promoted with the Install From Media method and your using a version of Veeam that predates v11.