Testing backup encryption keys

[pat_ren]

Hi all,

Just working on a process for testing all of our documented encryption keys to ensure everything we have documented is 100% correct as we inherited a new client recently where the previous IT provider was unable to help them with a restore from disaster due to poorly documented encryption keys, I don't want to be that guy, but nobody is perfect and mistakes can happen. I need to avoid cached credentials and ensure I am prompted for the key and the key I enter is correct. Most of our setups are pretty simple standalone VBR servers with a local backup, a cloud copy and a config backup.

For local backup jobs, I plan to use the extraction utility to open a vbk file - this should quickly confirm the encryption key is correct for a locally stored backup file, and doesn't require extracting anything.

For a copy job (specifically to object storage) it's not as easy, if it uses the same key as the local backup job then we can confirm it's correct by just testing the local backup, however, as it would usually have a different encryption key, the only way I can think of to quickly test this would be to download the smallest full backup from the copy job and then run the extraction utility again to prompt for the key and confirm it's correct. Am I missing something or is there any quicker way to do this?

Lastly, to test a config backup encryption key we could restore the config to a new VBR server, or we could simply change it and then we know it's correct and it's unlikely we will need to restore a config backup which is not the latest available backup.

Can you guys suggest any better ways to do this? I have a few hundred keys to check, I'm not expecting any issues but just looking to save time in my testing if possible. I probably could script up a test of the local backups with the extraction utility easy enough but that doesn't help for copy jobs to object storage or config backups. Open to any other suggestions you guys may have, thank you.

[mjr.epicfail]

Is it possible for you guys to just enter a new key, and do a new ACTIVE full backup, the new chain would only require the new key

[pat_ren]

Thanks, while that would be possible, then we would still not know if the last 12 months worth of backups keys are 100% valid or not, so I need a better option than that.

[mjr.epicfail]

Yeah totally understand.
However if all backup chains are accessible now, the encryption password hash should already be in your config DB. If you make sure you create backups of that configuration and encrypt that config backup (this is needed to also backup the encryption passwords).
You should always have a way to decrypt the backups (local or cloud). As the hash is already known.

There is also a way to decrypt the current password in the local DB, Veeam support has a tool for this.

[pat_ren]

Thanks again, I have looked into config backups and we can easily just change the key for those and run a new config backup if there is any uncertainty about the validity of the key. Then we do have that protection.

If Veeam support has a tool to decrypt the keys from the local DB I may look into this as I could automate that across all of the clients I want to check (which would save me a lot of time) - I will look into this further, thanks.

[pat_ren]

For anyone else in this position, found the answers I needed in this KB
https://www.veeam.com/kb4349

[pat_ren]

That didn't quite work as expected to also extract encryption keys but has put me on another path now to test some other stuff.

I contacted Veeam support too as mentioned above, but they won't help unless you're stuck in a situation where you can't recover due to a lost key, then it seems like maybe they will give it a go, but wouldn't share any further info beyond that.

[pat_ren]

Ok me again, just confirming as expected it was trivial to extract the encryption keys by simplify modifying the KB above - think I can work with this to automate testing most of my servers regularly and save myself a lot of time in future. Hopefully this does help someone else in future too as I wasted a lot of time looking into this and there's a lot of other bad advice out there on how to do this.

[Mildur]

Hi guys

Happy to share a sneak peak about our upcoming version 13
If everything goes as planned, you will be able to test encryption passwords directly from the encryption password manager:



Best,
Fabian

[pat_ren]

Hi Fabian,

Thanks for sharing, that's a great addition and really useful as there has not been a good way to do this so far, the testing methods are all either quite risky or quite manual. There was a similar feature in Veeam365 backup v7 but for some reason they removed it in v8.

My solution was just to script this testing for now, I was able to write some simple code get my plain text encryption keys out of all our standalone veeam agent jobs and vbr servers/jobs quite easily and so that turned out to be the best solution for me as I have a LOT of servers to test, and didn't want to do it the hard way.

V13 seems like a great step in the right direction.

[TWuser]

Good topic, and good to hear the news on v13. I'm also working through encryption verification before refreshing them all.

[MPECSInc]

Yeah totally understand.
However if all backup chains are accessible now, the encryption password hash should already be in your config DB. If you make sure you create backups of that configuration and encrypt that config backup (this is needed to also backup the encryption passwords).
You should always have a way to decrypt the backups (local or cloud). As the hash is already known.

There is also a way to decrypt the current password in the local DB, Veeam support has a tool for this.

StorageCraft ShadowProtect, the last product that we used prior to taking up with Veeam, had absolutely no way to recover the encryption keys. None. Nada. Zippo. Zilch.

There were plenty of warnings prior to hitting GO when setting up backup encryption.

All y'all just opened a Pandora's Box by making public the keys are easily accessible.

That renders the whole "security" narrative moot.

Am I wrong?