Question regarding enterprise manager keysets

[mcz]

Hi everyone,

I'd say this is quite a complex story, but I try to keep it simple. A while ago, I switched to v12 from v11. So far so good, but during upgrade of the enterprise manager I hit a failure and the installation aborted. Even worser, the installation has broken the EM, I wasn't able to run the old one and upgrade was also not possible.

So I created a case (#06050877) and finally we were able to upgrade using a backed up instance. What support then had mentioned was some inconsistency regarding keysets.

know possible issues with keysets after VBR DB migrations when something went wrong with it and we get "keyset is not valid" in EM log files after the VBR DB migration

Due to lack of time I closed the case and wannted to test the situation later. I though that if I use a fresh B&R instance, I could test if the EM was able to provide me a recovery key for an old backup. But a fresh instance does report "there is no EM connection", hence I cannot use the "lost password" option. Playing around with the current instance also looks dangerous to me as removing and re-importing had caused many issues in the past and I don't wannt do have it ever again.

So somehow I would love to have a way to verify if keysets for older backups are still existent. Any idea how I might be able to do that?

[david.domask]

Hi Michael,

I checked the case you listed, and the issue mentioned there was a very specific situation depending on how the encrypted backup was imported (imported by VBM versus by VBK); when you talk about removing and re-importing, you're discussing adding/removing the Veeam Server from Enterprise Manager or you're discussing importing the backups?

[mcz]

Hi David,

sorry for the confusion. Regarding adding/removing: I'm talking about the backups in B&R. I could remove an old backup for having to import again to use the password recovery via EM. But I don't wannt to do this as it might be "dangerous".

If I used a clone of the current B&R server, it would work for the clone but it would probably mess up the object storage as there are no local metadata anymore (whichs is good, btw). So I would love to have a safe way to verify if my EM has all the needed recovery keysets from all backups in the past. Thanks!

[david.domask]

Hi Michael,

No worries on confusion, it's why I clarified )

Is there a specific instance you can think of where this was a headache for you? Removing the backups and re-importing them is not dangerous and in fact a pretty normal operation, but I can understand apprehension if there were difficulties in the past, just I'm not aware of any such issues for some time (and they were very specific issues in the past).

I'm not sure on a good way to test this for you besides a few cloned servers but that seems a bit overkill -- the process itself which enables the lost password protection is pretty straight-forward so if it was not possible due to some issue at the time of backup, you would have seen an error most likely; I think mainly it's just a matter of ensuring you have a copy of the keysets exported from Enterprise Manager and the list is up to date, that should be all you need for protection -- Enterprise Manager will be able to allow the decryption as long as you have the keysets included: https://helpcenter.veeam.com/docs/backu ... _keys.html

If it were me I would just remove a backup from configuration and re-import it if I wanted assurance; I know you had some challenges with that in the past, but there should not be issues with this and if there are, a Support Case is the right way to go so it can be checked and the behavior better understood.

[mcz]

Thank you David! Actually you have mentioned where I should set the focus: How can I make sure that EM has all of it's (physical) keysets? I mean there is an entry in the DB, but the keyset seems to be a file, correct? How can I check if I'm having a file to a listed keyset? If that was fine, I'm sure it was enough for any issues in the future...

[david.domask]

Hi Michael, glad if I'm assuaging some of your concerns here

The keyset export is separate and is simply an option you have to create a backup of the existing keysets to avoid the situation I believe you're concerned about -- losing the lost password protection functionality.

You can see the creation times and list of keysets in Enterprise Manager itself, but it's only going to be ones that were created/imported on that Enterprise Manager installation. That is, if you lost the Enterprise Manager server for some reason and created a new one, it wouldn't list the previous keysets -- once you import the previous keysets, you'll see a list there.

So if you're wanting to know the list, I would just create a File Share Backup of the keysets, be sure to export a copy of the keysets every time you make a new one, and if you need to remake the Enterprise Manager Server, your change management process should include importing the previous keysets.

[mcz]

Thanks David, you're awesome! I think you have pointed me to the right path - I did a test right now and exported all listed keysets and they were downloaded without issues. Now that lets me assume that every keyset is not just and solely listed in the DB but also "correctly" stored on the EM server and ready to use for any "lost password" actions.

I also did a search through the logfiles via powershell:

get-childitem -Recurse -Filter *.log | ? {$_.LastWriteTime -gt (Get-Date).AddDays(-60)} | Select-String "keyset is not valid" -List | select Path

The result is that it hadn't found any results and that also looks quite promising.
Do you have any arguments/reservations against my assumptions?

Thanks David!

[david.domask]

Hi Michael! Thank you for the kind words, and yes, sounds like it's a pretty good plan for reassurance here. I do recommend make backups of those keysets and keep copies.

> I also did a search through the logfiles via powershell:

Sounds reasonable here, but keep in mind this just tells you that there wasn't an error _in the logs currently on the server within the last 60 days_, so I won't go so far as to say you never had such an issue but it doesn't seem likely here

[mcz]

One last question (hopefully): When you say "make backups", what do you mean? Downloading them and archive them somewhere? When I'm having a backup of the whole EM-server, those .pem-files are already somewhere on the filesystem? If the answer to the last question is yes, then I'm already having what I need - a GFS backup. Thanks!

[david.domask]

Exactly what I meant, and I agree sounds appropriate!