Veeam 12.1.2.172 to 12.2.0.334 Upgrade Fails To Upgrade Enterprise Manager

[sarnold]

Case ID: 07404674

I've done what I usually do when upgrading Veeam, which is to download and install the latest Windows Updates, restart the server, ensure Veeam Backup & Replication and Veeam Backup Enterprise Manager are fully started, and then mount the downloaded Veeam ISO for the upgrade and run the installer. I then upgrade Veeam Backup Enterprise Manager first, and then proceed to upgrade Veeam Backup & Replication.

This time, however, something's gone very wrong. I began the Veeam Backup Enterprise Manager upgrade, and everything looked good, until it ended with an error like "error occurred during certificate processing". I clicked to close the installer and relaunched the installer. However, now it only gives me the option to install Veeam Backup & Replication because it no longer thinks Enterprise Manager is installed. The installer now says under Veeam Backup Enterprise Manager that Veeam Backup & Replication must be installed first, before Enterprise Manager. However, even though it says this, if I start the upgrade of Veeam Backup & Replication and proceed through the wizard, near the last step, it shows a warning that it sees a current version of Veeam Backup Enterprise Manager and that this must be upgraded before Veeam Backup & Replication. At this point, I've ended the installer because I don't want to proceed and potentially make things worse.

So now, Enterprise Manager isn't working and is in some sort of limbo state, and the executable that the Veeam Backup Enterprise Manager service is pointing to is also non existent. Veeam Backup Enterprise Manager also doesn't appear in Programs and Features, or in the list when you click Veeam Backup & Replication in Programs and Features and click Uninstall.

If I run the \EnterpriseManager\BackupWeb_x64.msi installer and go through the wizard, it doesn't seem to see the previous version at all because it gets to the database creation page of the setup screen and it says it's going to create a PostgreSQL database, so I cancel the wizard here because I know VBEM was previously using the MSSQL installation that Veeam provides, and PostgreSQL isn't installed.

I've tried calling support twice so far, but the agents who've suggested things haven't sounded confident at all about what they're suggesting. I'm hoping this can be escalated to someone who has more experience in this area and knows what the actual problem is and how to fix it. This is the first time I've encountered an issue with Veeam upgrades since we first installed version 9 five years ago.

[david.domask]

Hi sarnold,

Thank you for your detailed write up and sorry to hear about the difficulties with the upgrade.

Checking the case, I can see that there was a remote session just recently and Support is continuing to review internally after the call, and based on the discussion from that call, the Support Engineer is reviewing options for moving forward with the case.

If you have concerns about the handling of the case, please use the Talk to a Manager button from the Veeam Case Portal. This will get you in touch with Support Management directly, and they will review your concerns and help ensure proper resources are aligned to resolve your issue.

Regarding the database as I can see that this is a point of concern for you, if there is a need to perform an uninstall and reinstall, keep in mind that the Configuration Database itself will be left alone in such cases. You can also take a native backup of the Configuration Database (with SQL Management Studio in your case) as an additional safety.

[sarnold]

Out of the 3 support engineers I've spoken to, none seem very confident about their suggestions so far. There's a lot of hesitation and "we could try this and it might work, not really sure, maybe?". I'm looking for a "here's the problem, this is how we fix it". Surely someone else has had a Veeam Backup Enterprise Manager upgrade fail before.

This is the first upgrade I've experienced that's gone sideways, and now backup jobs aren't running correctly either due to not being able to contact VBEM, and off-site backups flat out don't run at all now.

Regarding the database as I can see that this is a point of concern for you, if there is a need to perform an uninstall and reinstall, keep in mind that the Configuration Database itself will be left alone in such cases. You can also take a native backup of the Configuration Database (with SQL Management Studio in your case) as an additional safety.

I did end up taking a backup of both the B&R database, as well as the reporting database (which I believe is the one VBEM used). I've tried an installation of VBEM standalone (using the same 12.1.2.172 installer), but this doesn't work because Postgres isn't installed, and that's what it wants to use. I started this server on Veeam 9 and it's just using the built in version of Microsoft SQL server that it came with.

[david.domask]

Hi Sarnold,

If you haven't yet already, please use the Talk to a Manager button from the Veeam Case Portal if there are concerns on the handling of the case.

With your steps, if the DB fits on Microsoft SQL Express (10 GiB or less), then you can just install SQL Express to the server and during installation simply select Microsoft SQL.

[Rajesh1210]

Hi Sarnold,

Is your Veeam Enterprise Manager running on separate machine? If so, it would be worthful to check the below point from Release notes.

"Are you using Veeam Backup Enterprise Manager? If yes, consider the following:
o Important! Starting with Veeam Backup Enterprise Manager 12, a new port (port 9405) is used for
certificate communication between Enterprise Manager and Veeam Backup & Replication. Ensure
that your firewalls are configured to take into account this new port to avoid communication issues
between Enterprise Manager and Veeam Backup & Replication."

[sarnold]

If you haven't yet already, please use the Talk to a Manager button from the Veeam Case Portal

I've done this now, thank you.

during installation simply select Microsoft SQL.

That's the problem. If I attempt to run the Veeam main setup application from the 12.1 ISO, all 3 products (Backup & Replication, Enterprise Manager, and the Console) show that a newer version is already installed. If I run the 12.2 installer, it only gives me the option of upgrading Backup & Replication now since it no longer sees the existence of Enterprise Manager. If I navigate to the \EnterpriseManager directory in the ISO and run the BackupWeb_x64 setup application directly, it only wants to use Postgres as the database, and then won't proceed because Postgres isn't installed.

Is your Veeam Enterprise Manager running on separate machine?

Nope, same machine.

[ssdn108]

You mentioned 'reporter' DB somewhere on above. Do you have Veeam One also installed on same P/V by any chance? I think yes bcz 'reporters' db name is of Veeam One and not BEM from my memory. The upgrade order is> VBEM, Veeam One and then VBR. I just upgraded successfully from BEM 11.0.x to BEM 12.2 and no issues after that i.e. VBEM communicated with 13 VBRs out there at v11.0.x (i.e. data collection went fine), and also with Veeam One at v11.0.x (all sql express). My VBEM and Veeam One are on same VM (but no VBR on it). Next when i ugpraded Veeam One from v11.0.x to V12.2 (it was successfull upgrade & i was happy), i launched veeam one client and was able to see my 13 VBRs, data, etc. but i had a issue with reporter as it would never launch on port 1239 (default for Veeam One). Tried all options. 'Reports' from within Veeam One client won't launch either. Firewall was off, ports were open but no luck. But i felt relieved when SSMS had my reporter DB instance in it present, so i knew i had all the reports intact but couldn't launch the report from SSMS bcz customer did not had access to the DB (different issue). Net-net: Firefox on port 1239 worked & chrom worked on port 2741 but not on 1239. First, make sure you take your backups of VBEM and/or Veeeam one also via SSMS. Also, if it's a VM, take its snapshot.

[JGM2023]

Are you using trusted certs? We also experienced the same Issue.

We had to revert certs to use "Self Signed" certificates for the upgrade (Including API certificate), This seems to then work correctly.

We always seem to have Issues with Veeam ONE/Veeam Enterprise Manager when it comes to creating the Websites/Certificates In IIS. I am really not sure why Veeam have not provided this as a "Manual" step yet, to allow you to create the IIS website yourself If the Install fails to create it, then continue the Installation.

We've even had to turn off AV before with supports recommendation to Install the product... We should not have to turn off anti-virus just to Install a product, especially when It's supposed to be a security focused product.

Also agree that It's quite annoying when the Enterprise Manager Installation fails It almost completely removes the config so you can't "retry" you have to revert snapshots etc. and start again completely.

[sarnold]

You mentioned 'reporter' DB somewhere on above. Do you have Veeam One also installed on same P/V by any chance?

We don't have Veeam One installed. Your issue sounds different.

I think yes bcz 'reporters' db name is of Veeam One and not BEM from my memory.

The support agents I was working with said the 'VeeamBackupReporting' database was the Veeam Backup Enterprise Manager database, and the 'VeeamBackup' database was the main Backup & Replication database.

Are you using trusted certs? We also experienced the same Issue.

Our web/REST API certificates for Veeam Backup Enterprise Manager are signed by a public CA. We haven't had this issue before when upgrading Veeam Backup Enterprise Manager though.

We had to revert certs to use "Self Signed" certificates for the upgrade (Including API certificate), This seems to then work correctly.

The problem in our case is we can't even re-run the installer, because the installer is in some sort of weird state where the 12.1 installer thinks VBEM is already installed, and 12.2's installer (after the initial 12.2 VBEM upgrade attempt) now thinks VBEM isn't installed at all.

[athome]

Are you guys still having issues after the upgrade of Veeam Enterprise Manager?

[sarnold]

Are you guys still having issues after the upgrade of Veeam Enterprise Manager?

We haven't been successful upgrading Veeam Backup Enterprise Manager yet. We attempted the upgrade once, it failed, and I can't upgrade at all now.

[linus.karlsson]

Hi,

We are experiencing the same problem after trying to update Veeam Enterprise Manager from 12.1.2.172 to version 12.2.0.334.
We opened a support case yesterday and are waiting for help from the support.

Did you ever find a solution to this error?

Our support case ID:
#07413313

Best regards,
Linus

[david.domask]

Hi Linus, welcome to the forums.

Thank you for sharing the case number -- the topic creator's case is still on-going, and checking your case, the error stack looks a bit different so I'm not sure the two issues are related. Please continue with Support, as I believe your issue is unrelated to this topic. Thanks!

[mboelsch]

Are you using trusted certs? We also experienced the same Issue.

We had to revert certs to use "Self Signed" certificates for the upgrade (Including API certificate), This seems to then work correctly.

Thank you for mentioning this, been running into the exact same problem today while updating EM from 12.1.2.172 to 12.2.0.334. After reverting to selfsigned certificates in IIS and the REST API Service the update works as expected.
Never run into this problem before.

[joebranca]

Sometime after upgrading the Veeam BEM component to 12.2 (I haven't had an opportunity to upgrade VBR yet, which is on the same server), the VeeamBackupReporting DB has been undergoing a merge on the reporting database, scanning two large tables and merging the results and doing this on the local tempdb volume which has gotten full for the first time ever @ 60GB (actually at 30GB and I extended it to 60GB and it got full again). The DBA notes that the merge this is doing is needing 600GB total to run so TempDB is not nearly large enough. This is impacting a few other VBR backup processes when TempDB is full, so we'll need to increase TempDB to 700GB so this can finish.

I have never ran into anything near this kind of space issue with previous DB upgrades, is this expected with the 12.2 upgrade?

[Gostev]

Joe, let's stay on-topic please and don't mix completely different issues together in the same discussion. Creating new topics is completely free however, please start from opening a support case first to let our support engineers investigate your situation, and we go from there. Thanks

[sarnold]

Quick update to my case. I'm on Veeam support agent number 4, and this support agent recommended deleting a couple of services in order to attempt to clean up the botched Veeam Backup Enterprise Manager upgrade. I said I wasn't so comfortable with deleting services if it was going to result in an unclean uninstallation, but he assured me it would be okay.

I deleted the two service names he requested, using commands he provided. One of them deleted the wrong REST API service (the one associated with Backup & Replication, *not* the one associated with Veeam Backup Enterprise Manager). I'm currently working on restoring that service.

No progress yet on the actual fix for getting Veeam Backup Enterprise Manager working again.

[sarnold]

I'm currently working on restoring that service.

Good news; I was able to restore the service because before restarting the server after the accidental incorrect service deletion, I'd exported the corresponding service key from HLKM\SYSTEM\CurrentControlSet\Services\, and then after a reboot, restored the key and removed the DeleteFlag value. Restarted again, and the service was back and functioning.

In order to resolve the issue with VBEM not working, support (5th support agent) is now recommending completely reinstalling Windows on a new backup server, reconfiguring OS and networking, reinstalling Veeam applications, and restoring config and databases. I'd really like to avoid doing all of that if possible since everything else in Veeam, such as backups and restores, all seem to be working correctly. It's just Veeam Backup Enterprise Manager that's broken due to the single upgrade attempt from 12.1 to 12.2 failing.

[sarnold]

Just a quick update to say everything is functioning again thanks to the 5th support agent's suggestion (which was the same idea I had in my back pocket as a last resort, but glad it was tested by a support agent first before I actually did it).

Essentially, in my case, the cause of the issue was that during one of the certificate swaps for Veeam Backup Enterprise Manager, somehow the certificate store didn't update. The support agent sent me a screenshot before updating VBEM in their lab environment showing that by running the command

Code: Select all

netsh http show sslcert ipport=0.0.0.0:9398

on the backup server where Veeam Backup Enterprise Manager was installed, it showed a valid certificate, but with the Certificate Store Name value as (Null). This is apparently what caused the VBEM upgrade from 12.1 to 12.2 to fail in my case.

The way it was fixed was simply by running the 12.2 install wizard and selecting upgrade for Backup & Replication, ignoring the warning part way through the installation wizard that Enterprise Manager should be upgraded first, and proceeding with the Backup & Replication upgrade anyway. This was originally what I had in my mind as an idea, but given the warning, I didn't want to do this without a support agent actually testing it and saying it was a supported way forward. Fortunately, this succeeded with no issues.

Next, I just had to re-run the 12.2 install wizard again, and this time, select "Install" Veeam Backup Enterprise Manager, and at the Ready To Install screen of the installation wizard, select Customize Settings in the bottom left. This allowed me to change the database from PostgreSQL to Microsoft SQL, and this filled in the details of my current Enterprise Manager database and allowed me to continue the installation, upgrading the database as well.

Afterwards, I allowed the console to push the component updates to host servers and repositories, and everything is now showing as up to date on 12.2.0.334.