Upgrade from 9.5U3 to 9.5U4 failed with "Database version downgrade detected"

[eee777]

Hello guys,

I'm trying to install the latest update 4. I've followed the instructions from the release note (pages 25-27) and successfully updated the Enterprise manager first, rebooted (as requested by the setup in order to finish the update process) and then started the B&R update, but i forgot to check the . The upgrade process has failed with "Database version downgrade detected....Reboot and restart the setup".

Code: Select all

[23.01.2019 16:55:24][INFO] Installing Veeam products..
[23.01.2019 16:55:24][INFO] Installing Veeam Backup Catalog
[23.01.2019 16:55:29][INFO] Installation of Veeam Backup Catalog completed.
[23.01.2019 16:55:29][INFO] Installing Veeam Backup & Replication
[23.01.2019 16:58:48][INFO] Exception occurred (Veeam products installation section).
Info: System.ComponentModel.Win32Exception (0x80004005): Fatal error during installation
   at Veeam.Setup.MsiLibrary.VmMsiCommon.CheckError(VmMsiDatabase db, UInt32 result)
   at Veeam.Setup.MsiLibrary.VmMsiCommon.InstallProduct(String product, String commandLine, Boolean& bReboot)
   at Veeam.Setup.Wizard.VmInstallProduct.RunInstallation(Boolean& bReboot)
   at Veeam.Setup.Wizard.VmInstallEngine.Install(ShowControlFunc showActionData, Boolean bRepair, Boolean bUpgrade, VmDialogEngineHandler m_dialogEngineHandler)
   at Veeam.Setup.Wizard.PredefinedDialogs.VmInstallDialog.m_installWorker_DoWork(Object sender, DoWorkEventArgs e)
[23.01.2019 16:58:48][INFO] Starting services...
[23.01.2019 16:58:48][INFO] Services have been started.
[23.01.2019 16:59:06][INFO] Exiting...

I've restarted the setup after reboot and now it's asking me to "uninstall the console first". That's listed in the CP under Programs and Features:

Name: Veeam Backup and Replication
Publisher: Veeam Software AG
Installed on: 23.01.2019
Version: 9.5.4.2615

It does not work now (target machine actively refuses it 127.0.0.1:9392 error). There is some issue with our support ID (case #03377311) which prevent me to submit a case to your technical department. Any suggestions while i'm waiting for ticket response?

Thanks!
Regards,
E

[Dima P.]

Hello eee777,

but i forgot to check the .

Please clarify what option was not checked during EM upgrade as it looks like the part of the post was accidentally removed?

UPDATE: Is it possible to reach our support team via phone? I was told that it should be possible to quickly fix your issue online during webex.

[eee777]

Yes i forgot to check the job status before executing the upgrade. In the release not is mentioned that all jobs have to be finished with "Successful" status:

2. Ensure that the latest run for all existing jobs has completed successfully. If some jobs have failed, rerun the failed jobs. Otherwise, job upgrade procedure may fail.

Not sure if that may cause this issue. I've reached your phone support. We ended up re-installing the entire VBR. Now i'm restoring the database backup.

[mkretzer]

Is "warning" = successfull? We have warnings for some jobs because with 2700 VMs there are always having some with VMware tools issues...

[DaveWatkins]

I saw this same error doing the same upgrade on my test box. It left Veeam in a completely broken state too. I tried uninstalling what was left of U3 and did a clean install of U4 pointing it to the old DB and the same thing occured.

I then tried a new, clean DB using the SQL 2016 included and that succeeded. Trying to impoty my old config however failed with the same thing (and again left Veeam completely broken).

I suspect it's either because I had upgraded the SQL express to 2012, or because I'd applied a couple of hotfixes post U3. Not sure which but I haven't had time to get back to investigate further at this point. Happily it's my test systems so not critical, but it not fully reverting when the failure occurs was somewhat disappointing

[mkretzer]

I have a bad feeling about this. Did you all have Update 3 oder 3a before?

[engedib]

I had the same issue yesterday, worked with Veeam support to roll it back to 9.5 U3a

[Dima P.]

Folks,

2. Ensure that the latest run for all existing jobs has completed successfully. If some jobs have failed, rerun the failed jobs. Otherwise, job upgrade procedure may fail.

Upgrade engine does not care about the actual job status (it could be failed/warning/success or disabled). It's recommended to stop all the running jobs before starting the upgrade procedure. Hope that clears the confusion. Check our Help Center - Upgrading to Veeam Backup & Replication 9.5 Update 4 checklist.

[mkretzer]

So problem is running jobs? So if we disable all jobs (backup, copy, tape) we will be fine?

[Dima P.]

Yes, that could be the root cause. Keep in mind that restore activities should be suspended as well.

[DaveWatkins]

That absolutely wasn't the cause for me. My test box only has one job. Additionally shouldn't a clean install of U4 then imported the U3a config backup, not matter the job state?

In any case I was on U3a. When trying the config restore it actually shows the DB version it's using and the config backup was, as expected, much earlier than the U4 DB version. The last hotfix I had applied I _think_ was the VeeamZIP fix to stop it consuming a license and I'm pretty sure that was post the 3a install.

[mkretzer]

@gostev Can you tell us how widespread this is in support? Should we wait for a fix?

[Gostev]

This definitely has not been a widespread issue... we had 6000 downloads before GA alone, and not a single such case. I will check what's up with that.

[mkretzer]

Maybe a new bug in GA?

[SunriseFarms]

I have the same problem. All jobs are disabled. Server had been rebooted. Trying to upgrade Enterprise Manager, and it fails on "Installing Veeam Backup Catalog". Thought it might be a permissions issue on the database, but this does not appear to be the case. I've fought with it for most of the day. The error screen that comes up says "Installation was interrupted before application could be installed. You need to restart the installer to try again." VeeamOne has already been upgraded, so that is not the culprit. Is this a bad patch?

[Gostev]

Maybe a new bug in GA?

Highly unlikely, as the GA build itself had twice as many downloads than RTM already and with over 10000 downloads, I'd expect at least a few hundreds support cases by now for any real upgrade bug that is not a corner case.

[BLC]

I am having the same problem. I upgraded Veeam ONE... No issue. I then tried to upgrade EM and the install failed. I re-tried and the installer hung and froze.

UPDATE: So I rebooted the server then tried to install Veeam B&R (EM was greyed out). The install was successful.

I then tried to install EM again and it still fails. I will be opening up a SR.

[Gostev]

All, please remember to include support case ID for your "+1s" to be actionable. Thanks!

[chris_lalala]

Same Problem here with GA Version.

Last Jobs were successful. Same error when try to upgrade EM.

Code: Select all

[25.01.2019 10:07:31][INFO] Exception occurred (Veeam products installation section).
Info: System.ComponentModel.Win32Exception (0x80004005): Fatal error during installation
   at Veeam.Setup.MsiLibrary.VmMsiCommon.CheckError(VmMsiDatabase db, UInt32 result)
   at Veeam.Setup.MsiLibrary.VmMsiCommon.InstallProduct(String product, String commandLine, Boolean& bReboot)
   at Veeam.Setup.Wizard.VmInstallProduct.RunInstallation(Boolean& bReboot)
   at Veeam.Setup.Wizard.VmInstallEngine.Install(ShowControlFunc showActionData, Boolean bRepair, Boolean bUpgrade, VmDialogEngineHandler m_dialogEngineHandler)
   at Veeam.Setup.Wizard.PredefinedDialogs.VmInstallDialog.m_installWorker_DoWork(Object sender, DoWorkEventArgs e)
[25.01.2019 10:07:31][INFO] Starting services...
[25.01.2019 10:07:31][INFO] Services have been started.

[engedib]

All, please remember to include support case ID for your "+1s" to be actionable. Thanks!

My case ID: 03375628
Forgot to copy the detailed log, but it failed at one part of the database schema extension, then it displayed the downgrade detected error.

[Gostev]

The log above is actually not relevant (which is why we ask not to post random log snippet on the forums, they are usually of no use but they do trash the forums). However, looking at the relevant log file in the debug log package from a support case - thanks for opening btw - we think this may be OS locale specific bug with date/time conversion.

Overall, we have only had 1 (one) support case open on this issue before @engedib opened the second one. So, certainly does not look like a common issue at this time with 11312 downloads and counting.

Will keep you posted.

[engedib]

Should I retry the upgrade after changing the locale and time / date format from en-NZ to en-US?

[DaveWatkins]

Hmm, I'm in NZ too.... so it's another +13GMT bug because we're in daylight savings currently. No one ever expects +13

[engedib]

Don't want to imagine what would happen with those Veeam boxes on the Chathams (GMT +13:45)

[mkretzer]

So - are all users who have seen this in that timezone? It should be "safe" for the rest?

[infused]

Happened on two of my production boxes. All jobs are disabled and no failed jobs.

Going to open support case now (case id 03383348)

Edit: I'm also in New Zealand.

[Gostev]

@mkretzer seeing yet another .nz domain in Hayden's signature, I guess this is the answer to your question!

[BLC]

I am having the same problem. I upgraded Veeam ONE... No issue. I then tried to upgrade EM and the install failed. I re-tried and the installer hung and froze.

UPDATE: So I rebooted the server then tried to install Veeam B&R (EM was greyed out). The install was successful.

I then tried to install EM again and it still fails. I will be opening up a SR.

I opened up support request #03381416. The technician has me do a port scan on the Veeam server. The local server was accessing port 9443, so he recommended stopping the World Wide Publishing Service. That did the trick! EM install was successful.

[gambit120]

Yeah I saw this as well. I ended up uninstalling VBR and just installing the new version and 2016 SQL Express from scratch. Imported the config file afterwards. Luckily it was one of our small VBR instances. We have 2 more big prod environments 500-100VM's which I think I will leave for the meantime

[Rick_M]

I had the same issue - due to some Veeam Services using a service account instead of System.