Upgrade to v11 failed and rollback uninstalled Monitor

[skrause]

I haven't ever had major issues with a Veeam product update before so this is a bit surprising and now almost infuriating.

Our VeeamONE database is hosted on a separate SQL server and I was having issues with "failed to connect to 'SQLSERVER'" in the installer, opened a (Case #04675657) and as I was collecting logs and talking to my DBA to see if there were any failed login messages realized that there were some read permissions on the system databases that VeeamONE requires. We had moved the database between SQL clusters a couple of months ago so we probably missed those permissions then.

I asked the DBA to set the permissions (except the 'CREATE ANY DATABASE' one) listed and tried the installer again and everything appeared to be working as the installer went dilligently about its business. So I closed the ticket (as I now know prematurely.)

The installer got all the way through the installation and said "Unable to connect to the database, rolling back installation" and it rolled it back all right: It totally REMOVED VeeamONE Monitor Server from the machine but left reporter there.

I also cannot run the installer again because it says that "components are missing, please remove the components manually and run the installer again".

New case opened to reference the old one: Case #04678061

As a bit of background: the VeeamONE services run under a service account and connect to the database that way. I ran the installer under my account giving it those credentials in the Wizard. I have noticed in the past that if I open the VeeamONESettings.exe application under my account it has a popup that says "unable to connect to database" but then opens the config just fine. My account has no permissions on the database per our normal security policy of 'least privilege'.

I will most likely be restoring the VM that VeeamONE runs on to its backup from last night to get some form of VeeamONE monitor back, but this is definitely NOT a behavior I would have expected from a Veeam upgrade install.

[wishr]

Hi Steve,

Sad to hear you've got into this difficulty and thank you for the feedback. Let's see what our technical support team will discover if you are up for an investigation of course.

Thanks

[jspeicher]

have you come up with a resolution. Veeam 11 has crashed 2 of my systems. luckily wasn't heavily relying on veeam one, but this is 2 systems that crashed during the v11 upgrade, veeam backup and replication died on 1 of the 3 systems as well.

[wishr]

Hi Jason,

Did you have difficulties specifically with Veeam ONE (since this is a Veeam ONE forum)? Are these resolved now? Could you please share the numbers of the support cases you've opened on these issues so we could take a look at the details?

Thanks

[skrause]

So the issue was this:

The VeeamONE v11 installer (like the VeeamONESettings.exe application) tries to connect to the database using the credentials the application is launched under not the service account credentials at some point.

When I worked with the support engineer on the phone we tried to run the installer again (as me) after doing a full uninstall and it gave the same error. My DBA was able to see a failed login attempt on the SQL server using my credentials at the time that the installer bombed. Once we tried running the installer using RunAs:<service account> it worked fine.

Interestingly enough, the database was upgraded to version 11 during the first setup that failed and rolled back yesterday. So it is just something the installer is checking when it does its installation finalization that appears to not follow the account settings laid out in the wizard.

So to sum up: if you run VeeamONE under a service account and the database has permissions limited to the service account, you need to run the installer AS the service account and not as another administrator of the server in question.

I would assume that the product team will fix that at some point (or at least update the documentation to reflect this behavior).

[wishr]

Hi Steve,

Thank you for the feedback. Could you please also share your support case ID so we could track the issue?

[skrause]

I put it in the original post

They re-opened the original case and merged the two: Case#04675657

[wishr]

Yes, sorry about that!

Btw, we are looking into that issue.

[wishr]

Hi Steve,

Just a quick update: we've identified an issue in the upgrade logic and will fix it in the next version. Currently, the workaround is to run the setup on behalf of the service account, as you suggested above.

Thanks for bringing up this topic!

Thanks

[skrause]

Thanks for the update.

Depending on how long it takes to post the new installer version for download it might not hurt to add this to the known issues list on the docs?

[wishr]

Sure, we'll add it to the release notes doc, known issues, upgrade section.

Update: the updated release notes document is live.

[spiritie]

Got the same error. Only way to fix it was to log into Windows as the Service account which Veeam ONE runs under.

[wishr]

Hi Gert,

Have you tried launching the setup on behalf of the service account while being logged in to the OS normally?

Thanks

[spiritie]

Hi wishr

I tried this to start with but it also didn't work. The only thing that worked was to log into the user it self in Windows.

[wishr]

This is indeed strange. Passed this info to our QA team for analysis.

Thanks

[skrause]

You know, I may have been logged in as the service account to windows when it worked and not using "Run As a Different User" on the installer.

I know I was logged in as that user at some point but I thought it was just to test the "VeeamONESettings.exe" launch.

[wishr]

Thanks, gentlemen, we'll test this out one more time!

Update: we've performed a few tests again and can confidently say that the aforementioned issue does not happen if you run the setup on behalf of the service account. There is no need to be logged in to the Veeam ONE server using the service account. For future readers: if it does not work for you - please open a technical support case and share your case ID for reference.

Regards,
Fedor