Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V
Post Reply
skrause
Expert
Posts: 463
Liked: 98 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Upgrade to v11 failed and rollback uninstalled Monitor

Post by skrause » 1 person likes this post

I haven't ever had major issues with a Veeam product update before so this is a bit surprising and now almost infuriating.

Our VeeamONE database is hosted on a separate SQL server and I was having issues with "failed to connect to 'SQLSERVER'" in the installer, opened a (Case #04675657) and as I was collecting logs and talking to my DBA to see if there were any failed login messages realized that there were some read permissions on the system databases that VeeamONE requires. We had moved the database between SQL clusters a couple of months ago so we probably missed those permissions then.

I asked the DBA to set the permissions (except the 'CREATE ANY DATABASE' one) listed and tried the installer again and everything appeared to be working as the installer went dilligently about its business. So I closed the ticket (as I now know prematurely.)

The installer got all the way through the installation and said "Unable to connect to the database, rolling back installation" and it rolled it back all right: It totally REMOVED VeeamONE Monitor Server from the machine but left reporter there.

I also cannot run the installer again because it says that "components are missing, please remove the components manually and run the installer again".

New case opened to reference the old one: Case #04678061

As a bit of background: the VeeamONE services run under a service account and connect to the database that way. I ran the installer under my account giving it those credentials in the Wizard. I have noticed in the past that if I open the VeeamONESettings.exe application under my account it has a popup that says "unable to connect to database" but then opens the config just fine. My account has no permissions on the database per our normal security policy of 'least privilege'.

I will most likely be restoring the VM that VeeamONE runs on to its backup from last night to get some form of VeeamONE monitor back, but this is definitely NOT a behavior I would have expected from a Veeam upgrade install.
Steve Krause
Veeam Certified Architect

wishr
Veeam Software
Posts: 2290
Liked: 290 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by wishr »

Hi Steve,

Sad to hear you've got into this difficulty and thank you for the feedback. Let's see what our technical support team will discover if you are up for an investigation of course.

Thanks

jspeicher
Service Provider
Posts: 17
Liked: 1 time
Joined: May 09, 2013 7:49 pm
Full Name: Jason
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by jspeicher »

have you come up with a resolution. Veeam 11 has crashed 2 of my systems. luckily wasn't heavily relying on veeam one, but this is 2 systems that crashed during the v11 upgrade, veeam backup and replication died on 1 of the 3 systems as well.

wishr
Veeam Software
Posts: 2290
Liked: 290 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by wishr »

Hi Jason,

Did you have difficulties specifically with Veeam ONE (since this is a Veeam ONE forum)? Are these resolved now? Could you please share the numbers of the support cases you've opened on these issues so we could take a look at the details?

Thanks

skrause
Expert
Posts: 463
Liked: 98 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by skrause » 1 person likes this post

So the issue was this:

The VeeamONE v11 installer (like the VeeamONESettings.exe application) tries to connect to the database using the credentials the application is launched under not the service account credentials at some point.

When I worked with the support engineer on the phone we tried to run the installer again (as me) after doing a full uninstall and it gave the same error. My DBA was able to see a failed login attempt on the SQL server using my credentials at the time that the installer bombed. Once we tried running the installer using RunAs:<service account> it worked fine.

Interestingly enough, the database was upgraded to version 11 during the first setup that failed and rolled back yesterday. So it is just something the installer is checking when it does its installation finalization that appears to not follow the account settings laid out in the wizard.

So to sum up: if you run VeeamONE under a service account and the database has permissions limited to the service account, you need to run the installer AS the service account and not as another administrator of the server in question.

I would assume that the product team will fix that at some point (or at least update the documentation to reflect this behavior).
Steve Krause
Veeam Certified Architect

wishr
Veeam Software
Posts: 2290
Liked: 290 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by wishr »

Hi Steve,

Thank you for the feedback. Could you please also share your support case ID so we could track the issue?

skrause
Expert
Posts: 463
Liked: 98 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by skrause »

I put it in the original post :D

They re-opened the original case and merged the two: Case#04675657
Steve Krause
Veeam Certified Architect

wishr
Veeam Software
Posts: 2290
Liked: 290 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by wishr »

Yes, sorry about that!

Btw, we are looking into that issue.

wishr
Veeam Software
Posts: 2290
Liked: 290 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by wishr »

Hi Steve,

Just a quick update: we've identified an issue in the upgrade logic and will fix it in the next version. Currently, the workaround is to run the setup on behalf of the service account, as you suggested above.

Thanks for bringing up this topic!

Thanks

skrause
Expert
Posts: 463
Liked: 98 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by skrause »

Thanks for the update.

Depending on how long it takes to post the new installer version for download it might not hurt to add this to the known issues list on the docs?
Steve Krause
Veeam Certified Architect

wishr
Veeam Software
Posts: 2290
Liked: 290 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by wishr »

Sure, we'll add it to the release notes doc, known issues, upgrade section.

Update: the updated release notes document is live.

spiritie
Expert
Posts: 125
Liked: 14 times
Joined: Mar 01, 2016 10:16 am
Full Name: Gert van Niekerk
Location: Denmark
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by spiritie »

Got the same error. Only way to fix it was to log into Windows as the Service account which Veeam ONE runs under.

wishr
Veeam Software
Posts: 2290
Liked: 290 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by wishr »

Hi Gert,

Have you tried launching the setup on behalf of the service account while being logged in to the OS normally?

Thanks

spiritie
Expert
Posts: 125
Liked: 14 times
Joined: Mar 01, 2016 10:16 am
Full Name: Gert van Niekerk
Location: Denmark
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by spiritie »

Hi wishr

I tried this to start with but it also didn't work. The only thing that worked was to log into the user it self in Windows.

wishr
Veeam Software
Posts: 2290
Liked: 290 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by wishr »

This is indeed strange. Passed this info to our QA team for analysis.

Thanks

skrause
Expert
Posts: 463
Liked: 98 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by skrause »

You know, I may have been logged in as the service account to windows when it worked and not using "Run As a Different User" on the installer.

I know I was logged in as that user at some point but I thought it was just to test the "VeeamONESettings.exe" launch.
Steve Krause
Veeam Certified Architect

wishr
Veeam Software
Posts: 2290
Liked: 290 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Upgrade to v11 failed and rollback uninstalled Monitor

Post by wishr »

Thanks, gentlemen, we'll test this out one more time!

Update: we've performed a few tests again and can confidently say that the aforementioned issue does not happen if you run the setup on behalf of the service account. There is no need to be logged in to the Veeam ONE server using the service account. For future readers: if it does not work for you - please open a technical support case and share your case ID for reference.

Regards,
Fedor

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests