Auto Upgrade to V2 with no warning

[donkeymagic]

Without any notification to customers Veeam made the V2 repo's live yesterday for VBFS. Our automated yum updates patched the system to V2 without any warning causing no end of disruptions and failed backups. The auto upgrade itself didn't work as expected with a schema in the database not being upgraded as it should have been resulting in the system being offline for over 7 hour before tech support could resolve the issue. The icing on the cake was they were not even aware that V2 was being released yesterday and nor were we. This is completely unacceptable to push these updates out without any notification to customers.

Because the upgrade process took a long time to patch the database last nights backups were also missed impacting yet more operations. How have Veeam allowed this to be the method of release? It's completely unprofessional and not inline with how Veeam usually operates. Either I've missed an email alert explaining that the update would be live from the 30th or nothing has been sent out, equally yesterday all the guides online were still for V1 with V2 only just being available today. It's not good enough at all and I would like an explanation as to why this approach was taken with this system. I'd like to add that the repo's were live at 0900 BST on Wednesday the 30th before any of the online documentation was updated including this forum.

[MIvanov]

Hello Oliver, thanks for sharing. I'm terribly sorry for the bad experience and downtime it caused.

As far as I'm aware, Veeam Updater does not automatically update the installation without the user consent. It alerts that update is available.
Moreover, it is not always safe to install in-place updates without prior backup of components. I don't think it is a great idea to update production machines blindly.
It's always good to test an update first and then feel safe apply the update.

We typically email our customers that it is safe to upgrade only 2-3 weeks after the GA, not prior to GA. We intend to release updates much more often than once in 7 months. How do you see the notification mechanism? We can consider setting version lock, however, this will require work on Veeam Updater component.

[donkeymagic]

Well in my experience VBFS has updated along with the regular yum updates and it certainly was not initiated from the Veeam GUI which is how I would expect Veeam to be upgraded and goes inline with all of your other products when we have to enable the upgrade by either downloading the ISO and manually patching or for the appliances pushing some of the upgrades out from VBR. This was not selected yesterday just routing system upgrades as part of yum. We do not blindly install updates on our production servers and I resent your accusation that that is the practice that we do, are you also saying that automatic windows updates rolled out via WSUS is not in best practice and shouldn't be done? Equally the fact that your repo's and updates for VBFS V2 have been enabled before any official announcement is made to say it has been released or the updated documentation is available online who is blindly providing updates now?

Ultimately we never selected for Veeam to be updated and the fact that the email notifications came out today to say that it was live when actually it was live yesterday is where my problem arises. No date was specified in previous correspondence from Veeam purely stating that VBFS V2 was due in the near future, not having the notification emails stating that it was available or having the relevant documentation available online prior to the repos going live is bad practice in my eyes.

Generally we wait several weeks before upgrading but in this instance I have not been given the choice. Equally do you suppose that we hold off on all updates provided via yum seen as your software has auto upgraded using that method for several weeks potentially making the system vulnerable to other issues?

I definitely think the solution should have a version lock so it has to be initiated via the gui only and controlled the same way that all of your other software works. No other component for Veeam upgrades using yum or apt or windows update and must be initiated from within the software itself or externally.

[donkeymagic]

Just to further add there is a Veeam repo that is setup on the server that has auto upgraded, when looking through the packages available on that repo it lists that it's where VBFS gets its updates from. I have not set this repo up or added it so I can only assume it must have been created as part of the initial installation script when VBFS was installed at V1 in December last year. As the yum repo is enabled running yum update has updated all the packages for Veeam as well.

I know that disabling this repo will stop it working via yum but will it also stop the GUI check for updates from working as well?

Repo-id : veeam-repo/x86_64
Repo-name : veeam-repo
Repo-revision: 1693412074
Repo-updated : Wed Aug 30 17:14:35 2023
Repo-pkgs : 4
Repo-size : 623 M
Repo-baseurl : https://repository.veeam.com/yum/el/7/x86_64/
Repo-expire : 21,600 second(s) (last: Thu Aug 31 10:00:30 2023)
Filter : read-only:present
Repo-filename: /etc/yum.repos.d/veeam-repo.repo

repolist: 34,817
[****** yum.repos.d]# yum repo-pkgs veeam-repo list
Loaded plugins: changelog, fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.vinters.com
* epel: cdn.centos.no
* extras: mirrors.coreix.net
* updates: mirrors.coreix.net
Installed Packages
vbsf.x86_64 2.0.0-3955.el7 @veeam-repo
veeam-updater.x86_64 7.0.0.810-el7 @veeam-repo
[****** yum.repos.d]#

[MIvanov]

Oliver, I do accept your comment about making the update available before the documentation was not a good sequence of events.

Yes, this repo is part of the installation procedure, and this is how normally the product gets updated. If I'm not mistaken, Veeam Updater relies on the repository as well, so deleting repository would stop any further updates completely, even through the UI. You will have to manually do all updates via the console by downloading the rpm.

I'm not sure about disabling the repo, I will check with the team.

[MIvanov]

@donkeymagic
I have checked with the team that you can disable the repo and this should not break the UI updates check.

[donkeymagic]

Great stuff.

Is Veeam going to update the documentation to show that the repo exists and is setup as part of the initial script and it's a method of upgrading so other people are aware of this feature in the future? All the upgrade notes state in the documentation is that it's initiated via the GUI but obviously it can also be initiated from yum as well so that needs to be included.

[MIvanov]

Good idea, will do that

[MIvanov]

@donkeymagic just to follow up on this, 2.1 update has been delivered with prior notification and release notes publishing. Where you able to review everything this time before the update has been pushed out?

[donkeymagic]

We disabled our automatic yum updates and manually patch the system now which is tedious to ensure that this didn't happen again. We have seen that the update is available but we are in change freeze at the moment which won't lift until January. At that point we will enable updates again and look at patching the system.

It does seem that notification is better this time but I will have to take your word that the notifications came out prior to the repository going live.

[G_Mellor]

I'm going to be deploying VBSF shortly, so I was intrigued when I saw your post about this issue. I've been looking around and it appears that the existing vbsf-install-script.sh installs version 2.0 and then uses "yum versionlock" or "dnf versionlock" as appropriate to prevent any veeam or vbsf package from being upgraded automatically:

Code: Select all

#install vbsf app
  if [[ "$version" == "7" ]]
  then
    yum -y update
    yum -y install vbsf-2.0.0
    #lock package version
    yum versionlock veeam*
    yum versionlock vbsf*
  elif [[ "$version" == "8" ]]
  then
    dnf -y install vbsf-2.0.0
    #lock package version
    dnf versionlock veeam*
    dnf versionlock vbsf*
  fi
fi

If this was missing from the v1.0 installation script, then that would explain the behavior you experienced with it auto upgrading with everything else. I've no idea if the upgrade from v1.0 to 2.0 would have run these versionlock commands. You could check if they're currently in effect for you, and if they're not then run the versionlock commands yourself which should allow you to go back to your previously automated yum updates.

[donkeymagic]

Hi @G_Mellor, that's a good spot in the new installation script, looking through the original one there is no lock in place and nothing to specify a specific version just the repo that should be used and to enable it. The V1 script is extremely basic compared to the V2 script so improvements have been made.

My frustration through this incident was being told that we had bad practices and it shouldn't auto upgrade as part of yum when it clearly did plus the poor management of when the repo went live vs the actual site/announcement. My understanding is that things have since changed regarding how the releases are handled and exactly as you have put the version lock in the latest installer certainly will prevent this. I've recently deployed a new environment running Ubuntu and everything is much more slick, I think our issues have come from our first deployment being the initial release of VBFS as at the time we could only use CentOS 7 which only had 18 months left on it at the time but Veeam hadn't tested on anything else and was the only platform we were allowed to use at the time.

[sumeet]

Hi all,
Last week our environment with VBSF v2.1 auto upgraded to v3. Luckily Postgres version was 13.1 (which was installed separately). We did not realize the upgrade is done, and our backups started giving warnings for a new permission, for which I opened a case and that is when we realized the version is 3. I have a case open with support 07434080, which was primarily for the permissions, but now I'm asking support to help me with steps on how to disable the auto update. Do not see anything in the doco. This was a fresh deployment with v2.1

I have other deployments too, where in I was able to mark in UI to remind me of the update after a week, but do not see an option to disable the auto update.

This is no good as we always want to upgrade after our checks and verifications.
Has anyone else seen such a behavior with v3 release?

Thanks,
-Sumeet.

[anastasia.belyaeva]

Hello, Sumeet.
An option to auto-update to v3 should be turned off in the Updater itself. But we will double check once you add the Updater logs to the support case you've submitted.

We have also sent "head-up" email notifications about upcoming product release to assigned license administrators with the list of features and steps required to do the upgrade. Please make sure, that your email address is set as Primary or Secondary license administrator, or Case administrator for the license. For more information, please see: https://www.veeam.com/kb2211

We'll add changes to the list of required permissions as per your request in the support case. Thanks for brining this to our attention.

Regards

[G_Mellor]

*** ROOT CAUSE FOUND ***

I've just installed Veeam for Salesforce v3, and checked afterwards and "dnf versionlock list" reports four postgresql packages locked, but no vbsf packages locked.

I've just gone through the scripts trying to work out where it's locking the vbsf package, and in the script vbsf-install-script.sh it's using the command:

Code: Select all

dnf versionlock vbsf*

Later on, in server-configuration.sh, it using the following commands to lock the PostgreSQL packages:

Code: Select all

dnf versionlock postgresql15*

Following a fresh install, I checked what packages are locked and shows the following:

Code: Select all

# dnf versionlock list
Updating Subscription Management repositories.
Last metadata expiration check: 0:00:22 ago on Tue 01 Oct 2024 01:45:08 PM ACST.
postgresql15-server-0:15.8-1PGDG.rhel8.*
postgresql15-contrib-0:15.8-1PGDG.rhel8.*
postgresql15-libs-0:15.8-1PGDG.rhel8.*
postgresql15-0:15.8-1PGDG.rhel8.*

I tried running the "dnf versionlock vbsf*" command manually:

Code: Select all

# dnf versionlock vbsf*
Updating Subscription Management repositories.
Last metadata expiration check: 0:02:49 ago on Tue 01 Oct 2024 01:45:08 PM ACST.
No package found for: vbsf-install-script.sh

The issue is that the shell evaluates "dnf versionlock vbsf*" and checks if the vbsf* wildcard matches any files in the current directory... which matches "vbsf-install-script.sh", which means that the actual command that's run is:

Code: Select all

# dnf versionlock vbsf-install-script.sh

Which of course doesn't work as intended, as there's no package called "vbsf-install-script.sh".

Solution:
The script vbsf-install-script.sh needs to be updated so there are quotes around the vbsf* wildcard so the shell doesn't try and expand the wildcard, and hence passes "vbsf*" as an option to dnf:

Code: Select all

dnf versionlock "vbsf*"

So anyone who used the offered curl command to download the script called vbsf-install-script.sh, and then runs it in the same directory will find that the vbsf package isn't being version locked, and so leaving the vbsf package to be automatically upgraded by dnf.

[anastasia.belyaeva]

Hello, @G_Mellor
We have updated installation script according to you suggestions.

Thank you

[sumeet]

Thanks @G_Mellor for the details.

We have not had much help from the support case we have. We have provided the required logs, but still no good.
Even though the root cause is explained above, the support keeps asking me for more logs (not sure what they mean by logs from var/log)

I'm hoping that support checks with engineering and get me a fix to make sure auto upgrade does not happen in the following scenarios

1. My VBSF env which is still at v2.1
2. The upgraded v3 env so that in future it does not auto upgrade

-Sumeet.

[G_Mellor]

@sumeet - you can check what packages are version locked with the command:

Code: Select all

dnf versionlock list

If that doesn't list any packages starting with vbsf, then you can manually lock the currently installed Veeam Backup for Salesforce package with the command:

Code: Select all

dnf versionlock "vbsf*"

and then run the first command again to confirm the currently installed vbsf package is now listed as locked.

I've tried looking at the upgrade package that is installed if you upgrade from Veeam Backup for Salesforce from v2.1 to 3.0, but I can't see any reference to versionlock so I can't tell what would happen. It may automatically version lock the v3.0 package, but I don't know.

If you lock the vbsf package using the manual commands above, then an "apt update" or "dnf update" will not accidentally upgrade VBfS. If you manually upgrade VBfS to v3, I would recommend running the "dnf versionlock list" command again to check what packages are now version locked - and if the vbsf package listed does not start "vbsf-3.0..." then run the second command above to lock the new v3 package.

[sumeet]

@G_Mellor, thanks for the details.

My server RHEL 8 did not have the versionlock plugin and I had to install this.

$ yum install python3-dnf-plugin-versionlock

And then ran the command to versionlock. Now the list command shows - vbsf-0:3.0.0-1769.el8.*

So for future this should not auto upgrade, unless we initiate the upgrade.