Discussions specific to the VMware vSphere hypervisor
jhouse
Influencer
Posts: 22
Liked: 4 times
Joined: Jul 19, 2017 1:13 pm
Full Name: John House
Contact:

9.5 Update 4 The worst Update Ever!!!!

Post by jhouse » Feb 18, 2019 1:21 pm 1 person likes this post

On Friday of the last week went to update to 9.5 Update 4 because in the forum and a call to Veeam support everyone said it was ok to do. Well, it's not, after 30 hours of work all Day and Night on Friday and several hours on Sat and Sun and also having to rebuild a VM for the main Veeam Server. After completely rebuilding the main Veeam backup server and getting it back online and hearing from several Veeam people on how bad this update is I am here to say DON'T update to 9.5 Update 4 unless to have a lot of time to spend. Using Veeam for about 5 years this is the worst Experience I have ever had with a Veeam Update. For a Company who prides itself on getting this right, this one is a bust and more test should have been done. If your support department is having a lot of problems with an update be honest with your users and don't say in forums and in your Support Dept that everything is OK. Come on Veeam you are better than this. Support Number 03418521 This was a joke and a lot of work this weekend. :(

Gostev
SVP, Product Management
Posts: 24818
Liked: 3575 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by Gostev » Feb 18, 2019 1:55 pm 1 person likes this post

Hello, thanks for being an early U4 adopter, and very sorry for your experience!

It sounds like you may have had some bad luck - because according to our support management, based on factual metrics Update 4 has been a better than average release in terms of quality and stability. This was something we discussed just last Friday with all stakeholders, as we were doing the release planning for Update 4a (hotfixes rollup release), which we in R&D have the flexibility to accelerate as needed - and we always have support management making this call (whether to accelerate or not).

It's absolutely true that support department is heavily loaded, just like with every major release. However, according to their statistics, most load comes from support cases with questions about new functionality, so the support management does not believe that releasing Update 4a will have much impact on the support load. Again, this is information directly from support leadership, based on factual metrics (since just about everything in our support is measured).
jhouse wrote:
Feb 18, 2019 1:21 pm
If your support department is having a lot of problems with an update be honest with your users and don't say in forums and in your Support Dept that everything is OK.
Can you please provide a link to the specific forum post were Veeam staff was being dishonest about Update 4?

Thanks!

jhouse
Influencer
Posts: 22
Liked: 4 times
Joined: Jul 19, 2017 1:13 pm
Full Name: John House
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by jhouse » Feb 18, 2019 2:00 pm

Thank you for replying check with your support department they do agree with me that this is the worst Veeam upgrade and are having a lot of problems Please Check with your support people one on one and you will hear a different story.

Thanks John House

Gostev
SVP, Product Management
Posts: 24818
Liked: 3575 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by Gostev » Feb 18, 2019 2:05 pm

We will make certain to talk to the support engineer who was handling your case. However, making conclusion based on individual remarks would be a classic "survival bias" error. The fact that some particular support folks may have had a few tough cases in a row does mean anything yet. The release quality is measured on the global level and based on factual metrics, such as the number of major bugs and the number of times hot fixes for those bugs have been requested and downloaded.

jhouse
Influencer
Posts: 22
Liked: 4 times
Joined: Jul 19, 2017 1:13 pm
Full Name: John House
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by jhouse » Feb 18, 2019 2:14 pm

This was from several support people working on this case and people within Veeam!

Gostev
SVP, Product Management
Posts: 24818
Liked: 3575 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by Gostev » Feb 18, 2019 2:41 pm 1 person likes this post

My only point is that these people have no visibility on factual Update 4 metrics, and make comments based on perception and their own "tunnel vision". This is something we need to address internally, so thanks a lot for sharing this feedback.

And John, just to be clear - no one ever said Update 4 has no bugs. Contrary to that, I personally created the Top Issues Tracker sticky topic on the main forums just a few days within Update 4 release :) so, we've been very open and transparent about all the common support issues - as we always are.

The only thing I am arguing is your assessment of Update 4 being "worst update ever". This is simply not so based on the factual metrics. Update 4 is better than average update based on the number of major bugs, and is THE best update in terms of "number of quality issues to number of new features" ratio.

I do understand that none of these facts help YOU personally, because you personally has had a bad experience and lost your time dealing with issues - this is the only thing that matters to you, and I sincerely apologize for this. To avoid such experience in the future, I can only recommend that you don't install new major releases of any software (Veeam or not) until the first hotfixes rollup is released. This is generally the best practice for many of our enterprise customers, but also my own. For example, I lost count of how many issues I avoided by postponing the installation of those major Windows 10 update for half-year, comparing to many of my own colleagues :D

Gostev
SVP, Product Management
Posts: 24818
Liked: 3575 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by Gostev » Feb 18, 2019 2:50 pm 1 person likes this post

Update from support management after they had a chance to review your ticket.

It appears that you have worked with quite a few support engineers over the course of this support case, but among all of them only ONE has been with the company long enough to actually be able to make comparisons of Update 4 to any of the previous major Veeam Backup & Replication releases. Which only proves my point that any comments you may have heard were totally baseless, with Update 4 being the first ever major VBR release for those support engineers since they joined Veeam. Funny, right?

Support management is definitely taking an action on this, like reviewing all support calls recordings to determine how appropriate those comments were - so thanks again for taking a time to bring this to our attention. This is exactly why I love these forums!

jhouse
Influencer
Posts: 22
Liked: 4 times
Joined: Jul 19, 2017 1:13 pm
Full Name: John House
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by jhouse » Feb 18, 2019 3:04 pm 1 person likes this post

Gostev,
All I can say is when I ask support people if I should update and see the number of downloads and very few problems found and trusting Veeam and their updates for the last 5 years I need to be honest with you and other Veeam professionals that to be careful if they don't want to run into the same problems I had. This was not the Veeam I know and Trust, with our company backups and that is what I care about. I know that you are taking the company line and protecting Veeam but I have also seen other backup companies in the 30 Years I have been doing business with that protecting the company at all cost even when their support teams are having problems is not what I expect with Veeam you must be better than this, I do care about Veeam.

Gostev
SVP, Product Management
Posts: 24818
Liked: 3575 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by Gostev » Feb 18, 2019 4:25 pm 1 person likes this post

Well, I can only hope I'll once have a chance to provide you the solid proof that I am very much NOT like all those other people from other backup companies that you dealt with for the past 30 years. My priority at Veeam has always been protecting my products' users, as opposed to blindly protecting the company. Not that I don't care about the company - I just know that nothing will protect it better for the long term, then putting users above everything else.

ober72
Service Provider
Posts: 523
Liked: 74 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Location: CANADA
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by ober72 » Feb 18, 2019 5:13 pm 3 people like this post

Hi Folks,

We have done multiple updates to U4, no major issues at all. While I can understand the frustration of loosing a weekend to work this post has no concrete details and just states "don't upgrade". If I am to understand this issue happened on a single vbr server? Again we have done triple that number with no issues.

Anyways, if you could provide some details that would be great just in case their were some unique issues that other might encounter with similar environments.

Also I hope all works out in the end. I have had weekends wiped out by IT work many times so I know the feeling well.

cheers
Geoff Burke
Veeam Certified Architect

NightBird
Service Provider
Posts: 178
Liked: 32 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by NightBird » Feb 18, 2019 6:17 pm 3 people like this post

On my side, upgraded 4 customers for the moment, and 2 upgrades issues. No functionalities issues, but upgrade issues only.

First time since I started with Veeam B&R 2.0 that I have encounter upgrades issues.

First upgrade issue, I have to reinstall enterly server OS and restore Veeam configuration (Veeam console always said that one component was not upgraded, but it was...) => support engineer tried different things but no way... had to reinstall completly the OS (configuration restore was not working on the old OS).

2nd upgrade issue, upgrade OK but Veeam Backup Server service do not start anymore (socket issue port 9392 already in use) => reinstalled fresh VM with Veeam U4 OK, restored customer veeam configuration but at configuration restore, Veeam services are stopped, same issue than at the initial upgrade (socket issue port 9392 already in use), checked with support engineer with TCPview (sysinternal tools) port 9392 used but by nobody.. (bind to no process...) killed 9392 connections, connections come back again and again to 9392... no way... we had to stop anothers veeam services that were started at configuration restore, killed 9292 connections, Veeam backup server service then could start.

But U4 it's still one of the most beautiful version for me :)

jhouse
Influencer
Posts: 22
Liked: 4 times
Joined: Jul 19, 2017 1:13 pm
Full Name: John House
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by jhouse » Feb 18, 2019 6:29 pm

When we upgraded with Veeams help after upgrade worked with level 1 and he found in the control panel we showed 2 Veeam backup transport the old one and the new one. Could not get the old one to remove, then every time we started Veeam B/R update 4 it asked to update the server. They could not get this fixed and then was told we had to rebuild the server new which did take some time.

NightBird
Service Provider
Posts: 178
Liked: 32 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by NightBird » Feb 18, 2019 6:39 pm

My First Upgrade Issue, same problem but on vPower NFS... uninstalling Veeam, reinstalling Veeam with configuration restore... no way the problem persist... ==> finished with fresh OS installation. This one was a "all-in-one" physical server box so no VM snapshot to revert back or so... and no Veeam Agent backup of the server :'(

Frosty
Expert
Posts: 175
Liked: 39 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by Frosty » Feb 18, 2019 10:14 pm 3 people like this post

I had some nerves jangling, but I went ahead and upgraded my all-in-one physical backup server yesterday. No problems encountered. I have plenty of sympathy for the original poster; losing a weekend is NOT fun and leaves a bad taste in the mouth. But also worth noting what Gostev posted, so I will highlight it again:

"... I can only recommend that you don't install new major releases of any software (Veeam or not) until the first hotfixes rollup is released."

I chose not to do that this time, but normally I would wait for hotfixes and several further weeks of positive reports before upgrading. On this occasion though, I had to pull the trigger on installing U4 because other upgrades are backed up in the pipeline behind it (vCenter 6.7, vSphere 6.7, impacting my DR strategy, and so on) and I just couldn't wait any longer.

smoore33
Influencer
Posts: 11
Liked: 4 times
Joined: Mar 14, 2016 7:31 pm
Full Name: Steve
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by smoore33 » Feb 25, 2019 1:56 am 2 people like this post

Just from reading the number of new features added and the licensing model changes, I decided long ago to treat this like a dot zero major release. I am disappointed with the approach of merging update releases with so much new functionality. I understand some of the issues that motivate Veeam to do so, but I would prefer that version numbers provide expected reliability information and we are almost never early adopters. So I have become more cautious about Veeam, in general from this approach. Let's hope it isn't a trend.

xudaiqing
Influencer
Posts: 11
Liked: 1 time
Joined: Apr 14, 2017 5:25 pm
Full Name: xudaiqing
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by xudaiqing » Feb 25, 2019 5:04 am

My understand is they have changed release model long ago and every update number change is a major release, the second character change is the acture patch update.

I have 2 instance upgrades to U4 and no issue encoutered.

albertwt
Expert
Posts: 640
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by albertwt » Feb 25, 2019 6:03 am

JHouse,

What type of backup is having the issue with Veeam?

I also had to do the rollback of Veeam Backup to 9.5 Update 3 since the Windows Agent backup for 2x VMware SQL 2017 server node is not working while the license is running using Veeam Backup & Replication 9.5.4.2615 (Update 4).
--
/* Veeam software enthusiast user & supporter ! */

jensenjk
Novice
Posts: 5
Liked: 1 time
Joined: Nov 17, 2014 11:43 am
Full Name: Jørgen Kaare Jensen
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by jensenjk » Feb 25, 2019 7:09 am 1 person likes this post

Hi,

also had a lot of problems - had to rebuild the veeam server completely as update process only did it partially and thus services would not start afterwards.

StephanG
Enthusiast
Posts: 34
Liked: 2 times
Joined: Sep 07, 2014 11:15 am
Full Name: Stephan G
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by StephanG » Feb 25, 2019 7:23 am 2 people like this post

Just want to say that my update just worked like the ones before.
Deactivating all Veeam Services. Restart. Patched. Setting all to automatic. Restart. Worked.

But i have a small environment with only Hyper-V and about 20TB. So it is not complex in any way.

MOBO
Influencer
Posts: 15
Liked: 5 times
Joined: Jan 24, 2015 7:26 am
Full Name: Morten Boegeskov
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by MOBO » Feb 25, 2019 7:39 am 2 people like this post

i upgraded our setup last friday , no problems at all
rebooted the server , upgraded , rebooted again and then upgraded all the component

AndresP
Lurker
Posts: 2
Liked: 1 time
Joined: Aug 12, 2014 7:20 am
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by AndresP » Feb 25, 2019 8:07 am 1 person likes this post

Upgraded eintire hierarhy (1 Manager, 4 servers, 4 proxies/repositories) within 4-6 hours, two weeks ago no issues then and now. Like a charm.
Can the servers overal healt be the reason for failing upgrade? Upgrade check several prerequisites but still.

actiondomi
Lurker
Posts: 1
Liked: 2 times
Joined: Dec 03, 2018 2:53 pm
Full Name: Dominik Wagner
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by actiondomi » Feb 25, 2019 8:17 am 2 people like this post

I've upgraded two of my servers to Update 4 just last week.

No issues whatsoever...but we all know that software is a fickle mistress, so you just never know :D

ejenner
Expert
Posts: 403
Liked: 60 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by ejenner » Feb 25, 2019 8:33 am 2 people like this post

Our upgrade was also totally smooth. x2 B&R backup management servers, x3 repository servers, x3 proxy servers. A mixture of 2016, 2012 R2 and 2008 R2. We have Linux agents, Windows agents, VMware and Hyper-V backups. I have to say though, I suspect my installations are on clean dedicated servers which aren't being used for anything except Veeam. Which may help.

mcz
Expert
Posts: 301
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by mcz » Feb 25, 2019 8:44 am 4 people like this post

I think we all can understand John's frustration and many of us had similar situations in the past (me included). For me it also took a while to install update 4 but I don't want to blame veeam for that. In our case, something went wrong with our SQL database installation and then the upgrade to SQL Server 2016 failed over and over again. It took me nearly a whole day to sort these issues out (I didn't open a support case) but finally I found a way to get it running.

Here is my personal approach to mitigate installation/upgrade issues:
  • Download image/installation file and save it on veeam server
  • take a backup of the veeam server or do a replication
  • restore the veeam server in a testlab (be careful with duplicated bios.uuid, this can cause some hotadd-troubles)
  • start the upgrade process in the isolated machine
Of course you cannot verify if e.g. upgrade of all proxies would work but you can at least check if the whole upgrade process runs smoothly and if you can still log in, etc. You also have an indication about the estimated downtime during upgrade.

If it doesn't work in your testlab, then it won't work on your production machine and you can test and try to solve the issues (without any stress and pressure, because your production's backup server is still running). Hope this might help somebody in the future. :)

MihaiC.
Lurker
Posts: 2
Liked: 1 time
Joined: Oct 19, 2018 6:40 am
Full Name: Mihai Cical
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by MihaiC. » Feb 25, 2019 11:11 am 1 person likes this post

Hello,

I've upgraded 11 Veeam B&R instances, 3 more pending for this week (we need to migrate to vCenter 6.7 U1 this week, so waiting for U4a was no option for us) and I had only 1 upgrade issue with Veeam backup transport service so far.

During the first console opening, after the upgrade, you are prompted to upgrade required parts of the infrastructure.
Infrastructure upgrade ended successfully, however server, repository and proxy were still being shown as out of date.
After 1-2 hours of multiple "successful" infrastructure upgrades + server reboots, the final solution that worked was to use VeeamTransport.exe from the installation ISO to uninstall completely and then (re)install the service.

I've opened a severity 4 case (#03427821), for support team to check the logs.
I received no updates so far, therefor I can't give you technical details on what happened.
I've also done the upgrade from 9.0 to 9.5 and I had 0 issues on ~ 9 Veeam B&R instances.
Maybe I do not have enough past experience with Veeam upgrades, to be allowed to say that this is hardly a bad upgrade experience, even if I had one issue this time.
However I can tell you that we also use Veritas backup solutions and for them we schedule for Veritas support to be available at every upgrade and in a few cases they were really required to fix the upgrade.
Compare that upgrade plan with Veeam upgrades...

chrisit
Novice
Posts: 8
Liked: 1 time
Joined: Aug 23, 2017 4:34 pm
Full Name: Chris
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by chrisit » Feb 25, 2019 1:17 pm 1 person likes this post

We have updated our servers to Update 4 last week. All is well, but we had to run U4 update twice. The first time it hasn't updated Backup&Replication console, even though it has updated all other components including database.

spiritie
Enthusiast
Posts: 76
Liked: 7 times
Joined: Mar 01, 2016 10:16 am
Full Name: Gert van Niekerk
Location: Denmark
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by spiritie » Feb 25, 2019 1:48 pm 1 person likes this post

jhouse wrote:
Feb 18, 2019 1:21 pm
On Friday
How to set your self up for extra work on the weekend :)

Upgraded over 10 installations in a mix between RTM and GA to 9.5 update 4. 0 issues during the upgrade process.

Many of our VBR servers are very old I can remember one that is back from v6 or v7 and
has been upgraded with almost every update released and never had any upgrade issues.
I'm surprised this server has "endured" som many updates during the time and still performs perfectly.

Simon_LBC
Enthusiast
Posts: 29
Liked: 3 times
Joined: Dec 11, 2018 3:15 pm
Full Name: Simon C.
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by Simon_LBC » Feb 25, 2019 1:50 pm 1 person likes this post

I have updated my entire business setup to v9.5 Update 4 without much issues. In fact the only issue is that during update process SQL wants to update using built-in SYSTEM account, which is apparently wrong since I had "access denied" and so I had to enter manually local administrator account password. Then after this slight flaw, not a single more issue and I am running it for several days now.

stmux
Enthusiast
Posts: 35
Liked: never
Joined: May 24, 2013 2:51 pm
Full Name: ja
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by stmux » Feb 25, 2019 4:07 pm

It took more than 8 hours for me to update. Well, it wasn't even an update. After wasting hours trying to figure out why the update kept failing on a SQL error I had to uninstall 1536, SQL and vCenter Server and completely reinstall all of them to get update 4 working. Crazy.
mux

pshute
Expert
Posts: 208
Liked: 12 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by pshute » Feb 25, 2019 8:56 pm 2 people like this post

I'd just like to say that the transparency about support issues shown in this thread gives me more confidence about Veeam, not less. Not many products would let you see any hint of complaints and internal issues seen early in this thread.

Post Reply

Who is online

Users browsing this forum: No registered users and 43 guests