9.5 Update 4 The worst Update Ever!!!!

[jhouse]

On Friday of the last week went to update to 9.5 Update 4 because in the forum and a call to Veeam support everyone said it was ok to do. Well, it's not, after 30 hours of work all Day and Night on Friday and several hours on Sat and Sun and also having to rebuild a VM for the main Veeam Server. After completely rebuilding the main Veeam backup server and getting it back online and hearing from several Veeam people on how bad this update is I am here to say DON'T update to 9.5 Update 4 unless to have a lot of time to spend. Using Veeam for about 5 years this is the worst Experience I have ever had with a Veeam Update. For a Company who prides itself on getting this right, this one is a bust and more test should have been done. If your support department is having a lot of problems with an update be honest with your users and don't say in forums and in your Support Dept that everything is OK. Come on Veeam you are better than this. Support Number 03418521 This was a joke and a lot of work this weekend.

[Gostev]

Hello, thanks for being an early U4 adopter, and very sorry for your experience!

It sounds like you may have had some bad luck - because according to our support management, based on factual metrics Update 4 has been a better than average release in terms of quality and stability. This was something we discussed just last Friday with all stakeholders, as we were doing the release planning for Update 4a (hotfixes rollup release), which we in R&D have the flexibility to accelerate as needed - and we always have support management making this call (whether to accelerate or not).

It's absolutely true that support department is heavily loaded, just like with every major release. However, according to their statistics, most load comes from support cases with questions about new functionality, so the support management does not believe that releasing Update 4a will have much impact on the support load. Again, this is information directly from support leadership, based on factual metrics (since just about everything in our support is measured).

If your support department is having a lot of problems with an update be honest with your users and don't say in forums and in your Support Dept that everything is OK.

Can you please provide a link to the specific forum post were Veeam staff was being dishonest about Update 4?

Thanks!

[jhouse]

Thank you for replying check with your support department they do agree with me that this is the worst Veeam upgrade and are having a lot of problems Please Check with your support people one on one and you will hear a different story.

Thanks John House

[Gostev]

We will make certain to talk to the support engineer who was handling your case. However, making conclusion based on individual remarks would be a classic "survival bias" error. The fact that some particular support folks may have had a few tough cases in a row does mean anything yet. The release quality is measured on the global level and based on factual metrics, such as the number of major bugs and the number of times hot fixes for those bugs have been requested and downloaded.

[jhouse]

This was from several support people working on this case and people within Veeam!

[Gostev]

My only point is that these people have no visibility on factual Update 4 metrics, and make comments based on perception and their own "tunnel vision". This is something we need to address internally, so thanks a lot for sharing this feedback.

And John, just to be clear - no one ever said Update 4 has no bugs. Contrary to that, I personally created the Top Issues Tracker sticky topic on the main forums just a few days within Update 4 release so, we've been very open and transparent about all the common support issues - as we always are.

The only thing I am arguing is your assessment of Update 4 being "worst update ever". This is simply not so based on the factual metrics. Update 4 is better than average update based on the number of major bugs, and is THE best update in terms of "number of quality issues to number of new features" ratio.

I do understand that none of these facts help YOU personally, because you personally has had a bad experience and lost your time dealing with issues - this is the only thing that matters to you, and I sincerely apologize for this. To avoid such experience in the future, I can only recommend that you don't install new major releases of any software (Veeam or not) until the first hotfixes rollup is released. This is generally the best practice for many of our enterprise customers, but also my own. For example, I lost count of how many issues I avoided by postponing the installation of those major Windows 10 update for half-year, comparing to many of my own colleagues

[Gostev]

Update from support management after they had a chance to review your ticket.

It appears that you have worked with quite a few support engineers over the course of this support case, but among all of them only ONE has been with the company long enough to actually be able to make comparisons of Update 4 to any of the previous major Veeam Backup & Replication releases. Which only proves my point that any comments you may have heard were totally baseless, with Update 4 being the first ever major VBR release for those support engineers since they joined Veeam. Funny, right?

Support management is definitely taking an action on this, like reviewing all support calls recordings to determine how appropriate those comments were - so thanks again for taking a time to bring this to our attention. This is exactly why I love these forums!

[jhouse]

Gostev,
All I can say is when I ask support people if I should update and see the number of downloads and very few problems found and trusting Veeam and their updates for the last 5 years I need to be honest with you and other Veeam professionals that to be careful if they don't want to run into the same problems I had. This was not the Veeam I know and Trust, with our company backups and that is what I care about. I know that you are taking the company line and protecting Veeam but I have also seen other backup companies in the 30 Years I have been doing business with that protecting the company at all cost even when their support teams are having problems is not what I expect with Veeam you must be better than this, I do care about Veeam.

[Gostev]

Well, I can only hope I'll once have a chance to provide you the solid proof that I am very much NOT like all those other people from other backup companies that you dealt with for the past 30 years. My priority at Veeam has always been protecting my products' users, as opposed to blindly protecting the company. Not that I don't care about the company - I just know that nothing will protect it better for the long term, then putting users above everything else.

[ober72]

Hi Folks,

We have done multiple updates to U4, no major issues at all. While I can understand the frustration of loosing a weekend to work this post has no concrete details and just states "don't upgrade". If I am to understand this issue happened on a single vbr server? Again we have done triple that number with no issues.

Anyways, if you could provide some details that would be great just in case their were some unique issues that other might encounter with similar environments.

Also I hope all works out in the end. I have had weekends wiped out by IT work many times so I know the feeling well.

cheers

[NightBird]

On my side, upgraded 4 customers for the moment, and 2 upgrades issues. No functionalities issues, but upgrade issues only.

First time since I started with Veeam B&R 2.0 that I have encounter upgrades issues.

First upgrade issue, I have to reinstall enterly server OS and restore Veeam configuration (Veeam console always said that one component was not upgraded, but it was...) => support engineer tried different things but no way... had to reinstall completly the OS (configuration restore was not working on the old OS).

2nd upgrade issue, upgrade OK but Veeam Backup Server service do not start anymore (socket issue port 9392 already in use) => reinstalled fresh VM with Veeam U4 OK, restored customer veeam configuration but at configuration restore, Veeam services are stopped, same issue than at the initial upgrade (socket issue port 9392 already in use), checked with support engineer with TCPview (sysinternal tools) port 9392 used but by nobody.. (bind to no process...) killed 9392 connections, connections come back again and again to 9392... no way... we had to stop anothers veeam services that were started at configuration restore, killed 9292 connections, Veeam backup server service then could start.

But U4 it's still one of the most beautiful version for me

[jhouse]

When we upgraded with Veeams help after upgrade worked with level 1 and he found in the control panel we showed 2 Veeam backup transport the old one and the new one. Could not get the old one to remove, then every time we started Veeam B/R update 4 it asked to update the server. They could not get this fixed and then was told we had to rebuild the server new which did take some time.

[NightBird]

My First Upgrade Issue, same problem but on vPower NFS... uninstalling Veeam, reinstalling Veeam with configuration restore... no way the problem persist... ==> finished with fresh OS installation. This one was a "all-in-one" physical server box so no VM snapshot to revert back or so... and no Veeam Agent backup of the server :'(

[Frosty]

I had some nerves jangling, but I went ahead and upgraded my all-in-one physical backup server yesterday. No problems encountered. I have plenty of sympathy for the original poster; losing a weekend is NOT fun and leaves a bad taste in the mouth. But also worth noting what Gostev posted, so I will highlight it again:

"... I can only recommend that you don't install new major releases of any software (Veeam or not) until the first hotfixes rollup is released."

I chose not to do that this time, but normally I would wait for hotfixes and several further weeks of positive reports before upgrading. On this occasion though, I had to pull the trigger on installing U4 because other upgrades are backed up in the pipeline behind it (vCenter 6.7, vSphere 6.7, impacting my DR strategy, and so on) and I just couldn't wait any longer.

[smoore33]

Just from reading the number of new features added and the licensing model changes, I decided long ago to treat this like a dot zero major release. I am disappointed with the approach of merging update releases with so much new functionality. I understand some of the issues that motivate Veeam to do so, but I would prefer that version numbers provide expected reliability information and we are almost never early adopters. So I have become more cautious about Veeam, in general from this approach. Let's hope it isn't a trend.

[xudaiqing]

My understand is they have changed release model long ago and every update number change is a major release, the second character change is the acture patch update.

I have 2 instance upgrades to U4 and no issue encoutered.

[albertwt]

JHouse,

What type of backup is having the issue with Veeam?

I also had to do the rollback of Veeam Backup to 9.5 Update 3 since the Windows Agent backup for 2x VMware SQL 2017 server node is not working while the license is running using Veeam Backup & Replication 9.5.4.2615 (Update 4).

[jensenjk]

Hi,

also had a lot of problems - had to rebuild the veeam server completely as update process only did it partially and thus services would not start afterwards.

[StephanG]

Just want to say that my update just worked like the ones before.
Deactivating all Veeam Services. Restart. Patched. Setting all to automatic. Restart. Worked.

But i have a small environment with only Hyper-V and about 20TB. So it is not complex in any way.

[MOBO]

i upgraded our setup last friday , no problems at all
rebooted the server , upgraded , rebooted again and then upgraded all the component

[AndresP]

Upgraded eintire hierarhy (1 Manager, 4 servers, 4 proxies/repositories) within 4-6 hours, two weeks ago no issues then and now. Like a charm.
Can the servers overal healt be the reason for failing upgrade? Upgrade check several prerequisites but still.

[actiondomi]

I've upgraded two of my servers to Update 4 just last week.

No issues whatsoever...but we all know that software is a fickle mistress, so you just never know

[ejenner]

Our upgrade was also totally smooth. x2 B&R backup management servers, x3 repository servers, x3 proxy servers. A mixture of 2016, 2012 R2 and 2008 R2. We have Linux agents, Windows agents, VMware and Hyper-V backups. I have to say though, I suspect my installations are on clean dedicated servers which aren't being used for anything except Veeam. Which may help.

[mcz]

I think we all can understand John's frustration and many of us had similar situations in the past (me included). For me it also took a while to install update 4 but I don't want to blame veeam for that. In our case, something went wrong with our SQL database installation and then the upgrade to SQL Server 2016 failed over and over again. It took me nearly a whole day to sort these issues out (I didn't open a support case) but finally I found a way to get it running.

Here is my personal approach to mitigate installation/upgrade issues:

• Download image/installation file and save it on veeam server

• take a backup of the veeam server or do a replication

• restore the veeam server in a testlab (be careful with duplicated bios.uuid, this can cause some hotadd-troubles)

• start the upgrade process in the isolated machine

Of course you cannot verify if e.g. upgrade of all proxies would work but you can at least check if the whole upgrade process runs smoothly and if you can still log in, etc. You also have an indication about the estimated downtime during upgrade.

If it doesn't work in your testlab, then it won't work on your production machine and you can test and try to solve the issues (without any stress and pressure, because your production's backup server is still running). Hope this might help somebody in the future.

[MihaiC.]

Hello,

I've upgraded 11 Veeam B&R instances, 3 more pending for this week (we need to migrate to vCenter 6.7 U1 this week, so waiting for U4a was no option for us) and I had only 1 upgrade issue with Veeam backup transport service so far.

During the first console opening, after the upgrade, you are prompted to upgrade required parts of the infrastructure.
Infrastructure upgrade ended successfully, however server, repository and proxy were still being shown as out of date.
After 1-2 hours of multiple "successful" infrastructure upgrades + server reboots, the final solution that worked was to use VeeamTransport.exe from the installation ISO to uninstall completely and then (re)install the service.

I've opened a severity 4 case (#03427821), for support team to check the logs.
I received no updates so far, therefor I can't give you technical details on what happened.
I've also done the upgrade from 9.0 to 9.5 and I had 0 issues on ~ 9 Veeam B&R instances.
Maybe I do not have enough past experience with Veeam upgrades, to be allowed to say that this is hardly a bad upgrade experience, even if I had one issue this time.
However I can tell you that we also use Veritas backup solutions and for them we schedule for Veritas support to be available at every upgrade and in a few cases they were really required to fix the upgrade.
Compare that upgrade plan with Veeam upgrades...

[chrisit]

We have updated our servers to Update 4 last week. All is well, but we had to run U4 update twice. The first time it hasn't updated Backup&Replication console, even though it has updated all other components including database.

[spiritie]

On Friday

How to set your self up for extra work on the weekend

Upgraded over 10 installations in a mix between RTM and GA to 9.5 update 4. 0 issues during the upgrade process.

Many of our VBR servers are very old I can remember one that is back from v6 or v7 and
has been upgraded with almost every update released and never had any upgrade issues.
I'm surprised this server has "endured" som many updates during the time and still performs perfectly.

[Simon_LBC]

I have updated my entire business setup to v9.5 Update 4 without much issues. In fact the only issue is that during update process SQL wants to update using built-in SYSTEM account, which is apparently wrong since I had "access denied" and so I had to enter manually local administrator account password. Then after this slight flaw, not a single more issue and I am running it for several days now.

[stmux]

It took more than 8 hours for me to update. Well, it wasn't even an update. After wasting hours trying to figure out why the update kept failing on a SQL error I had to uninstall 1536, SQL and vCenter Server and completely reinstall all of them to get update 4 working. Crazy.
mux

[pshute]

I'd just like to say that the transparency about support issues shown in this thread gives me more confidence about Veeam, not less. Not many products would let you see any hint of complaints and internal issues seen early in this thread.