Discussions specific to the VMware vSphere hypervisor
nitramd
Expert
Posts: 198
Liked: 46 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by nitramd » Feb 25, 2019 9:20 pm 1 person likes this post

@pshute

Agreed. I expect that this is one of the reasons Anton runs metrics, for better transparency.

With that said, it's also great to see end users post their upgrade results which seem to match the metrics.

mkaec
Expert
Posts: 323
Liked: 72 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by mkaec » Feb 25, 2019 9:51 pm 2 people like this post

I regularly giver other vendors (non-competitors to Veeam) links to these forums as examples of how to properly setup a successful forum.

rossj_nz
Novice
Posts: 3
Liked: 2 times
Joined: Feb 25, 2019 7:42 am
Full Name: Ross Johnson
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by rossj_nz » Feb 26, 2019 2:51 am 1 person likes this post

I am a Service Provider with a number of years experience with Veeam B&R. I would class myself as a Veeam supporter and user and have had a number of issues caused by Update 4. I think that Veeam have this wrong where their versioning is misleading clients into "update mentality" being sort of minor not major. Typically an update contains bug fixes and maybe some enhancements. Updates should not be new major functionality which looking at Update 4 it is full of. For my organisation the main reason for implementing Update 4 is the support of vCloud Director 9.5. Update 4 should have been version 9.6

Update 4 was further complicated by the change in licensing from Update 3. To the extent that for a Service Provider the upgrade confirms your existing license is valid and promptly expires it with 90 days grace period. Flawed logic at play here. In my case i then found that the Veeam Portal has some error or corruption that did not provide a valid Update 4 Service Provider license. This then links to the ability to log a support call via the Veeam Portal and meaning the only way to log a support call is over the phone. So not a great experience.

The issues I encountered so far are: Issue 1: Veeam Console Update 4 crashing on opening due to a bug in the time am/pm upper or lower case setting. https://www.veeam.com/kb2884 Issue 2: Veeam Guest Catalog Service will not start after Update 4 upgrade from Update 3. Will log a support call for this. Issue 3: A backup job that has been working fine for months now crashes once a week with the error "Unable to allocate memory for storage metadata bank. Failed to open storage for read access". Will log a support call for this.

So unless you need Update 4, ignore the number of Update 4 downloads being quoted as this is near meaningless. It is the number of update 4 licences that have been activated that is the important number, my advice is to wait. If you are a Service Provider then check that Update 4 licenses appear before you embark on the Update.

mcz
Expert
Posts: 307
Liked: 55 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by mcz » Feb 26, 2019 7:51 am 2 people like this post

...correct and we also see that there are more cases where the update worked without any issues. So it doesn't look like that it's the worst update ever (which I personally never doubted).

Gostev
SVP, Product Management
Posts: 24940
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by Gostev » Feb 26, 2019 8:15 pm 10 people like this post

All, thanks for your passionate feedback! We're blessed to have users who don't only come to forums to post nasty comments or [rightfully] complain about technical issues, but also those who take their time to post good things when things DO work. This is extremely rare, and I am very thankful to you all for that.

One thing that will always be special about Veeam is the sheer size of our customer base. With over 330K paying users, and over 1M of free edition users, there will always be customers running into various environment-specific problems after every major release. And the reason is obvious: our test labs can never cover the incredible variety of real-world environments.

Ross provided the perfect example of one issue that impacted upgrade for our customers from a few countries because of their very special regional settings format. Can we possibly blame our QC for not having an upgrade testing lab for every country in the world? Probably not. Will this bug impact at least 0.1% of all Veeam customers (which means hundreds), making them all upset and thinking "this is the worst release ever"? No doubt.

Our QC is obsessed with quality, which is why we never announce release dates. We just know that our QC leadership will simply not sign off the build until every QC team lead is comfortable. This drives me personally crazy, because as the PM I am always teared apart by everyone to ship the new release ASAP. But, when I take an unbiased view, I do recognize that in the past 10 years they've mastered finding that fine line when further in-house testing is largely pointless, and it's time to ship the product to the masses. I see the proof to that in every release, seeing how most bugs reported by customers would be impossible to find in our own labs in the reasonable time frame.

So, while we do ship the product at the right time, when QC feels it's ready - issues are always expected following the major release for the reasons I mentioned earlier. However, customer support is the highest priority for Veeam R&D, and they work really hard to provide hot fixes in a matter of days. This is one compliment that I keep hearing from end users: Veeam is by far the fastest vendor to provide hot fixes among all Enterprise software vendors they're working with. So you can be sure that even if circumstances force you to be an early adopted of a major release, we will go out of our way to make it work for you.

As far as the overall release quality trend, this is something you have to trust me with. Because not only I saw every B&R release since v2, but I also have access to all support statistics, such as:
• Number of support cases for the new release relative to the total number of customers
• Number of bugs relative to the number of new features introduced
• Number of major and critical bugs in the new release

Long story short, the trend has been positive since version 6.0, so the release quality is constantly improving. It may be hard to see this, because with 4000 new customers added every month, the absolute number of customers experiencing issues with every major release growth faster than the actual release quality. Although ironically, I can tell you that Update 4 is the first ever release for which this usual topic about "this release being the worst ever" took over 1 month and 90K downloads since GA to appear, instead of the usual week or two. While unlike support metrics, this one is not scientific at all - this tells me personally a lot.

So really, business is usual - and I can only promise you two things:
1. Veeam R&D will do even better job with the next release in regards to its quality.
2. But regardless of that, I will see you all in the same exact topic about the next release 1 year from now :D

Some things just never change!

mcz
Expert
Posts: 307
Liked: 55 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by mcz » Feb 27, 2019 8:25 am 9 people like this post

I can only confirm what Anton has mentioned: I had two support cases in the past where we had issues after upgrades and needed a hotfix. One caused SureBackup to fail and the other prevented indexing from being successful. It took quite a while to get the root cause of the issues (T3 and T4 support was needed) but I tell you these guys are incredible!! Within 2 days I got the hotfix for an issue that has never been seen before and after deployment it just worked as a charme.

Also I would like to mention that the veeam support is for sure the best support in the world. They understand you, they listen to you, they are ALWAYS friendly, they are very competent, have sense of humor (which helps a lot when IT is stressing you) and they have a very good response time. They even help you if the issue is caused by 3rd party components like OS. As long as they can help you, they will. You never hear something like "sorry but please ask your admin to change this or that, this is not part of our product". Never. Even in days of upgrade periods you get a response within a day. If you have a case where you feel that you might need the next tier because a deeper investigation is needed, you can manually transfer the ticket to the next level in the portal - did you know that (thank you Anton for this nice hint some month ago)? What I do appreciate a lot is the knowledge support engineers are sharing with you. They don't tell you "I don't know" or "I cannot talk about that", no, they explain you very well how the components are working. This helps a lot for understanding the problem and solving future cases on your own. I can remember I once talked to a T4 engineer who was also involved in development and he told me exactly how one specific component was working. It is still incredible to me when I think about it...

Compared to other bit IT companies veeam's support is just supreme. I don't wannt to post a name here but I'm working on tickets with a big OS vendor from time to time and the first problem is always communication: The other person hears you but they don't understand your point - very cumbersome. Response time? No exception that you are waiting 2-3 days for a response regarding a simple question / configuration. You send messages, saying it's urgent. Slow response. Sometimes it's better to do a trial & error by your own instead of waiting. If you call a veeam engineer and he isn't available, he will contact you within some hours letting you know that he has been informed that you have called - this is so great.

Even if your upgrade fails or is causing big issues you can be sure that support will do everything the can do to help you. This is my veeam-experience within the last years. :D

rboynton
Enthusiast
Posts: 57
Liked: 7 times
Joined: Jun 25, 2015 12:59 am
Full Name: Rick Boynton
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by rboynton » Feb 27, 2019 8:41 pm 3 people like this post

In our case, we had an issue where the enterprise installation bombed, so had to get help from support. After he determined what was going on, the rest of the upgrade worked fine. We actually reloaded our Veeam B&R (physical) server yesterday from 2008r2 to 2012r2 so we could use a new Tape Autoloader (drive was not recognized by 2008r2, and no driver was available). After the upgrade (with installing the latest firmware on the server as well), the performance is better than before. We simply backed up the configuration, then restored it onto the new installation. Only thing we had to do was to rescan the repositories & re-enter all the saved credentials. Sure appreciated being able to restore the DB so easily. Saved a TON of hours trying to redo all the jobs!

rossj_nz
Novice
Posts: 3
Liked: 2 times
Joined: Feb 25, 2019 7:42 am
Full Name: Ross Johnson
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by rossj_nz » Mar 01, 2019 4:44 am 1 person likes this post

Its fair to say that Veeam gets these updates right almost all of the time. Its also great that Veeam allows comments that are factual and put in a reasonable way when things do go wrong. Veeam are the leading backup Vendor when it comes to support for products like vSphere and vCloud Director as these products are updated by VMware 2 to 3 times a year.

I would also say that the best support experience i have ever had with a Vendor was from Veeam with a severity 1 issue. Veeam support contacted me and starting working on the issue within 5 minutes of me logging the support ticket.

One thing that i would like to see taken on board is the definition of an update vs a major or point release and the signalling that this sends to their customers.

Zew
Expert
Posts: 214
Liked: 42 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by Zew » Mar 01, 2019 4:02 pm

I upgraded, was pretty smooth. Loved the new splash loading screen... mhmmm delightful.

The only issue I had, which there was already another Fourm post about, was a new failed job physical-cloud-f49/rescan-of-manually-a ... 56589.html "Rescan of Manually added failed"

DaveWatkins
Expert
Posts: 349
Liked: 93 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by DaveWatkins » Mar 04, 2019 2:15 am

@Gostev

Curious, does QA integrate tests to catch these date/time format bugs into future testing or is it considered too small of a problem to test for?

itrabbit
Influencer
Posts: 20
Liked: 6 times
Joined: Nov 24, 2016 6:50 am
Full Name: Matt Dunleavy
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by itrabbit » Mar 04, 2019 2:54 am

I won't be upgrading any time soon, I don't understand this new licensing and I have tried several times to understand it.

Our backups are solid and I don't want to go through the pain of failed upgrades or issues with licensing, so I will wait a bit more.

We have 6 Socket Essential Bundle (which is the max sockets you can have for this license) + 7 Windows Server agents. So I really don't understand whats going to change and I don't want to be paying more for something we already bought.

Andreas Neufert
Veeam Software
Posts: 3907
Liked: 706 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by Andreas Neufert » Mar 04, 2019 9:08 am

DaveWatkins wrote:
Mar 04, 2019 2:15 am
Curious, does QA integrate tests to catch these date/time format bugs into future testing or is it considered too small of a problem to test for?
When we spot issues like this, we add it to our test plan. It would not make sense if the next product has the exact same bug!

ITP-Stan
Service Provider
Posts: 99
Liked: 11 times
Joined: Feb 18, 2013 10:45 am
Full Name: Stan (IF-IT4U)
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by ITP-Stan » Mar 05, 2019 8:17 am

My only criticism is that the new licensing model is complex and it has not been talked about enough prior to the launch of U4.
It took me by surprise as I'm sure a lot others as well.

And I agree with others that Veeam should go back to a logical version numbering. This should have been 10.0

skrause
Expert
Posts: 443
Liked: 93 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by skrause » Mar 05, 2019 2:40 pm

I would say this was my "worst" experience with an update to Veeam for only one reason: The requirement for .NET 4.6 caught me by surprise and meant I needed to reboot my B&R and repository servers a couple of extra times for the .NET install and the subsequent .NET patches. Otherwise everything else went pretty smoothly.

I did have to call my SE to get a handle on the impact of the new licensing model, so the communication from Veeam on that could have been clearer.
Steve Krause
Veeam Certified Architect

idstephen
Lurker
Posts: 2
Liked: never
Joined: Dec 07, 2016 5:48 pm
Full Name: Ian Stephen
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by idstephen » Mar 08, 2019 3:45 pm

9.5 Update 4 was very smooth at my site! Thank you Veeam for making it so...

kqf_chris
Lurker
Posts: 2
Liked: 3 times
Joined: Apr 27, 2017 12:52 pm
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by kqf_chris » Mar 08, 2019 4:14 pm 3 people like this post

I upgraded to U4. The process itself went smoothly but I did get hit with a nasty tape bug where Update 4 would delete my full backups after adding the incremental(s). I contacted support and they promptly sent me a hotfix which took care of it.

However, the real issues for me were licensing-related and lack of training on the Veeam side. It took forever to get any info on the new per-instance licensing. It seems like it was tossed out there without much planning. None of my questions could be answered in a timely fashion.

Aside from all that, Veeam is the best backup product I have ever used, bar none.

C_Montero
Novice
Posts: 3
Liked: 2 times
Joined: Mar 26, 2019 7:04 pm
Full Name: Christian Montero
Contact:

Re: 9.5 Update 4 The worst Update Ever!!!!

Post by C_Montero » May 29, 2019 9:04 pm 1 person likes this post

I have updated 3 systems to U4, the upgrade process went well on all 3 of them, 2 of them are VMware and 1 of them is Hyper-V.

The only issue I had was that one of the deployments with Hyper-V had an issue with some virtual machines and VSS, the jobs got stuck at some point (with no error message), then after a quick run with tech support all was solved the same day and we give them an additional day to test if everything was OK. And it was.

So, all in all it was a good update without issues of great importance.

Post Reply

Who is online

Users browsing this forum: thomas.lee and 18 guests