9.5 Update 4 The worst Update Ever!!!!

[nitramd]

@pshute

Agreed. I expect that this is one of the reasons Anton runs metrics, for better transparency.

With that said, it's also great to see end users post their upgrade results which seem to match the metrics.

[mkaec]

I regularly giver other vendors (non-competitors to Veeam) links to these forums as examples of how to properly setup a successful forum.

[rossj_nz]

I am a Service Provider with a number of years experience with Veeam B&R. I would class myself as a Veeam supporter and user and have had a number of issues caused by Update 4. I think that Veeam have this wrong where their versioning is misleading clients into "update mentality" being sort of minor not major. Typically an update contains bug fixes and maybe some enhancements. Updates should not be new major functionality which looking at Update 4 it is full of. For my organisation the main reason for implementing Update 4 is the support of vCloud Director 9.5. Update 4 should have been version 9.6

Update 4 was further complicated by the change in licensing from Update 3. To the extent that for a Service Provider the upgrade confirms your existing license is valid and promptly expires it with 90 days grace period. Flawed logic at play here. In my case i then found that the Veeam Portal has some error or corruption that did not provide a valid Update 4 Service Provider license. This then links to the ability to log a support call via the Veeam Portal and meaning the only way to log a support call is over the phone. So not a great experience.

The issues I encountered so far are: Issue 1: Veeam Console Update 4 crashing on opening due to a bug in the time am/pm upper or lower case setting. https://www.veeam.com/kb2884 Issue 2: Veeam Guest Catalog Service will not start after Update 4 upgrade from Update 3. Will log a support call for this. Issue 3: A backup job that has been working fine for months now crashes once a week with the error "Unable to allocate memory for storage metadata bank. Failed to open storage for read access". Will log a support call for this.

So unless you need Update 4, ignore the number of Update 4 downloads being quoted as this is near meaningless. It is the number of update 4 licences that have been activated that is the important number, my advice is to wait. If you are a Service Provider then check that Update 4 licenses appear before you embark on the Update.

[mcz]

...correct and we also see that there are more cases where the update worked without any issues. So it doesn't look like that it's the worst update ever (which I personally never doubted).

[Gostev]

All, thanks for your passionate feedback! We're blessed to have users who don't only come to forums to post nasty comments or [rightfully] complain about technical issues, but also those who take their time to post good things when things DO work. This is extremely rare, and I am very thankful to you all for that.

One thing that will always be special about Veeam is the sheer size of our customer base. With over 330K paying users, and over 1M of free edition users, there will always be customers running into various environment-specific problems after every major release. And the reason is obvious: our test labs can never cover the incredible variety of real-world environments.

Ross provided the perfect example of one issue that impacted upgrade for our customers from a few countries because of their very special regional settings format. Can we possibly blame our QC for not having an upgrade testing lab for every country in the world? Probably not. Will this bug impact at least 0.1% of all Veeam customers (which means hundreds), making them all upset and thinking "this is the worst release ever"? No doubt.

Our QC is obsessed with quality, which is why we never announce release dates. We just know that our QC leadership will simply not sign off the build until every QC team lead is comfortable. This drives me personally crazy, because as the PM I am always teared apart by everyone to ship the new release ASAP. But, when I take an unbiased view, I do recognize that in the past 10 years they've mastered finding that fine line when further in-house testing is largely pointless, and it's time to ship the product to the masses. I see the proof to that in every release, seeing how most bugs reported by customers would be impossible to find in our own labs in the reasonable time frame.

So, while we do ship the product at the right time, when QC feels it's ready - issues are always expected following the major release for the reasons I mentioned earlier. However, customer support is the highest priority for Veeam R&D, and they work really hard to provide hot fixes in a matter of days. This is one compliment that I keep hearing from end users: Veeam is by far the fastest vendor to provide hot fixes among all Enterprise software vendors they're working with. So you can be sure that even if circumstances force you to be an early adopted of a major release, we will go out of our way to make it work for you.

As far as the overall release quality trend, this is something you have to trust me with. Because not only I saw every B&R release since v2, but I also have access to all support statistics, such as:
• Number of support cases for the new release relative to the total number of customers
• Number of bugs relative to the number of new features introduced
• Number of major and critical bugs in the new release

Long story short, the trend has been positive since version 6.0, so the release quality is constantly improving. It may be hard to see this, because with 4000 new customers added every month, the absolute number of customers experiencing issues with every major release growth faster than the actual release quality. Although ironically, I can tell you that Update 4 is the first ever release for which this usual topic about "this release being the worst ever" took over 1 month and 90K downloads since GA to appear, instead of the usual week or two. While unlike support metrics, this one is not scientific at all - this tells me personally a lot.

So really, business is usual - and I can only promise you two things:
1. Veeam R&D will do even better job with the next release in regards to its quality.
2. But regardless of that, I will see you all in the same exact topic about the next release 1 year from now

Some things just never change!

[mcz]

I can only confirm what Anton has mentioned: I had two support cases in the past where we had issues after upgrades and needed a hotfix. One caused SureBackup to fail and the other prevented indexing from being successful. It took quite a while to get the root cause of the issues (T3 and T4 support was needed) but I tell you these guys are incredible!! Within 2 days I got the hotfix for an issue that has never been seen before and after deployment it just worked as a charme.

Also I would like to mention that the veeam support is for sure the best support in the world. They understand you, they listen to you, they are ALWAYS friendly, they are very competent, have sense of humor (which helps a lot when IT is stressing you) and they have a very good response time. They even help you if the issue is caused by 3rd party components like OS. As long as they can help you, they will. You never hear something like "sorry but please ask your admin to change this or that, this is not part of our product". Never. Even in days of upgrade periods you get a response within a day. If you have a case where you feel that you might need the next tier because a deeper investigation is needed, you can manually transfer the ticket to the next level in the portal - did you know that (thank you Anton for this nice hint some month ago)? What I do appreciate a lot is the knowledge support engineers are sharing with you. They don't tell you "I don't know" or "I cannot talk about that", no, they explain you very well how the components are working. This helps a lot for understanding the problem and solving future cases on your own. I can remember I once talked to a T4 engineer who was also involved in development and he told me exactly how one specific component was working. It is still incredible to me when I think about it...

Compared to other bit IT companies veeam's support is just supreme. I don't wannt to post a name here but I'm working on tickets with a big OS vendor from time to time and the first problem is always communication: The other person hears you but they don't understand your point - very cumbersome. Response time? No exception that you are waiting 2-3 days for a response regarding a simple question / configuration. You send messages, saying it's urgent. Slow response. Sometimes it's better to do a trial & error by your own instead of waiting. If you call a veeam engineer and he isn't available, he will contact you within some hours letting you know that he has been informed that you have called - this is so great.

Even if your upgrade fails or is causing big issues you can be sure that support will do everything the can do to help you. This is my veeam-experience within the last years.

[rboynton]

In our case, we had an issue where the enterprise installation bombed, so had to get help from support. After he determined what was going on, the rest of the upgrade worked fine. We actually reloaded our Veeam B&R (physical) server yesterday from 2008r2 to 2012r2 so we could use a new Tape Autoloader (drive was not recognized by 2008r2, and no driver was available). After the upgrade (with installing the latest firmware on the server as well), the performance is better than before. We simply backed up the configuration, then restored it onto the new installation. Only thing we had to do was to rescan the repositories & re-enter all the saved credentials. Sure appreciated being able to restore the DB so easily. Saved a TON of hours trying to redo all the jobs!

[rossj_nz]

Its fair to say that Veeam gets these updates right almost all of the time. Its also great that Veeam allows comments that are factual and put in a reasonable way when things do go wrong. Veeam are the leading backup Vendor when it comes to support for products like vSphere and vCloud Director as these products are updated by VMware 2 to 3 times a year.

I would also say that the best support experience i have ever had with a Vendor was from Veeam with a severity 1 issue. Veeam support contacted me and starting working on the issue within 5 minutes of me logging the support ticket.

One thing that i would like to see taken on board is the definition of an update vs a major or point release and the signalling that this sends to their customers.

[Zew]

I upgraded, was pretty smooth. Loved the new splash loading screen... mhmmm delightful.

The only issue I had, which there was already another Fourm post about, was a new failed job physical-cloud-f49/rescan-of-manually-a ... 56589.html "Rescan of Manually added failed"

[DaveWatkins]

@Gostev

Curious, does QA integrate tests to catch these date/time format bugs into future testing or is it considered too small of a problem to test for?

[itrabbit]

I won't be upgrading any time soon, I don't understand this new licensing and I have tried several times to understand it.

Our backups are solid and I don't want to go through the pain of failed upgrades or issues with licensing, so I will wait a bit more.

We have 6 Socket Essential Bundle (which is the max sockets you can have for this license) + 7 Windows Server agents. So I really don't understand whats going to change and I don't want to be paying more for something we already bought.

[Andreas Neufert]

Curious, does QA integrate tests to catch these date/time format bugs into future testing or is it considered too small of a problem to test for?

When we spot issues like this, we add it to our test plan. It would not make sense if the next product has the exact same bug!

[ITP-Stan]

My only criticism is that the new licensing model is complex and it has not been talked about enough prior to the launch of U4.
It took me by surprise as I'm sure a lot others as well.

And I agree with others that Veeam should go back to a logical version numbering. This should have been 10.0

[skrause]

I would say this was my "worst" experience with an update to Veeam for only one reason: The requirement for .NET 4.6 caught me by surprise and meant I needed to reboot my B&R and repository servers a couple of extra times for the .NET install and the subsequent .NET patches. Otherwise everything else went pretty smoothly.

I did have to call my SE to get a handle on the impact of the new licensing model, so the communication from Veeam on that could have been clearer.

[idstephen]

9.5 Update 4 was very smooth at my site! Thank you Veeam for making it so...

[kqf_chris]

I upgraded to U4. The process itself went smoothly but I did get hit with a nasty tape bug where Update 4 would delete my full backups after adding the incremental(s). I contacted support and they promptly sent me a hotfix which took care of it.

However, the real issues for me were licensing-related and lack of training on the Veeam side. It took forever to get any info on the new per-instance licensing. It seems like it was tossed out there without much planning. None of my questions could be answered in a timely fashion.

Aside from all that, Veeam is the best backup product I have ever used, bar none.

[C_Montero]

I have updated 3 systems to U4, the upgrade process went well on all 3 of them, 2 of them are VMware and 1 of them is Hyper-V.

The only issue I had was that one of the deployments with Hyper-V had an issue with some virtual machines and VSS, the jobs got stuck at some point (with no error message), then after a quick run with tech support all was solved the same day and we give them an additional day to test if everything was OK. And it was.

So, all in all it was a good update without issues of great importance.