V8 Lack of stability

[Scouserste81]

One of many case ID's Case # 00906479
Veeam,

I find it very frustrating that I am about to write in these forums about our issues as your support system is currently slow, unresponsive and have no idea how to help us so far.

Since migrating to Veeam version 8 we have had continuous stability issues with our system, this is very sad as prior to this version Veeam always delivered on the primary objective of any backup company..... Reliability and stability.

I appreciate companies have to move products forward but Veeam appear to have sacrificed there name of reliability and stability and pushed to many changes in one go making a product unusable, unstable and no confidence in backup data.

1. Firstly we tried the Netapp integration and had constant issues until we got an updated Veeam Agent ( well done for fixing) however this took an age due to lack of advice from the support engineers advising what needed replacing.

2. We have for weeks had issues with backup to tape (copy D2D file). This has been shipping far more data than we expected and from what we can see has no concept of separating the data files for full and daily copies.... after weeks of back and forth we get told its a bug and asked to apply update 2.... Its a shame that the support could not identify this sooner as a hotfix has been available for a number of weeks.

3. We are also now seeing deadlock issues and also have a disk to disk job continually reporting "An item with the same key has already been added", again support have no idea and simply advise us to reboot system or re-create jobs.... the backup equivalent of switch it off and back on again, it doesn't work.

We had none of these issues before we moved to version 8. We have now had weeks / months of instability with our backups and we cannot get data to tape reliably due to the system crashing or needing an update. It is a shame Veeam refuse to acknowledge all these combined issues as a priority 1 ticket as any other company due to ongoing length of time and multiple issues would deal with this as a matter of urgency.

Overall Veeam is now giving an unreliable product and the worst support I have seen for many years.... Would I now recommend Veeam to a friend... absolutely not..!!

[dannythake]

Unfortunately, it pains me to say, I have also had the similar woes of v8. I've spent far too long trying to get things to work properly when I should be focussed on other tasks within my job.

Love the Product, the support team are as helpful as possible, and the features set is great. But with the amount of private fixes I've now been provided, I've started to get a little concerned. I currently have 3 support tickets open, and a further few closed. When you have that many open at one time, the stability of the product comes into question. If the Dev/Support team manage to get my last few issues resolved (which I'm sure they will), they'll have great difficulty in getting me to upgrade to a newer version in future - Far too time consuming chasing down all the issues that come out of it, even though I didn't upgrade until Patch 1 was released!

The flip side though, there aren't many companies that will get their R&D teams to provide private fixes at all. The rough and the smooth I guess!

[Shestakov]

Hello Scouserste81,

I`ve checked the support case you provided and it looks like it was opened today, you had a Webex and sent logs. No references to another cases are provided. Could you provide case numbers of those where you got support level worse than expected?

Regarding listed issues, as far as I understood, you need an assistance only with the 3rd one, while first two are resolved. Is that correct?
Do you see the error during backup copy job?

By the way, if you are not satisfied with a support engineer, you can easily escalate your case to the upper level.
Thanks!

[Scouserste81]

Hi Shestakov,

Thank you for the reply. The ticket around our tape issues 00887245.

Other tickets we have had since version 8 update 00866423, 00744433. these are just what I have access to at the moment.

The tape issues we have cannot yet be confirmed as fixed or not as we cannot keep the system stable enough to successfully ship the volume of data to tape that we now are currently behind.

The ticket reference as today is a second attempt of reporting this issue. This issue has been a product of applying update 2 as recommended as the fix for tape issues.

I think this quite clearly shows my point that the Veeam product V8 has lost its way with reliability and stability.

With regard to support there have been a number of occasions we have attempted to escalate our tickets as a P1 case as we are not getting data offsite via tape, on numerous occasions we have been told this is not a priority in your eyes. Also, each time we speak to engineers there default answer is give me logs and we will get back to you... unfortunately we don't hear and we have to chase.

Overall the performance of Veeam product and service is degrading, Im sorry its taken a forum post to get someone to take note of our complaints.

[Shestakov]

Thanks for the reply.

Since we add many more features to our major releases than we used to (for example, over 200 new features in v8), the absolute amount of bugs in or around the new functionality is of course proportionally larger than back when our What's New used to fit into a couple of pages.

And of course, the most stable product is the one that is simply never touched. However, it makes it get stuck in a stone age eventually, so we need to find a balance here.

What matters is that we do provide hotfixes and updates in a timely manner. I see that you have installed the latest update, so let`s see if this fixes your outstanding issues. We definitely take into account your feedback, especially since you are our client for a long time.

If you need any assistance with technical questions, I am ready to help.

I`ll also send your feedback directly to support management.
Thanks for sharing your experience!

[Gostev]

This issue has been a product of applying update 2 as recommended as the fix for tape issues.
I think this quite clearly shows my point that the Veeam product V8 has lost its way with reliability and stability.

I do agree that tape support was much more stable back when we did not have one remember that we have only added it quite recently comparing to the age of the product, and v8 added many new tape features our users felt were missing before. I think this quite clearly shows that adding new complex functionality to the product increases the absolute amount of bugs, and perception of the product becoming less reliable with time. And, the same stands true for lots of other unique functionality we keep adding to our product - it simply needs time to stabilize against all those environment-specific issues that we are unable to catch in our QC labs. For example, NetApp integration that you mention in the first post has also been introduced in v8 only.

The most stable product is the one that is not evolved in terms of new functionality, and rather remaining in the state of permanent bug fix. However, such products get stuck in a stone age! So, we need to find a balance here. I think, what is most important is the fact that we continue to provide you with fixes and resolutions promptly, just as Danny have noted we do already. Please understand though that we may not know there is an issue with how your support case is handled unless you tell us! This is why we always encourage everyone to use Talk to a Manager functionality of the Customer Support Portal, when you are not happy with the quality of support.

Finally, jokes aside - yes, tape is the beast and we are spending incredible amount of time spending vendor/make/model specific issues across different tape libraries. And it also happens that some fixes occasionally brake compatibility with other models. Welcome to the world of tape support for over 100'000+ tape deployments of all kinds... but, with our customer's help and willingness to work with us, I am confident that we will eventually make everyone happy!

[dannythake]

The most stable product is the one that is not evolved in terms of new functionality, and rather remaining in the state of permanent bug fix. However, such products get stuck in a stone age! So, we need to find a balance here. I think, what is most important is the fact that we continue to provide you with fixes and resolutions promptly, just as Danny have noted we do already. Please understand though that we may not know there is an issue with how your support case is handled unless you tell us! This is why we always encourage everyone to use Talk to a Manager functionality of the Customer Support Portal, when you are not happy with the quality of support.

Completely agree with these comments, and I think we're all happy to see improvements and features being built into the software. But not at the cost of the basic, essential features. I had to have a private fix to get my backups ON to tape, now I've had another to get them back OFF tape! Doesn't fill you with confidence.

I'm going to stick my neck out slightly here though and say that my personal issues with v8 can certainly not be pinned on the Support Team - I'm almost certain they weren't the guys/girls who developed it. The issue in my particular cases, is that I have currently waited around a week for a private fix to be written and tested (of which I am 100% grateful), but what if I was in a disaster recovery situation? A week without my off-site backup? Lets assume my production site (A location with around 20 ceramic firing kilns, with very real risk of fire) were to go up in smoke. Well I could use my replica's, oh wait, my DR site is in the same business park so I cant gain access. That's ok, we've got tape, oh, umm, a bug means we cant get your business up and running Mr Managing Director. But it will only be a week to fix!

[m_zolkin]

Hi Scouserste81,
Thank you for sharing your experience and ticket IDs with us. I will go through them and make sure we are doing our best.
In future, if you feel something went wrong and you are not satisfied with the level of service being provided by our team, don't hesitate to use "Talk to Manager" button at the Customer Support Portal. 100% feedback receives responses.