Lack of support - Veeam/Infinidat storage Plugin

[esoo]

Veeam Case: 04458963
Infinidat Cate: 49460

So upon our recent upgrade to v10a and Ibox plugin 1.1.5, we've been swamped with login/session expired event/event flood warnings from our Infinidat IBOX arrays from our Veeam Managers. There is over a 1000 logins a day, even during idle periods and these are creating massive volumes of events, which then trigger event flood warnings. We've had to suppress alerts that we should be getting just to deal with the situation.

Infinidat says go talk to Veeam
Veeam says go talk to Infinidat

Neither side wants to own the issue and I get left with a hot-potato of a problem that no-one is handling. I've asked our Veeam account manager if they can do something and will shortly do the same with our Infinidat one. Being stuck between vendors pointing fingers is never a good place to be.

[Gostev]

Hello, Erik!

Sorry to hear about this! But I can only quote what is explained when you click New Topic:

If you're unhappy with the support experience, please contact our support management directly by clicking Talk to a Manager on the Customer Portal.

Just copy & paste what you said above into the form, and I'm sure the support management will take over that hot-potato from you as it would appear that the issue got stuck at support organizations' level, and was never escalated to the corresponding Alliance teams of both vendors, as it should have been.

Thanks!

[esoo]

Thank you Gostev. I posted here as anytime in the past it comes to a "code" issue support has always said to post here. And I just tried a new topic post here, and it didn't say that on the topic entry screen.

[Gostev]

Do you not see the two lines in red font leading to the rules of posting technical issues when clicking the New Topic button? These include the information I quoted above, and more details about the purpose of these forums.

[esoo]

First thing on a Monday morning (at least my time), do you expect/think I actually clicked that?

[Gostev]

No, certainly not I also don't blame you for not clicking - I just wanted to make sure there's no some browser-specific issue where these lines cannot be seen in principle, which your previous response sounded like to me.

[foggy]

Hi Erik, looks unexpected indeed as in one of the previous plug-in updates Inifinidat has already addressed the excessive number of login sessions to the storage. I will contact their R&D team for details.

[foggy]

It turned out that in the latest version of their plug-in (1.1.5) Infinidat reverted the changes that they made before to reduce the amount of new connections that sort of explains why you've started seeing this behavior after the upgrade. So as a workaround you could try the previous plug-in version (1.1.0), should reduce the number of logins by 10-20%. Our support can share the plug-in setup package with you.

From the Veeam side, we did some optimizations in this regard in Veeam B&R v11 and are considering further improvements but not in the closest future.

[esoo]

My only concern, and I'll verify with Infinidat, is why they reverted that change. They did say it was causing issues, so not sure how to proceed.

I do still have a copy of the 1.1.0 Plugin as I hardly ever delete downloads until they are way out of date. Still have v8 install files hanging around.

[Andreas Neufert]

This question can only fully be answered by Infinidat as Veeam has no code access to their plug-in. So we can not say what they exactly changed.

Overall our product opens connections for multiple tasks that we do in the background like rescans for volumes and checking of new snapshots.
Maybe check if the volume scope in the "add storage wizard" can be reduced to specific volumes that you use for VMware with the plug-in processing.
This way we ignore all other volumes which might reduce the connection count significantly if you have a lot of other volumes/snapshots.

That a high API usage (connections) causes warnings within the storage system is a bit unclear. Admittedly, there are better ways to hadle this on the Veeam side with reusage of connections. But as Alexander shared this is something for the future and can not be changed in a simple bugfix.

[Gostev]

Here comes our Alliance team I was talking about

[esoo]

To me, the most important thing about this situation isn't necessarily getting this solved - there is a a workaround (albeit poor in my opinion), but how the organizations responded which prompted this post. There is nothing worse than being a customer and being told by two vendors it's the other's problem. Even worse to me is that both sides *knew* it was a problem and it wasn't communicated that there was a roadmap to fix it.

Pure, 3Par and NetApp haven't thrown this kind of alert, so should Infinidat? I actually side with Infinidat here - high login rates can be indicative of security issues. And the issue is to suppress the alert requires suppressing all event floods, which I think is bad as that could hide other issues.

[foggy]

That is exactly what Anton was meaning in his first post. Lower tier engineers are simply not aware of such a roadmap, not to mention that even our R&D wasn't aware of these changes to the latest Infinidat plug-in until you posted about your issue. That's why it is always recommended to escalate such cases and thanks for also posting here the details allowing us to quickly track it down with our colleagues at Infinidat.

[Gostev]

It is important to realize and differentiate that, at least from Veeam's perspective, it was not really a response from the organization - but rather from the particular support engineer. And it was not the "vendor telling you", but rather the particular engineer. While it sounds like from your perspective support engineer equals vendor, this is never the case. Making such a connection is similar to getting insulted by some McDonalds employee, and concluding that whatever that employee said is the official position of McDonalds about you (instead of just talking to the manager on duty).

You have to keep in mind there's always a chance the support engineer on your case simply does not have enough experience solving cross-vendor problems, which is exactly why it is important to escalate such cases to the support management, who will assign someone more experienced to your case. On top of actually solving your issue, this will train the initial support engineer on how to approach similar issues in future (because his or her manager will get involved). Win-win!

And I don't know what makes you think that support engineers on both sides *knew* it was a problem. No, of course T1 folks you were talking to certainly don't have the level of knowledge of actual R&D folks responsible for integration with the particular vendor, who answered you above. Although clearly, it was a surprise even for them. In general, T1 folks know "very little about everything" so to speak, which is what they need to be able to process an incredible variety of incoming cases on the product with thousands of features and integrations. But they rarely know those nitty-gritty details of each particular feature and integration... this is just not realistic to expect from T1 folks. There are actual feature specialists on higher support tiers, who T1 folks are supposed to get help from - but even those guys often have to ask R&D directly. And as it comes to the roadmap specifically, even R&D folks won't be able to comment on this (as it's my team's responsibility).

[EDIT] oops, Foggy beat me to that.

[esoo]

While I can understand regarding your differentiation of the engineer's response vs "the organization's response", I suggest you read the response to me from 11/2 at 12:59 (EST). I'm not going to quote it here as I feel that is inappropriate, but the wording there implies that the support engineer did reach out to his colleagues in an attempt to solve the issue, and the collective answer was to talk to Infinidat. As such, this generates my opinion of an "organizational response" as opposed to the response of an individual engineer

Secondarily to that, as a customer, I shouldn't have to escalate to support management in a case such as this. To me that should be part of the support process - an engineer who doesn't know how to support a situation properly should automatically reach out to a manager asking for help. If I look back on that same support message to me, it would appear that no one in the chat advised internal escalation.

This is not a new problem - it wasn't like my forum post made it first aware to people so to me that is known. It was present in v1.1.0 of the plugin, Infindat worked around it, but had to revert for v1.1.5. Now the question is, for which I don't currently have an answer - how did Infinidat interact with Veeam upon that regression?

Veeam has a public forum so you get to deal with me here, but I do want to say I'm also talking to Infinidat. This isn't just a one-sided affair.

[Gostev]

I'm not going to quote it here as I feel that is inappropriate, but the wording there implies that the support engineer did reach out to his colleagues in an attempt to solve the issue, and the collective answer was to talk to Infinidat. As such, this generates my opinion of an "organizational response" as opposed to the response of an individual engineer

Yes, and as it is now confirmed by our Alliance R&D folks, it was the technically correct response due to the issue being a regression introduced by Infinidat in the plug-in version 1.1.5

Secondarily to that, as a customer, I shouldn't have to escalate to support management in a case such as this. To me that should be part of the support process - an engineer who doesn't know how to support a situation properly should automatically reach out to a manager asking for help.

Correct, and this is exactly what this engineer did: reached out to his colleagues in an attempt to solve the issue. So in this particular case, support processes worked as they should have, without you having to escalate. The only thing that could have been done better, is support orgs including alliance team from both sides right away, instead of playing soccer with the customer as a ball. In such situations, escalating to the support management is the best way to resolve the situation, as T1 folks don't necessarily know how to detect and resolve such "stalemate" situations.

The main point is, whenever you feel something goes wrong, don't just accept it assuming this is how Veeam is, and walk away unhappy. Instead, escalate the case to support management straight from the customer support portal, and they will deal with the issue. What you shared in the first post is actually the perfect note to include in the escalation form, as it makes the bigger issue - something a support engineer of one given vendor can't see - immediately obvious. Seeing this note, the support management would have immediately engaged our alliance team for next steps.

[Gostev]

Now the question is, for which I don't currently have an answer - how did Infinidat interact with Veeam upon that regression?

Based on the fact that this regression in plug-in version 1.1.5 was a surprise for everyone here at Veeam, I guess we need to give Alliance teams from both vendors some time to actually interact in this regard, and come to some joint conclusion. I'm sure Andreas or Foggy will post an update once they have it.

[esoo]

Yes, and as it is now confirmed by our Alliance R&D folks, it was the technically correct response due to the issue being a regression introduced by Infinidat in the plug-in version 1.1.5

"Infinidat, here's your ball back"

When I first read this post (based on the forum notification and your first revision which contained nothing further), this was my immediate feeling. I would've quoted your original right away, but the forum software makes you wait.

Correct, and this is exactly what this engineer did: reached out to his colleagues in an attempt to solve the issue. So in this particular case, support processes worked as they should have, without you having to escalate. The only thing that could have been done better, is support including alliance folks from both sides right away, instead of playing soccer with the customer as a ball. In such situations, escalating to the support management is the best way to resolve the situation. T1 support folks don't necessarily know how to take such "stalemate" situations further.

The main point is, whenever you feel something goes wrong, don't just accept it assuming this is how Veeam is and walk away unhappy. Instead, escalate the case to support management straight from the customer support portal, and they will deal with the issue. What you shared in the first post is actually the perfect note to include in the escalation form, as it makes the bigger issue - something a support engineer of one given vendor can't see - immediately obvious. Seeing this note, the support management would have immediately engaged our alliance team for next steps.

Paraphrasing:
Me: "Infinidat says this is a Veeam problem" (Oct 27 case note)
SE: "Checked with my colleagues, go talk to Infinidat" (Nov 2)

You say this is proper procedure, and that I should escalate. If this is truly the case, then the support engineer should know to do that - I shouldn't have to be involved. I've been a Tier 2 sysadmin for 15 years - if tier 1 can't resolve, escalate internally has been standard procedure at every place I've been. If the engineer did do that (which could be recorded in the case notes that I don't see), then that could have been communicated better to me. A better response would've been something like "I've talked to the relevant teams, and there is a limitation to the way we call the plugin. If you could get Infinidat to reach out to our alliance team, maybe they can work something out"

Infinidat support did contact their developers - who were aware of the problem, and they clearly could have communicated better with Veeam (based on what I have read here). I have made my Infinidat contact aware of this point (and thread) as well.

[esoo]

Based on the fact that this regression in plug-in version 1.1.5 was a surprise for everyone here at Veeam, I guess we need to give Alliance teams from both vendors some time to actually interact in this regard, and come to some joint conclusion. I'm sure Andreas or Foggy will post an update once they have it.

As I stated above, and I let Infinidat know as well, this is something that should've happened when they reverted the change.

While there is a work-around, suppressing messages could hide that something bad is happening on/with the array, and that to me is an unacceptable solution.

[Gostev]

You say this is proper procedure, and that I should escalate. If this is truly the case, then the support engineer should know to do that - I shouldn't have to be involved. I've been a Tier 2 sysadmin for 15 years - if tier 1 can't resolve, escalate internally has been standard procedure at every place I've been. If the engineer did do that (which could be recorded in the case notes that I don't see), then that could have been communicated better to me. A better response would've been something like "I've talked to the relevant teams, and there is a limitation to the way we call the plugin. If you could get Infinidat to reach out to our alliance team, maybe they can work something out"

Well, I can only repeat yet again my very first post in this thread: if you feel something could be improved with the support experience you have received, or better yet if you have the specific suggestions like above, PLEASE use the "Talk to a Manager" functionality on the Customer Portal to share this feedback directly with the support management. They will review all details of your case, which we don't even have access to, and call you directly to discuss the situation - and how they plan to improve this.

Please understand that these are R&D forums, which is a totally separate organization here at Veeam. There are no support people or support managers on these forums, which is specifically emphasized when you click New Topic. And this is exactly why forum rules ask you to direct all feedback on support experience to the corresponding form, instead of posting it here, where it cannot be actioned.

All we can do is share some tips with you on how to best approach support issues - things that we know will work and get you immediate attention from people in charge of our customer support. We're just trying to help and explain how things work - there's really no point in arguing these tips with us, saying it's not how things should work based on your experience, that you should not have to be escalating etc. If you really care for this to be heard by someone who can action your feedback and improve the situation, you should contact support management directly - which is super easy to do and would have taken much less time than what you spent typing all the posts above, because normally they just call back to discuss the situation directly.

[foggy]

Hi @esoo, if the number of connections is still an issue in your environment, feel free to contact our support for a private build of the Infinidat plug-in where they have brought the relevant changes back.