Discussions specific to the VMware vSphere hypervisor
Post Reply
SiStone
Lurker
Posts: 2
Liked: never
Joined: Sep 20, 2019 12:52 pm
Full Name: Simon Stone
Contact:

HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by SiStone » Sep 20, 2019 1:41 pm

Posting this as the suggestion of Veeam Support following a lengthy investigation into an issue we have been seeing since the install of a pair of HPE Nimble Storage arrays (case 03674597)

So, it goes something like this. We have a pair of brand new HPE Nimble storage arrays. We were lucky enough to get them just as NimbleOS v5.1 came out with synchronous replication. So, we took advantage of this and had the two arrays configured as a syncrep pair.

The issues started when we installed Veeam and set up the Nimble integration. We immediately started getting 'storage disconnection' notifications raised by vCenter. After a bit of detective work, we spotted that the notificaitons coincided with the scheduled storage discovery jobs beind triggered from Veeam. Around the same time we had a number of times when the management services on one or more of our ESXi hosts would crash, resulting in the host bein unmanageable from within vCenter. The host itself (and the VM's on that host) were still runiing, but we were unable to manage them until the management services were restarted from SSH.

Tickets were then logged with vmWare and Veeam to investigate. vmWare stated in no uncertain terms that the host management service failure was a direct result (and an apparent known issue in ESXi) of the persistent storage disconnects. The storage disconnects appeared to be an issue within Veeam, tricky bit was determining what exactly the problem is.

After a lengthy investigation and several remote sessions, it turns out that the Nimble syncrep functionality doesn't play well with Veeam. I assume it is something to do with the arrays presenting multiple volumes with the same LUN ID, and Veeam scanning the same volume on multiple arrays which is causing vCenter to get screwed up. vCenter then sees what it thinks are LUNs getting disconnected, which is wat it is reporting.

As a result of the work with the Veaam tech guys, we have put a temporary workaround in place which kills off the scheduled storage discovery jobs, and I think it also changes the way that manual scans are working, and this has stopped the storage disconnects, which in turn has stopped the ESXi management services from crashing. This workaround is not without it's drawbacks. It breaks the storage snapshot chain for a single VM (so, if a VM is removed from a volume in a storage snapshot, it is removed from all snapshots in the chain). It also means we need to do manual scans when we add new VM's. Not massively onerous, but inconvenient to say the least.

My concern is that it turns out that this issue (and full Nimble Syncrep functionality) will not be supported in Veeam v9, and there are also no current plans for it to be fully supported in the upcoming v10 either. I suspect strongly that most customers taking Nimble arrays on will want to use syncrep, especially as with NimbleOS v5.3 (which is now available) this becomes generally available. For the industry leading backup and replication product to advertise integration with these arrays, but for it not to function properly, seems strange to me.

Has anyone else come across problems like this? Thoughts on the v10 roadmap?

foggy
Veeam Software
Posts: 18242
Liked: 1557 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy » Sep 20, 2019 3:44 pm

Hi Simon, HPE Nimble synchronous replication is not currently supported by Veeam B&R but we're looking into fully supporting it in our future releases. Thanks for sharing your findings.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3514 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Gostev » Sep 22, 2019 8:55 pm

SiStone wrote:
Sep 20, 2019 1:41 pm
For the industry leading backup and replication product to advertise integration with these arrays, but for it not to function properly, seems strange to me.
Are you saying we should not have been advertising our integration with Nimble arrays for the past 3 years, because we should have somehow predicted that in late 2019, a new Nimble array feature will be released that our current integration will not be compatible with out of the box? This makes no sense.

Not to mention that the majority of our customers don't even use asynchronous replication, which we do fully support since day 1. So, I beg to disagree with your prediction that "most customers taking Nimble arrays on will want to use syncrep".

Having said that, as Foggy already mentioned, this is one of the features we're looking to support in future releases.

Andreas Neufert
Veeam Software
Posts: 3812
Liked: 683 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert » Sep 23, 2019 10:08 am

And we are working closely with the Nimble team since the beta of this feature to integrate it in our product. For technical reasons related to Nimble themself we are not there yet.

SiStone
Lurker
Posts: 2
Liked: never
Joined: Sep 20, 2019 12:52 pm
Full Name: Simon Stone
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by SiStone » Sep 23, 2019 10:58 am

Gostev wrote:
Sep 22, 2019 8:55 pm
Are you saying we should not have been advertising our integration with Nimble arrays for the past 3 years, because we should have somehow predicted that in late 2019, a new Nimble array feature will be released that our current integration will not be compatible with out of the box? This makes no sense.
No, I am not saying that at all, and I am pretty sure my post didn't say that! I have worked with Veeam and Nimble arrays since before the integration was first launched, and the integration was a massive leap forward giving all sorts of benefits. What I was surprised about was that the support and development teams seemed entirely unaware of the new functionality in the arrays. I would have expected (rightly or wrongly) that Nimble (HPE) and Veeam would work togehter to support new functionality, or at the very least ensure new array functionality does not cause the kind of issues we have seen. Does this mean that Veem are not given advance notification (or even sight of) of upcoming software releases by HPE Nimble?

I do however, disagree with your statement on Syncrep. I fully expect the adoption of syncrep functionality in Nimble arrays, where their hardware supports it, to be high. This is not only based on the fact that it has been one of the most requested features since I started using Nimble arrays around 4 years ago, but also it was the one area they lagged behind other comparable offerings such as 3PAR.

Again, I was only posting this at the request of the Veeam support team in an attempt to bring it into focus in hope that it would be better supported soon.

Andreas Neufert
Veeam Software
Posts: 3812
Liked: 683 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert » Sep 23, 2019 11:40 am 1 person likes this post

As I said we are fully aware from R&D perspective about this since HPE gave us ahead notice. Since then we are working with HPE on this, but there are technical things that hinder us from implementation. I can not comment about details based on NDA.

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 36 guests