Discussions specific to the VMware vSphere hypervisor
Post Reply
SiStone
Lurker
Posts: 2
Liked: 1 time
Joined: Sep 20, 2019 12:52 pm
Full Name: Simon Stone
Contact:

HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by SiStone » 1 person likes this post

Posting this as the suggestion of Veeam Support following a lengthy investigation into an issue we have been seeing since the install of a pair of HPE Nimble Storage arrays (case 03674597)

So, it goes something like this. We have a pair of brand new HPE Nimble storage arrays. We were lucky enough to get them just as NimbleOS v5.1 came out with synchronous replication. So, we took advantage of this and had the two arrays configured as a syncrep pair.

The issues started when we installed Veeam and set up the Nimble integration. We immediately started getting 'storage disconnection' notifications raised by vCenter. After a bit of detective work, we spotted that the notificaitons coincided with the scheduled storage discovery jobs beind triggered from Veeam. Around the same time we had a number of times when the management services on one or more of our ESXi hosts would crash, resulting in the host bein unmanageable from within vCenter. The host itself (and the VM's on that host) were still runiing, but we were unable to manage them until the management services were restarted from SSH.

Tickets were then logged with vmWare and Veeam to investigate. vmWare stated in no uncertain terms that the host management service failure was a direct result (and an apparent known issue in ESXi) of the persistent storage disconnects. The storage disconnects appeared to be an issue within Veeam, tricky bit was determining what exactly the problem is.

After a lengthy investigation and several remote sessions, it turns out that the Nimble syncrep functionality doesn't play well with Veeam. I assume it is something to do with the arrays presenting multiple volumes with the same LUN ID, and Veeam scanning the same volume on multiple arrays which is causing vCenter to get screwed up. vCenter then sees what it thinks are LUNs getting disconnected, which is wat it is reporting.

As a result of the work with the Veaam tech guys, we have put a temporary workaround in place which kills off the scheduled storage discovery jobs, and I think it also changes the way that manual scans are working, and this has stopped the storage disconnects, which in turn has stopped the ESXi management services from crashing. This workaround is not without it's drawbacks. It breaks the storage snapshot chain for a single VM (so, if a VM is removed from a volume in a storage snapshot, it is removed from all snapshots in the chain). It also means we need to do manual scans when we add new VM's. Not massively onerous, but inconvenient to say the least.

My concern is that it turns out that this issue (and full Nimble Syncrep functionality) will not be supported in Veeam v9, and there are also no current plans for it to be fully supported in the upcoming v10 either. I suspect strongly that most customers taking Nimble arrays on will want to use syncrep, especially as with NimbleOS v5.3 (which is now available) this becomes generally available. For the industry leading backup and replication product to advertise integration with these arrays, but for it not to function properly, seems strange to me.

Has anyone else come across problems like this? Thoughts on the v10 roadmap?

foggy
Veeam Software
Posts: 19475
Liked: 1768 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy »

Hi Simon, HPE Nimble synchronous replication is not currently supported by Veeam B&R but we're looking into fully supporting it in our future releases. Thanks for sharing your findings.

Gostev
SVP, Product Management
Posts: 26726
Liked: 4281 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Gostev »

SiStone wrote: Sep 20, 2019 1:41 pmFor the industry leading backup and replication product to advertise integration with these arrays, but for it not to function properly, seems strange to me.
Are you saying we should not have been advertising our integration with Nimble arrays for the past 3 years, because we should have somehow predicted that in late 2019, a new Nimble array feature will be released that our current integration will not be compatible with out of the box? This makes no sense.

Not to mention that the majority of our customers don't even use asynchronous replication, which we do fully support since day 1. So, I beg to disagree with your prediction that "most customers taking Nimble arrays on will want to use syncrep".

Having said that, as Foggy already mentioned, this is one of the features we're looking to support in future releases.

Andreas Neufert
VP, Product Management
Posts: 4533
Liked: 860 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert »

And we are working closely with the Nimble team since the beta of this feature to integrate it in our product. For technical reasons related to Nimble themself we are not there yet.

SiStone
Lurker
Posts: 2
Liked: 1 time
Joined: Sep 20, 2019 12:52 pm
Full Name: Simon Stone
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by SiStone »

Gostev wrote: Sep 22, 2019 8:55 pm Are you saying we should not have been advertising our integration with Nimble arrays for the past 3 years, because we should have somehow predicted that in late 2019, a new Nimble array feature will be released that our current integration will not be compatible with out of the box? This makes no sense.
No, I am not saying that at all, and I am pretty sure my post didn't say that! I have worked with Veeam and Nimble arrays since before the integration was first launched, and the integration was a massive leap forward giving all sorts of benefits. What I was surprised about was that the support and development teams seemed entirely unaware of the new functionality in the arrays. I would have expected (rightly or wrongly) that Nimble (HPE) and Veeam would work togehter to support new functionality, or at the very least ensure new array functionality does not cause the kind of issues we have seen. Does this mean that Veem are not given advance notification (or even sight of) of upcoming software releases by HPE Nimble?

I do however, disagree with your statement on Syncrep. I fully expect the adoption of syncrep functionality in Nimble arrays, where their hardware supports it, to be high. This is not only based on the fact that it has been one of the most requested features since I started using Nimble arrays around 4 years ago, but also it was the one area they lagged behind other comparable offerings such as 3PAR.

Again, I was only posting this at the request of the Veeam support team in an attempt to bring it into focus in hope that it would be better supported soon.

Andreas Neufert
VP, Product Management
Posts: 4533
Liked: 860 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert » 1 person likes this post

As I said we are fully aware from R&D perspective about this since HPE gave us ahead notice. Since then we are working with HPE on this, but there are technical things that hinder us from implementation. I can not comment about details based on NDA.

ftepe
Novice
Posts: 3
Liked: never
Joined: Jul 10, 2018 7:11 am
Full Name: Florian Tepe
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by ftepe »

Hi,

this post was hard to find but I'm dealing with exactly the same issue! Are there any updates as so far since the last two month?

foggy
Veeam Software
Posts: 19475
Liked: 1768 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy »

Hi Florian, no major updates so far.

ftepe
Novice
Posts: 3
Liked: never
Joined: Jul 10, 2018 7:11 am
Full Name: Florian Tepe
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by ftepe »

Thanks for the fast response!

ftepe
Novice
Posts: 3
Liked: never
Joined: Jul 10, 2018 7:11 am
Full Name: Florian Tepe
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by ftepe »

Hi,

any news after another month? Will it be supported in V10?

Andreas Neufert
VP, Product Management
Posts: 4533
Liked: 860 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert »

Hi Florian,

There will be no change for this in v10. As I said above we are working with HPE on it, basically defining the priorities with them and working based on their input on overall HPE feature. Potentially it makes sense as well to raise awareness at HPE for this feature request through your HPE sales rep.

Based on our communication rules given by our top management, we can not comment about upcoming features and timelines until a feature is in the beta stage. We will update this thread here when/if we reach that stage.

Andreas

andrie
Influencer
Posts: 14
Liked: 3 times
Joined: Nov 20, 2015 12:37 pm
Full Name: Andy
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by andrie » 2 people like this post

We had similar issues with Veeam and Nimble storage snapshots without using the storage integration.
Veeam brings the storage snapshots online to scan them and the default setting on Nimble for ESX is to present the volumes and snapshots. This causes the storage disconnects when the snapshots are taken offline again.
The solution for us was to change the volume presentation on the Nimble systems from "Volume and Snapshots" to "Volume only".
This prevents the ESX hosts from seeing the storage snapshots during the Veeam scanning.

Andreas Neufert
VP, Product Management
Posts: 4533
Liked: 860 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert »

Thanks for sharing. I have asked the doucmentation team to add this there.

bernd.m
Service Provider
Posts: 1
Liked: 2 times
Joined: Feb 13, 2020 1:25 pm
Full Name: Bernd Mueller
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by bernd.m » 2 people like this post

I had the described issues at two customer sites too, and will confirm, that the solution from @andrie, to switch the ACL from "Volume & Snapshots" to "Volume Only" for the involved ESXi hosts is working fine.
I also want to warn you, that this setting, despite the Nimble GUI gives no warning at all, ist NOT an online procedure!
All IO to the volume hast to be stopped to edit the ACL from "Volume & Snapshots" to "Volume Only"!

mwhite
Lurker
Posts: 2
Liked: 1 time
Joined: Oct 03, 2017 6:27 am
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by mwhite » 1 person likes this post

Just to ping this, the Nimble Veeam integration guides states the following;

"A VMware datastore, which is targeted by Veeam, and backed by a Nimble volume will require that the volume's ACL is set to Volume Only. This is to prevent any connections being made between VMware and online snapshots created by Veeam. Connections to online snapshots (made by Veeam) can result in VMware showing dropped paths and the possibility of the datastore going into PDL."

I hope though that the synchronous replication support is not too far away

stuartmacgreen
Expert
Posts: 124
Liked: 24 times
Joined: May 01, 2012 11:56 am
Full Name: Stuart Green
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by stuartmacgreen »

+1 for this syncrep support implementation.

Thanks
Backup Infrastructure: Veeam B&R 10a• vSphere 6.5 u3 •HPE Nimble AF40 [FC] (Primary) •HPE Nimble HF40 [FC] (Repositories) • HPE MSL3040 LTO-7 (Tape)

foggy
Veeam Software
Posts: 19475
Liked: 1768 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy »

Thanks for that, I will make sure we have a similar note in the documentation on our side as well.

stuartmacgreen
Expert
Posts: 124
Liked: 24 times
Joined: May 01, 2012 11:56 am
Full Name: Stuart Green
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by stuartmacgreen »

Can you give any insight/detail to the 'workaround'? (reg key?)
Curious... did you exclude any Nimble Volumes within the Nimble system listed in Veeam Storage Infrastructure? (ie the Downstream Sync Rep volumes.)

Thanks
SiStone wrote: Sep 20, 2019 1:41 pm Posting this as the suggestion of Veeam Support following a lengthy investigation into an issue we have been seeing since the install of a pair of HPE Nimble Storage arrays (case 03674597)
Backup Infrastructure: Veeam B&R 10a• vSphere 6.5 u3 •HPE Nimble AF40 [FC] (Primary) •HPE Nimble HF40 [FC] (Repositories) • HPE MSL3040 LTO-7 (Tape)

foggy
Veeam Software
Posts: 19475
Liked: 1768 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy »

Hi Stuart, according to the case notes, the registry value that Simon is talking about is SanRescanSnapshotsParseFS = 0. Excluding the volume copies from the rescan did not help in his case.

Post Reply

Who is online

Users browsing this forum: No registered users and 16 guests