Discussions specific to the VMware vSphere hypervisor
SiStone
Lurker
Posts: 2
Liked: 4 times
Joined: Sep 20, 2019 12:52 pm
Full Name: Simon Stone
Contact:

HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by SiStone » 4 people like this post

Posting this as the suggestion of Veeam Support following a lengthy investigation into an issue we have been seeing since the install of a pair of HPE Nimble Storage arrays (case 03674597)

So, it goes something like this. We have a pair of brand new HPE Nimble storage arrays. We were lucky enough to get them just as NimbleOS v5.1 came out with synchronous replication. So, we took advantage of this and had the two arrays configured as a syncrep pair.

The issues started when we installed Veeam and set up the Nimble integration. We immediately started getting 'storage disconnection' notifications raised by vCenter. After a bit of detective work, we spotted that the notificaitons coincided with the scheduled storage discovery jobs beind triggered from Veeam. Around the same time we had a number of times when the management services on one or more of our ESXi hosts would crash, resulting in the host bein unmanageable from within vCenter. The host itself (and the VM's on that host) were still runiing, but we were unable to manage them until the management services were restarted from SSH.

Tickets were then logged with vmWare and Veeam to investigate. vmWare stated in no uncertain terms that the host management service failure was a direct result (and an apparent known issue in ESXi) of the persistent storage disconnects. The storage disconnects appeared to be an issue within Veeam, tricky bit was determining what exactly the problem is.

After a lengthy investigation and several remote sessions, it turns out that the Nimble syncrep functionality doesn't play well with Veeam. I assume it is something to do with the arrays presenting multiple volumes with the same LUN ID, and Veeam scanning the same volume on multiple arrays which is causing vCenter to get screwed up. vCenter then sees what it thinks are LUNs getting disconnected, which is wat it is reporting.

As a result of the work with the Veaam tech guys, we have put a temporary workaround in place which kills off the scheduled storage discovery jobs, and I think it also changes the way that manual scans are working, and this has stopped the storage disconnects, which in turn has stopped the ESXi management services from crashing. This workaround is not without it's drawbacks. It breaks the storage snapshot chain for a single VM (so, if a VM is removed from a volume in a storage snapshot, it is removed from all snapshots in the chain). It also means we need to do manual scans when we add new VM's. Not massively onerous, but inconvenient to say the least.

My concern is that it turns out that this issue (and full Nimble Syncrep functionality) will not be supported in Veeam v9, and there are also no current plans for it to be fully supported in the upcoming v10 either. I suspect strongly that most customers taking Nimble arrays on will want to use syncrep, especially as with NimbleOS v5.3 (which is now available) this becomes generally available. For the industry leading backup and replication product to advertise integration with these arrays, but for it not to function properly, seems strange to me.

Has anyone else come across problems like this? Thoughts on the v10 roadmap?

foggy
Veeam Software
Posts: 20872
Liked: 2048 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy »

Hi Simon, HPE Nimble synchronous replication is not currently supported by Veeam B&R but we're looking into fully supporting it in our future releases. Thanks for sharing your findings.

Gostev
SVP, Product Management
Posts: 30020
Liked: 5934 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Gostev »

SiStone wrote: Sep 20, 2019 1:41 pmFor the industry leading backup and replication product to advertise integration with these arrays, but for it not to function properly, seems strange to me.
Are you saying we should not have been advertising our integration with Nimble arrays for the past 3 years, because we should have somehow predicted that in late 2019, a new Nimble array feature will be released that our current integration will not be compatible with out of the box? This makes no sense.

Not to mention that the majority of our customers don't even use asynchronous replication, which we do fully support since day 1. So, I beg to disagree with your prediction that "most customers taking Nimble arrays on will want to use syncrep".

Having said that, as Foggy already mentioned, this is one of the features we're looking to support in future releases.

Andreas Neufert
VP, Product Management
Posts: 6161
Liked: 1280 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert »

And we are working closely with the Nimble team since the beta of this feature to integrate it in our product. For technical reasons related to Nimble themself we are not there yet.

SiStone
Lurker
Posts: 2
Liked: 4 times
Joined: Sep 20, 2019 12:52 pm
Full Name: Simon Stone
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by SiStone »

Gostev wrote: Sep 22, 2019 8:55 pm Are you saying we should not have been advertising our integration with Nimble arrays for the past 3 years, because we should have somehow predicted that in late 2019, a new Nimble array feature will be released that our current integration will not be compatible with out of the box? This makes no sense.
No, I am not saying that at all, and I am pretty sure my post didn't say that! I have worked with Veeam and Nimble arrays since before the integration was first launched, and the integration was a massive leap forward giving all sorts of benefits. What I was surprised about was that the support and development teams seemed entirely unaware of the new functionality in the arrays. I would have expected (rightly or wrongly) that Nimble (HPE) and Veeam would work togehter to support new functionality, or at the very least ensure new array functionality does not cause the kind of issues we have seen. Does this mean that Veem are not given advance notification (or even sight of) of upcoming software releases by HPE Nimble?

I do however, disagree with your statement on Syncrep. I fully expect the adoption of syncrep functionality in Nimble arrays, where their hardware supports it, to be high. This is not only based on the fact that it has been one of the most requested features since I started using Nimble arrays around 4 years ago, but also it was the one area they lagged behind other comparable offerings such as 3PAR.

Again, I was only posting this at the request of the Veeam support team in an attempt to bring it into focus in hope that it would be better supported soon.

Andreas Neufert
VP, Product Management
Posts: 6161
Liked: 1280 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert » 1 person likes this post

As I said we are fully aware from R&D perspective about this since HPE gave us ahead notice. Since then we are working with HPE on this, but there are technical things that hinder us from implementation. I can not comment about details based on NDA.

ftepe
Novice
Posts: 3
Liked: 1 time
Joined: Jul 10, 2018 7:11 am
Full Name: Florian Tepe
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by ftepe » 1 person likes this post

Hi,

this post was hard to find but I'm dealing with exactly the same issue! Are there any updates as so far since the last two month?

foggy
Veeam Software
Posts: 20872
Liked: 2048 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy »

Hi Florian, no major updates so far.

ftepe
Novice
Posts: 3
Liked: 1 time
Joined: Jul 10, 2018 7:11 am
Full Name: Florian Tepe
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by ftepe »

Thanks for the fast response!

ftepe
Novice
Posts: 3
Liked: 1 time
Joined: Jul 10, 2018 7:11 am
Full Name: Florian Tepe
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by ftepe »

Hi,

any news after another month? Will it be supported in V10?

Andreas Neufert
VP, Product Management
Posts: 6161
Liked: 1280 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert »

Hi Florian,

There will be no change for this in v10. As I said above we are working with HPE on it, basically defining the priorities with them and working based on their input on overall HPE feature. Potentially it makes sense as well to raise awareness at HPE for this feature request through your HPE sales rep.

Based on our communication rules given by our top management, we can not comment about upcoming features and timelines until a feature is in the beta stage. We will update this thread here when/if we reach that stage.

Andreas

andrie
Enthusiast
Posts: 27
Liked: 8 times
Joined: Nov 20, 2015 12:37 pm
Full Name: Andy
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by andrie » 2 people like this post

We had similar issues with Veeam and Nimble storage snapshots without using the storage integration.
Veeam brings the storage snapshots online to scan them and the default setting on Nimble for ESX is to present the volumes and snapshots. This causes the storage disconnects when the snapshots are taken offline again.
The solution for us was to change the volume presentation on the Nimble systems from "Volume and Snapshots" to "Volume only".
This prevents the ESX hosts from seeing the storage snapshots during the Veeam scanning.

Andreas Neufert
VP, Product Management
Posts: 6161
Liked: 1280 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert »

Thanks for sharing. I have asked the doucmentation team to add this there.

bernd.m
Service Provider
Posts: 1
Liked: 2 times
Joined: Feb 13, 2020 1:25 pm
Full Name: Bernd Mueller
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by bernd.m » 2 people like this post

I had the described issues at two customer sites too, and will confirm, that the solution from @andrie, to switch the ACL from "Volume & Snapshots" to "Volume Only" for the involved ESXi hosts is working fine.
I also want to warn you, that this setting, despite the Nimble GUI gives no warning at all, ist NOT an online procedure!
All IO to the volume hast to be stopped to edit the ACL from "Volume & Snapshots" to "Volume Only"!

mwhite
Lurker
Posts: 2
Liked: 1 time
Joined: Oct 03, 2017 6:27 am
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by mwhite » 1 person likes this post

Just to ping this, the Nimble Veeam integration guides states the following;

"A VMware datastore, which is targeted by Veeam, and backed by a Nimble volume will require that the volume's ACL is set to Volume Only. This is to prevent any connections being made between VMware and online snapshots created by Veeam. Connections to online snapshots (made by Veeam) can result in VMware showing dropped paths and the possibility of the datastore going into PDL."

I hope though that the synchronous replication support is not too far away

stuartmacgreen
Expert
Posts: 144
Liked: 33 times
Joined: May 01, 2012 11:56 am
Full Name: Stuart Green
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by stuartmacgreen »

+1 for this syncrep support implementation.

Thanks
Backup Infrastructure: Veeam B&R 11.0.1.1261 P20220302 (Mar 2022) • vSphere 7.0u3h •HPE Nimble AF40 [FC] (Primary) •HPE Nimble HF40 [FC] (Repositories) • HPE MSL3040 LTO-7 (Tape)

foggy
Veeam Software
Posts: 20872
Liked: 2048 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy »

Thanks for that, I will make sure we have a similar note in the documentation on our side as well.

stuartmacgreen
Expert
Posts: 144
Liked: 33 times
Joined: May 01, 2012 11:56 am
Full Name: Stuart Green
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by stuartmacgreen »

Can you give any insight/detail to the 'workaround'? (reg key?)
Curious... did you exclude any Nimble Volumes within the Nimble system listed in Veeam Storage Infrastructure? (ie the Downstream Sync Rep volumes.)

Thanks
SiStone wrote: Sep 20, 2019 1:41 pm Posting this as the suggestion of Veeam Support following a lengthy investigation into an issue we have been seeing since the install of a pair of HPE Nimble Storage arrays (case 03674597)
Backup Infrastructure: Veeam B&R 11.0.1.1261 P20220302 (Mar 2022) • vSphere 7.0u3h •HPE Nimble AF40 [FC] (Primary) •HPE Nimble HF40 [FC] (Repositories) • HPE MSL3040 LTO-7 (Tape)

foggy
Veeam Software
Posts: 20872
Liked: 2048 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy »

Hi Stuart, according to the case notes, the registry value that Simon is talking about is SanRescanSnapshotsParseFS = 0. Excluding the volume copies from the rescan did not help in his case.

Hnns
Novice
Posts: 4
Liked: never
Joined: Nov 17, 2020 8:06 pm
Full Name: Henry Nevans
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Hnns »

Any updates on this from Veeam? It seems this has been a known issue for over a year now..

foggy
Veeam Software
Posts: 20872
Liked: 2048 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy »

No updates as of now, this configuration is not currently supported.

Hnns
Novice
Posts: 4
Liked: never
Joined: Nov 17, 2020 8:06 pm
Full Name: Henry Nevans
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Hnns »

I think it's been well established that it's not currently supported, the question is are there any plans on actually making this supported, yes or no, and when?

foggy
Veeam Software
Posts: 20872
Liked: 2048 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy » 1 person likes this post

Yes, we plan to support it, but I cannot provide any particular ETA at the moment. We're moving towards that direction though.

theman
Service Provider
Posts: 4
Liked: never
Joined: Sep 04, 2017 9:43 pm
Full Name: ak
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by theman »

#04508534
I'm facing similar issue where nimble storage is integrated with Veeam (B& R -Version 10) and VM's backup jobs are configured for volumes for storage snapshots which are in synchronous replication. I tried to SanRescanSnapshotsParseFS = 0 option but didn't help.

IT SEEMS THAT NIMBLE INTEGRATION IS NOT SUPPORTED WITH VEEAM WHEN NIMBLE STORAGE VOLUMES ARE IN SYNCHRONOUS REPLICATION. CAN ANYONE CONFIRM THIS?

foggy
Veeam Software
Posts: 20872
Liked: 2048 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy »

As you can read above, this configuration is not supported indeed.

Coldfirex
Enthusiast
Posts: 69
Liked: 12 times
Joined: May 22, 2015 1:41 pm
Full Name: Alan Shearer
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Coldfirex » 1 person likes this post

Weve been dealing with this for months and just came across the this thread.
Will v11 support it finally?

Gostev
SVP, Product Management
Posts: 30020
Liked: 5934 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Gostev »

Unfortunately, no. As far as I know, the main reason is huge delays in getting the corresponding hardware from HPE for our R&D lab, which were in turn caused by COVID restrictions. I believe this has been mostly resolved, and last I heard from the corresponding PM team, they were planning to work on this feature for v12.

foggy
Veeam Software
Posts: 20872
Liked: 2048 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by foggy » 6 people like this post

That's correct, this is on our priority list for v12.

cluescher
Lurker
Posts: 1
Liked: 1 time
Joined: Apr 10, 2020 4:18 pm
Full Name: Carl Luescher
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by cluescher » 1 person likes this post

+1 for this syncrep support implementation

Andreas Neufert
VP, Product Management
Posts: 6161
Liked: 1280 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

Post by Andreas Neufert » 2 people like this post

Hi Carl, thanks for the feedback.

Hi everyone. If you have nothing material to add to the discussion, please just use the Like button instead of posting +1. This will help to not SPAM the forum and will keep it clean and easily consumable. We had some threads with tons of +1s and people are missing the important information in between them.

Post Reply

Who is online

Users browsing this forum: AlexHeylin, cosmik, Google [Bot], rennerstefan and 36 guests