HPE Nimble & Veeam Storage Discovery Jobs causing vCenter storage disconnects

[SiStone]

Posting this as the suggestion of Veeam Support following a lengthy investigation into an issue we have been seeing since the install of a pair of HPE Nimble Storage arrays (case 03674597)

So, it goes something like this. We have a pair of brand new HPE Nimble storage arrays. We were lucky enough to get them just as NimbleOS v5.1 came out with synchronous replication. So, we took advantage of this and had the two arrays configured as a syncrep pair.

The issues started when we installed Veeam and set up the Nimble integration. We immediately started getting 'storage disconnection' notifications raised by vCenter. After a bit of detective work, we spotted that the notificaitons coincided with the scheduled storage discovery jobs beind triggered from Veeam. Around the same time we had a number of times when the management services on one or more of our ESXi hosts would crash, resulting in the host bein unmanageable from within vCenter. The host itself (and the VM's on that host) were still runiing, but we were unable to manage them until the management services were restarted from SSH.

Tickets were then logged with vmWare and Veeam to investigate. vmWare stated in no uncertain terms that the host management service failure was a direct result (and an apparent known issue in ESXi) of the persistent storage disconnects. The storage disconnects appeared to be an issue within Veeam, tricky bit was determining what exactly the problem is.

After a lengthy investigation and several remote sessions, it turns out that the Nimble syncrep functionality doesn't play well with Veeam. I assume it is something to do with the arrays presenting multiple volumes with the same LUN ID, and Veeam scanning the same volume on multiple arrays which is causing vCenter to get screwed up. vCenter then sees what it thinks are LUNs getting disconnected, which is wat it is reporting.

As a result of the work with the Veaam tech guys, we have put a temporary workaround in place which kills off the scheduled storage discovery jobs, and I think it also changes the way that manual scans are working, and this has stopped the storage disconnects, which in turn has stopped the ESXi management services from crashing. This workaround is not without it's drawbacks. It breaks the storage snapshot chain for a single VM (so, if a VM is removed from a volume in a storage snapshot, it is removed from all snapshots in the chain). It also means we need to do manual scans when we add new VM's. Not massively onerous, but inconvenient to say the least.

My concern is that it turns out that this issue (and full Nimble Syncrep functionality) will not be supported in Veeam v9, and there are also no current plans for it to be fully supported in the upcoming v10 either. I suspect strongly that most customers taking Nimble arrays on will want to use syncrep, especially as with NimbleOS v5.3 (which is now available) this becomes generally available. For the industry leading backup and replication product to advertise integration with these arrays, but for it not to function properly, seems strange to me.

Has anyone else come across problems like this? Thoughts on the v10 roadmap?

[foggy]

Hi Simon, HPE Nimble synchronous replication is not currently supported by Veeam B&R but we're looking into fully supporting it in our future releases. Thanks for sharing your findings.

[Gostev]

For the industry leading backup and replication product to advertise integration with these arrays, but for it not to function properly, seems strange to me.

Are you saying we should not have been advertising our integration with Nimble arrays for the past 3 years, because we should have somehow predicted that in late 2019, a new Nimble array feature will be released that our current integration will not be compatible with out of the box? This makes no sense.

Not to mention that the majority of our customers don't even use asynchronous replication, which we do fully support since day 1. So, I beg to disagree with your prediction that "most customers taking Nimble arrays on will want to use syncrep".

Having said that, as Foggy already mentioned, this is one of the features we're looking to support in future releases.

[Andreas Neufert]

And we are working closely with the Nimble team since the beta of this feature to integrate it in our product. For technical reasons related to Nimble themself we are not there yet.

[SiStone]

Are you saying we should not have been advertising our integration with Nimble arrays for the past 3 years, because we should have somehow predicted that in late 2019, a new Nimble array feature will be released that our current integration will not be compatible with out of the box? This makes no sense.

No, I am not saying that at all, and I am pretty sure my post didn't say that! I have worked with Veeam and Nimble arrays since before the integration was first launched, and the integration was a massive leap forward giving all sorts of benefits. What I was surprised about was that the support and development teams seemed entirely unaware of the new functionality in the arrays. I would have expected (rightly or wrongly) that Nimble (HPE) and Veeam would work togehter to support new functionality, or at the very least ensure new array functionality does not cause the kind of issues we have seen. Does this mean that Veem are not given advance notification (or even sight of) of upcoming software releases by HPE Nimble?

I do however, disagree with your statement on Syncrep. I fully expect the adoption of syncrep functionality in Nimble arrays, where their hardware supports it, to be high. This is not only based on the fact that it has been one of the most requested features since I started using Nimble arrays around 4 years ago, but also it was the one area they lagged behind other comparable offerings such as 3PAR.

Again, I was only posting this at the request of the Veeam support team in an attempt to bring it into focus in hope that it would be better supported soon.

[Andreas Neufert]

As I said we are fully aware from R&D perspective about this since HPE gave us ahead notice. Since then we are working with HPE on this, but there are technical things that hinder us from implementation. I can not comment about details based on NDA.

[ftepe]

Hi,

this post was hard to find but I'm dealing with exactly the same issue! Are there any updates as so far since the last two month?

[foggy]

Hi Florian, no major updates so far.

[ftepe]

Thanks for the fast response!

[ftepe]

Hi,

any news after another month? Will it be supported in V10?

[Andreas Neufert]

Hi Florian,

There will be no change for this in v10. As I said above we are working with HPE on it, basically defining the priorities with them and working based on their input on overall HPE feature. Potentially it makes sense as well to raise awareness at HPE for this feature request through your HPE sales rep.

Based on our communication rules given by our top management, we can not comment about upcoming features and timelines until a feature is in the beta stage. We will update this thread here when/if we reach that stage.

Andreas

[andrie]

We had similar issues with Veeam and Nimble storage snapshots without using the storage integration.
Veeam brings the storage snapshots online to scan them and the default setting on Nimble for ESX is to present the volumes and snapshots. This causes the storage disconnects when the snapshots are taken offline again.
The solution for us was to change the volume presentation on the Nimble systems from "Volume and Snapshots" to "Volume only".
This prevents the ESX hosts from seeing the storage snapshots during the Veeam scanning.

[Andreas Neufert]

Thanks for sharing. I have asked the doucmentation team to add this there.

[bernd.m]

I had the described issues at two customer sites too, and will confirm, that the solution from @andrie, to switch the ACL from "Volume & Snapshots" to "Volume Only" for the involved ESXi hosts is working fine.
I also want to warn you, that this setting, despite the Nimble GUI gives no warning at all, ist NOT an online procedure!
All IO to the volume hast to be stopped to edit the ACL from "Volume & Snapshots" to "Volume Only"!

[mwhite]

Just to ping this, the Nimble Veeam integration guides states the following;

"A VMware datastore, which is targeted by Veeam, and backed by a Nimble volume will require that the volume's ACL is set to Volume Only. This is to prevent any connections being made between VMware and online snapshots created by Veeam. Connections to online snapshots (made by Veeam) can result in VMware showing dropped paths and the possibility of the datastore going into PDL."

I hope though that the synchronous replication support is not too far away

[stuartmacgreen]

+1 for this syncrep support implementation.

Thanks

[foggy]

Thanks for that, I will make sure we have a similar note in the documentation on our side as well.

[stuartmacgreen]

Can you give any insight/detail to the 'workaround'? (reg key?)
Curious... did you exclude any Nimble Volumes within the Nimble system listed in Veeam Storage Infrastructure? (ie the Downstream Sync Rep volumes.)

Thanks

Posting this as the suggestion of Veeam Support following a lengthy investigation into an issue we have been seeing since the install of a pair of HPE Nimble Storage arrays (case 03674597)

[foggy]

Hi Stuart, according to the case notes, the registry value that Simon is talking about is SanRescanSnapshotsParseFS = 0. Excluding the volume copies from the rescan did not help in his case.

[Hnns]

Any updates on this from Veeam? It seems this has been a known issue for over a year now..

[foggy]

No updates as of now, this configuration is not currently supported.

[Hnns]

I think it's been well established that it's not currently supported, the question is are there any plans on actually making this supported, yes or no, and when?

[foggy]

Yes, we plan to support it, but I cannot provide any particular ETA at the moment. We're moving towards that direction though.

[theman]

#04508534
I'm facing similar issue where nimble storage is integrated with Veeam (B& R -Version 10) and VM's backup jobs are configured for volumes for storage snapshots which are in synchronous replication. I tried to SanRescanSnapshotsParseFS = 0 option but didn't help.

IT SEEMS THAT NIMBLE INTEGRATION IS NOT SUPPORTED WITH VEEAM WHEN NIMBLE STORAGE VOLUMES ARE IN SYNCHRONOUS REPLICATION. CAN ANYONE CONFIRM THIS?

[foggy]

As you can read above, this configuration is not supported indeed.

[Coldfirex]

Weve been dealing with this for months and just came across the this thread.
Will v11 support it finally?

[Gostev]

Unfortunately, no. As far as I know, the main reason is huge delays in getting the corresponding hardware from HPE for our R&D lab, which were in turn caused by COVID restrictions. I believe this has been mostly resolved, and last I heard from the corresponding PM team, they were planning to work on this feature for v12.

[foggy]

That's correct, this is on our priority list for v12.

[cluescher]

+1 for this syncrep support implementation

[Andreas Neufert]

Hi Carl, thanks for the feedback.

Hi everyone. If you have nothing material to add to the discussion, please just use the Like button instead of posting +1. This will help to not SPAM the forum and will keep it clean and easily consumable. We had some threads with tons of +1s and people are missing the important information in between them.