HPE Nimble -> Cannot find LUN with SCSI ID...

[colohost]

Hey all, we're attempting to convert a functional VB&R 11 (20210507) Enterprise Plus install over to using both linux proxies and storage snapshots, from what is currently Windows proxies doing vCenter/vSphere snapshots. The underlying storage is HPE Nimble and VB&R has been connected to it. The proxies are Ubuntu 20LTS with Nimble Connection Manager software installed for optimal multipath support. The transport is iSCSI. After getting our first backup test job going, it had success and failures. There were three Nimble LUNs where VM's existed, all three had snapshots successfully taken on the storage side, VBR added snapshot access rules to the array for the proxies to be able to read them, and then it was able to back up the VM's on two of the three LUNs before completing and removing the snapshots.

One of the three LUNs had a failure to back up all its VMs with "Processing servername Error: Cannot find LUN with SCSI ID xyz"

I was able to confirm on the linux side via both Nimble Connection Manager and the native multipath command that the snapshot in question was visible to the OS. I was also able to confirm with parted that the LUN in question was visible at least from a partitioning standpoint; so it was able to read the actual disk and spit out the partition table. I'm not able to attempt mounting the snapshots because the version of VMFS6 we're running appears incompatible with the current version of vmfs6-tools, but given the partition can be read, I don't think there's a connectivity issue to those snaps. The other two successful backup LUNs are the same version of VMFS, same target array, same proxy. Zero bytes were transferred, so it's not an intermittent issue.

I have a ticket going but was hoping maybe someone else has run into this and would have ideas.

[HannesK]

Hello,
can you please post the Veeam case number you mentioned to see what's going on and to allow future readers to refer to your solution.

Did you try it out without the additional Nimble software?

Thanks,
Hannes

[colohost]

Ticket 04833843

I've just deployed a CentOS 7 system to test an old stable OS, no multipath, no Nimble Connection Manager, same failure on just the one LUN.

I'll report back if we find a solution. Haven't heard from Veeam in two days so not sure what's going on there.

[Gostev]

Haven't heard from Veeam in two days so not sure what's going on there.

The weekend is going

[colohost]

It's not a huge deal since our vCenter-based snaps are still working, and, I understand it's the weekend, however I'm on an Enterprise Plus plan with production support and Veeam's support policy suggests the business hours for my plan are 24x7x365. If the person working the case knew on Friday that they were going off shift and would not respond to me for three days, or transfer to someone else, it would have been better to have set that expectation.

[Gostev]

From that perspective, I agree. @m_zolkin FYI

What I was rather trying to say with my comment, is that when we're talking about environment/storage specific issues like "1 particular LUN can't be seen", then there's usually little our support can do on their own without engaging the actual storage vendor, which is largely impossible to do for support cases that were open in the Friday afternoon...

[jimnoco]

Did you ever find a fix for this? I'm running into the same issue on a brand new Nimble array.

[Andreas Neufert]

Customer is working with HPE on this is I think the last entry if I understand the message flow correctly of the support case.

[Fegume]

Hello I have the same problem with a HPE 3PAR Storage.

Do you have any news?

thanks!

[rennerstefan]

Hi,

as this thread is already some month old, can you provide some details on what exact issue you are facing? and what the environment looks like?
Most of the case there is either a logical error in the design or a issue with physical connections. Also you ask for 3par while the thread is nimble and there is differences in both.
Also, if not done yet, please also open a case at Veeam support and share the Case id.

thanks

[colohost]

Just wanted to follow up on this; HPE support was completely useless and we gave up rather than continue to waste staff time on it. The LUN in question had all its VM's migrated to a different one that didn't have the issue and then we deleted it.

[Andreas Neufert]

Not good, but at least there is a workaround. Thanks for sharing.

[ChrisGundry]

Has anyone had this and actually fixed it before? I have found a few references to it online, but no real fixes...

We are doing backups/replicas using storage snapshots from Nimble arrays, which has been setup and working for years and there are no changes in the last week. But in the early hours of this morning all our replica jobs started failing intermittently. I re-ran a few of the jobs and some worked, others still had an error.

I ran a backup job to check and we get the same error on the backup jobs as well. So all backups/replicas impacted it seems.

We had this issue once before, maybe 2 years ago, but were unable to log it due to an issue with the Veeam portal at the time. The issue happened again today out of the blue. Rebooting our Veeam servers resolved the issue last time, so we rebooted again this time and once again the issue is resolved. I had raised a sev 2 support case, but it was critical for us to get things working again ASAP, hence the reboot.

But I am curious if anyone can share any experience with this error?

Thanks

[foggy]

Hi Chris, please see above for a similar issue and I'd appreciate it if you share the support case ID. Thanks!

[ChrisGundry]

Yes, I had seen this post, but no answers.

My case is #05499971

[MartinO]

Was there any resolution to the above cases? We're experiencing the same issue now, I've opened a case with Veeam and HPE. HPE are on standby until Veeam confirm that there's nothing on there side since it appears to be throwing different errors based on which proxy we use, even though both are configured the same, and both can backup via DirectSAN access just fine.

Veeam #02913615

[rennerstefan]

Hi Martin,
can you briefly share what issue you are facing? As I mentioned above most times the issues are not the same so I want to make sure we understand what you see.
Thanks also for providing the case ID.
Thanks

[ChrisGundry]

We have not seen the issue again since we rebooted our Veeam proxies. But we have only seen the issue twice in around 2.5 years so... Veeam couldn't get anywhere with it, took days to get a reply and by that point we had to reboot the servers to get the backups/replicas working again. After that they said they couldn't do anything without being able to replicate the problem.

[MartinO]

Hi Martin,
can you briefly share what issue you are facing? As I mentioned above most times the issues are not the same so I want to make sure we understand what you see.
Thanks also for providing the case ID.
Thanks

We have two proxies, both are used successfully for DirectSAN access when backing up VMs with regular snapshots. When forced to use storage snapshots, Veeam creates the storage snapshot OK, and it can be seen on the Nimble, but then Veeam fails to back up from it.

Using backup proxy VMware Backup Proxy for retrieving Hard disk 1 data from storage snapshot on nimble***
Error: Cannot find LUN with SCSI ID d7ae712ac4f58a476c9ce9003f0dd848

If I force select the second proxy, the job immediately fails with the error:
Task failed. Error: Cannot find suitable proxy for retrieving data from storage snapshot, and the option to failover to backup from VM snapshot is disabled

I've logged it with Nimble support, who've said they're happy to have a three-way call between all of us to get this sorted a bit faster, so I just need to book that in with the Veeam support agent once I've got some time.