Dell PowerStore Storage Plug-in Failure

[nunciate]

Well, we bought new Dell PowerStore 3000T running the 3.2 software and I thought I'd give the storage plug-in a try.
I had tried the old Dell Compellent plug-in previously and it worked great. My only issue is we were running very slim margins and our volumes were quite full. This is the reason for the upgrades.

Anyway, I installed this plug-in in our DR location. I ran a couple of test jobs. I saw it create temp snapshots on the VM in the job and then it placed snapshots on the volumes. The VM snapshots were removed and the backups ran successfully. I thought great this is going to work. I let all my backups run overnight in DR. When I arrive the next morning 2 of my 6 VMware hosts were dead. CPU Maxed out and disconnected from vCenter. All the VM on those hosts were also disconnected. It took about half a day to unwind everything. So I removed the plug-in and we are back to normal but that was seriously concerning.

Does anyone else see this kind of issue? I didn't open a ticket and I didn't pull logs as I just wanted to get everything up and running again. Just thought it was pretty crazy that it effectively took down my DR environment.

[ronnmartin61]

Just to be sure I'm reading this correctly you installed the SC Series plugin for use on PowerStore? Guessing the answer is "no" and you used the PowerStore plugin instead. In any case the current PowerStore plugin only supports up to PowerStore OS 2.x (https://helpcenter.veeam.com/docs/backu ... ml?ver=120) however the new plugin appears to be close to release see vmware-vsphere-f24/dell-powerstore-stor ... 28-30.html

[Chrispyyy]

The PowerStore plug-in causes ESXI hosts to crash as it constantly retries incompatible commands without any sort of failure mechanism. Because of this, it’s not officially compatible with 3.X upwards - which after nearly eight months of being released is disappointing.

As per @foggy’s comment in my PowerStore thread, it has now entered QA and should be released soon. You never know, he may be able to shed a light on any potential release date.

[nunciate]

Ah well that makes sense. I thought i read 3.2 was supported. I will wait before retrying. And yes I installed the PowerStore plugin not the SC.

[nunciate]

Well, I installed the new PowerStore plugin v2.0.5 because I thought it said it support the 3.2 OS. I am now on the PowerStore 3.5.01 version and after putting in the plug-in many of my backups ran just fine but one of my hosts became unresponsive. I can only assume because of something with the plug-in as it is the only thing that changed. Anyway, I pulled the SAN out of Veeam and won't try this again. Too many issues trying to do storage snapshots in my opinion. I still have several VM reporting issues with the SAN now so I'll be spending my day cleaning up this mess.

[rennerstefan]

Sorry to hear.
Honestly we, and me personally, would be very interested in the issues you see as of course our main reason for plug-ins is to make your live easier and the backup process more streamlined.
Feel free to open a ticket at Veeam support and send me the support case ID here.
Thanks

[foggy]

Yes, we'd definitely like to be able to troubleshoot the issues you're seeing as the new plug-in does support all the latest PowerStore OS versions including 3.6. By the 'one of my hosts became unresponsive' do you mean the ESXi host? the proxy? Anything else? What do you mean by 'unresponsive'? Thanks!

[Chrispyyy]

Well, I installed the new PowerStore plugin v2.0.5 because I thought it said it support the 3.2 OS. I am now on the PowerStore 3.5.01 version and after putting in the plug-in many of my backups ran just fine but one of my hosts became unresponsive. I can only assume because of something with the plug-in as it is the only thing that changed. Anyway, I pulled the SAN out of Veeam and won't try this again. Too many issues trying to do storage snapshots in my opinion. I still have several VM reporting issues with the SAN now so I'll be spending my day cleaning up this mess.

Running recommended code OS 3.5.0.2 on Veeam v12.1 with v2.0.5, when carrying out a restore - during dismount it’s causing an APD error and thus freezing/crashing the host. Is this what you’re experiencing as well?

[foggy]

Are you probably talking about the issue caused by a host group configuration that was discussed in this thread?

[Chrispyyy]

Hi Foggy,
No, these are not in host groups - learnt that lesson the hard way.

[Chrispyyy]

Are you probably talking about the issue caused by a host group configuration that was discussed in this thread?

Raised with support w/ logs, support ticket #02844824.

[foggy]

Not a valid support case number - please verify. This one may be your support contract ID.

[Chrispyyy]

Ah, yes - sorry. Support #07052844.

[Chrispyyy]

Update: I’ve managed to recreate this issue consistently on an isolated host for testing, got a remote session with Support on Weds.

I’ll have a play with it tomorrow to see if I can see anything in the meantime. As soon as the dismount happens, the storage just drops completely with APD frozen until forced reboot.

Fingers crossed they can get to the bottom of it/gather relevant logs to raise with those who need to see it. Sucks as I was really enjoying the benefits of storage snapshots on my PowerStore.

[Chrispyyy]

Been playing around, installed the old v1.0.5 plug-in. This is able to unmount/detach without causing APD errors on the host.
In conclusion, this appears to be an issue with v2.0.5 plug-in. I can't provide definitive root cause as I upgraded the PowerStoreOS, Veeam and plug-in at the same time (in that order).

Environment details for validation:
Dell PowerStore 500T running PowerStoreOS 3.5.0.2
Veeam Backup and Replication v12.1

Thanks.

[rennerstefan]

Thanks for all your detailed feedback and also for opening the case.
I can see that support is already actively working with you on the ticket for us to get more details.
As soon as there is any explanation for the behaviour or any news we will share it.

[Chrispyyy]

Just to update, I was advised you were able to recreate the issue in the lab - but support said to contact Dell.

Upon contacting Dell, they said they can’t do anything (I’m guessing a support answer, rather than dev) and referred me back to you guys.

Do you have anyone you can reach out to at Dell about this issue in the development/storage division?

I feel like I’m in a ping-pong situation again, where you keep batting the problem between support teams - so a direct contact would be much better IMO.

Thanks,
Chris

[foggy]

Hi Chris, I will review the case and reach out to Dell R&D if needed, thanks for the heads up.

[foggy]

Hi Chris, I apologize, it was a bad case handling on our side. It should have been escalated internally to confirm the issue is indeed on the plug-in side - so we could contact the plug-in development team directly then, instead of sending you to Dell support. May I ask you to open a new case to be processed properly? If you share the case ID here or via PM with me, I will be able to route it right away. Thanks and sorry for the inconvenience!

[Chrispyyy]

Hi Alex/Foggy,

No worries, these things happen.
I’m out of office until Monday, I’ll raise a duplicate ticket with all relevant information at the earliest opportunity.

Really appreciate you leading on this.

Thanks,
Chris

[adogramacian]

Hi folks, has anyone received an update on this issue? We will be implementing a PowerStore 1200T in a month or 2, and would like to use the PowerStore snapshots feature.