Host-based backup of VMware vSphere VMs.
Post Reply
nunciate
Expert
Posts: 248
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Dell PowerStore Storage Plug-in Failure

Post by nunciate »

Well, we bought new Dell PowerStore 3000T running the 3.2 software and I thought I'd give the storage plug-in a try.
I had tried the old Dell Compellent plug-in previously and it worked great. My only issue is we were running very slim margins and our volumes were quite full. This is the reason for the upgrades.

Anyway, I installed this plug-in in our DR location. I ran a couple of test jobs. I saw it create temp snapshots on the VM in the job and then it placed snapshots on the volumes. The VM snapshots were removed and the backups ran successfully. I thought great this is going to work. I let all my backups run overnight in DR. When I arrive the next morning 2 of my 6 VMware hosts were dead. CPU Maxed out and disconnected from vCenter. All the VM on those hosts were also disconnected. It took about half a day to unwind everything. So I removed the plug-in and we are back to normal but that was seriously concerning.

Does anyone else see this kind of issue? I didn't open a ticket and I didn't pull logs as I just wanted to get everything up and running again. Just thought it was pretty crazy that it effectively took down my DR environment.
ronnmartin61
Veeam Software
Posts: 436
Liked: 128 times
Joined: Mar 07, 2016 3:55 pm
Full Name: Ronn Martin
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by ronnmartin61 »

Just to be sure I'm reading this correctly you installed the SC Series plugin for use on PowerStore? Guessing the answer is "no" and you used the PowerStore plugin instead. In any case the current PowerStore plugin only supports up to PowerStore OS 2.x (https://helpcenter.veeam.com/docs/backu ... ml?ver=120) however the new plugin appears to be close to release see vmware-vsphere-f24/dell-powerstore-stor ... 28-30.html
Chrispyyy
Enthusiast
Posts: 39
Liked: 9 times
Joined: Jan 04, 2023 6:28 pm
Full Name: Chrispyyy
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by Chrispyyy »

The PowerStore plug-in causes ESXI hosts to crash as it constantly retries incompatible commands without any sort of failure mechanism. Because of this, it’s not officially compatible with 3.X upwards - which after nearly eight months of being released is disappointing.

As per @foggy’s comment in my PowerStore thread, it has now entered QA and should be released soon. You never know, he may be able to shed a light on any potential release date.
nunciate
Expert
Posts: 248
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by nunciate »

Ah well that makes sense. I thought i read 3.2 was supported. I will wait before retrying. And yes I installed the PowerStore plugin not the SC.
nunciate
Expert
Posts: 248
Liked: 39 times
Joined: May 21, 2013 9:08 pm
Full Name: Alan Wells
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by nunciate »

Well, I installed the new PowerStore plugin v2.0.5 because I thought it said it support the 3.2 OS. I am now on the PowerStore 3.5.01 version and after putting in the plug-in many of my backups ran just fine but one of my hosts became unresponsive. I can only assume because of something with the plug-in as it is the only thing that changed. Anyway, I pulled the SAN out of Veeam and won't try this again. Too many issues trying to do storage snapshots in my opinion. I still have several VM reporting issues with the SAN now so I'll be spending my day cleaning up this mess.
rennerstefan
Veeam Software
Posts: 627
Liked: 146 times
Joined: Jan 22, 2015 2:39 pm
Full Name: Stefan Renner
Location: Germany
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by rennerstefan »

Sorry to hear.
Honestly we, and me personally, would be very interested in the issues you see as of course our main reason for plug-ins is to make your live easier and the backup process more streamlined.
Feel free to open a ticket at Veeam support and send me the support case ID here.
Thanks
Stefan Renner

Veeam PMA
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by foggy »

Yes, we'd definitely like to be able to troubleshoot the issues you're seeing as the new plug-in does support all the latest PowerStore OS versions including 3.6. By the 'one of my hosts became unresponsive' do you mean the ESXi host? the proxy? Anything else? What do you mean by 'unresponsive'? Thanks!
Chrispyyy
Enthusiast
Posts: 39
Liked: 9 times
Joined: Jan 04, 2023 6:28 pm
Full Name: Chrispyyy
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by Chrispyyy »

nunciate wrote: Dec 13, 2023 2:18 pm Well, I installed the new PowerStore plugin v2.0.5 because I thought it said it support the 3.2 OS. I am now on the PowerStore 3.5.01 version and after putting in the plug-in many of my backups ran just fine but one of my hosts became unresponsive. I can only assume because of something with the plug-in as it is the only thing that changed. Anyway, I pulled the SAN out of Veeam and won't try this again. Too many issues trying to do storage snapshots in my opinion. I still have several VM reporting issues with the SAN now so I'll be spending my day cleaning up this mess.
Running recommended code OS 3.5.0.2 on Veeam v12.1 with v2.0.5, when carrying out a restore - during dismount it’s causing an APD error and thus freezing/crashing the host. Is this what you’re experiencing as well?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by foggy »

Are you probably talking about the issue caused by a host group configuration that was discussed in this thread?
Chrispyyy
Enthusiast
Posts: 39
Liked: 9 times
Joined: Jan 04, 2023 6:28 pm
Full Name: Chrispyyy
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by Chrispyyy »

Hi Foggy,
No, these are not in host groups - learnt that lesson the hard way. :)
Chrispyyy
Enthusiast
Posts: 39
Liked: 9 times
Joined: Jan 04, 2023 6:28 pm
Full Name: Chrispyyy
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by Chrispyyy » 1 person likes this post

foggy wrote: Dec 15, 2023 12:04 am Are you probably talking about the issue caused by a host group configuration that was discussed in this thread?
Raised with support w/ logs, support ticket #02844824.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by foggy »

Not a valid support case number - please verify. This one may be your support contract ID.
Chrispyyy
Enthusiast
Posts: 39
Liked: 9 times
Joined: Jan 04, 2023 6:28 pm
Full Name: Chrispyyy
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by Chrispyyy » 1 person likes this post

Ah, yes - sorry. Support #07052844.
Chrispyyy
Enthusiast
Posts: 39
Liked: 9 times
Joined: Jan 04, 2023 6:28 pm
Full Name: Chrispyyy
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by Chrispyyy »

Update: I’ve managed to recreate this issue consistently on an isolated host for testing, got a remote session with Support on Weds.

I’ll have a play with it tomorrow to see if I can see anything in the meantime. As soon as the dismount happens, the storage just drops completely with APD frozen until forced reboot.

Fingers crossed they can get to the bottom of it/gather relevant logs to raise with those who need to see it. Sucks as I was really enjoying the benefits of storage snapshots on my PowerStore.
Chrispyyy
Enthusiast
Posts: 39
Liked: 9 times
Joined: Jan 04, 2023 6:28 pm
Full Name: Chrispyyy
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by Chrispyyy »

Been playing around, installed the old v1.0.5 plug-in. This is able to unmount/detach without causing APD errors on the host.
In conclusion, this appears to be an issue with v2.0.5 plug-in. I can't provide definitive root cause as I upgraded the PowerStoreOS, Veeam and plug-in at the same time (in that order).

Environment details for validation:
Dell PowerStore 500T running PowerStoreOS 3.5.0.2
Veeam Backup and Replication v12.1

Thanks.
rennerstefan
Veeam Software
Posts: 627
Liked: 146 times
Joined: Jan 22, 2015 2:39 pm
Full Name: Stefan Renner
Location: Germany
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by rennerstefan »

Thanks for all your detailed feedback and also for opening the case.
I can see that support is already actively working with you on the ticket for us to get more details.
As soon as there is any explanation for the behaviour or any news we will share it.
Stefan Renner

Veeam PMA
Chrispyyy
Enthusiast
Posts: 39
Liked: 9 times
Joined: Jan 04, 2023 6:28 pm
Full Name: Chrispyyy
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by Chrispyyy »

Just to update, I was advised you were able to recreate the issue in the lab - but support said to contact Dell.

Upon contacting Dell, they said they can’t do anything (I’m guessing a support answer, rather than dev) and referred me back to you guys.

Do you have anyone you can reach out to at Dell about this issue in the development/storage division?

I feel like I’m in a ping-pong situation again, where you keep batting the problem between support teams - so a direct contact would be much better IMO.

Thanks,
Chris
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by foggy »

Hi Chris, I will review the case and reach out to Dell R&D if needed, thanks for the heads up.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by foggy » 1 person likes this post

Hi Chris, I apologize, it was a bad case handling on our side. It should have been escalated internally to confirm the issue is indeed on the plug-in side - so we could contact the plug-in development team directly then, instead of sending you to Dell support. May I ask you to open a new case to be processed properly? If you share the case ID here or via PM with me, I will be able to route it right away. Thanks and sorry for the inconvenience!
Chrispyyy
Enthusiast
Posts: 39
Liked: 9 times
Joined: Jan 04, 2023 6:28 pm
Full Name: Chrispyyy
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by Chrispyyy » 1 person likes this post

Hi Alex/Foggy,

No worries, these things happen.
I’m out of office until Monday, I’ll raise a duplicate ticket with all relevant information at the earliest opportunity.

Really appreciate you leading on this.

Thanks,
Chris
adogramacian
Lurker
Posts: 1
Liked: never
Joined: Feb 06, 2024 9:46 pm
Full Name: Arthur Dogramacian
Contact:

Re: Dell PowerStore Storage Plug-in Failure

Post by adogramacian »

Hi folks, has anyone received an update on this issue? We will be implementing a PowerStore 1200T in a month or 2, and would like to use the PowerStore snapshots feature.
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 22 guests