-
- Veteran
- Posts: 251
- Liked: 39 times
- Joined: May 21, 2013 9:08 pm
- Full Name: Alan Wells
- Contact:
Dell PowerStore Storage Plug-in Failure
Well, we bought new Dell PowerStore 3000T running the 3.2 software and I thought I'd give the storage plug-in a try.
I had tried the old Dell Compellent plug-in previously and it worked great. My only issue is we were running very slim margins and our volumes were quite full. This is the reason for the upgrades.
Anyway, I installed this plug-in in our DR location. I ran a couple of test jobs. I saw it create temp snapshots on the VM in the job and then it placed snapshots on the volumes. The VM snapshots were removed and the backups ran successfully. I thought great this is going to work. I let all my backups run overnight in DR. When I arrive the next morning 2 of my 6 VMware hosts were dead. CPU Maxed out and disconnected from vCenter. All the VM on those hosts were also disconnected. It took about half a day to unwind everything. So I removed the plug-in and we are back to normal but that was seriously concerning.
Does anyone else see this kind of issue? I didn't open a ticket and I didn't pull logs as I just wanted to get everything up and running again. Just thought it was pretty crazy that it effectively took down my DR environment.
I had tried the old Dell Compellent plug-in previously and it worked great. My only issue is we were running very slim margins and our volumes were quite full. This is the reason for the upgrades.
Anyway, I installed this plug-in in our DR location. I ran a couple of test jobs. I saw it create temp snapshots on the VM in the job and then it placed snapshots on the volumes. The VM snapshots were removed and the backups ran successfully. I thought great this is going to work. I let all my backups run overnight in DR. When I arrive the next morning 2 of my 6 VMware hosts were dead. CPU Maxed out and disconnected from vCenter. All the VM on those hosts were also disconnected. It took about half a day to unwind everything. So I removed the plug-in and we are back to normal but that was seriously concerning.
Does anyone else see this kind of issue? I didn't open a ticket and I didn't pull logs as I just wanted to get everything up and running again. Just thought it was pretty crazy that it effectively took down my DR environment.
-
- Veeam Software
- Posts: 478
- Liked: 156 times
- Joined: Mar 07, 2016 3:55 pm
- Full Name: Ronn Martin
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Just to be sure I'm reading this correctly you installed the SC Series plugin for use on PowerStore? Guessing the answer is "no" and you used the PowerStore plugin instead. In any case the current PowerStore plugin only supports up to PowerStore OS 2.x (https://helpcenter.veeam.com/docs/backu ... ml?ver=120) however the new plugin appears to be close to release see vmware-vsphere-f24/dell-powerstore-stor ... 28-30.html
-
- Enthusiast
- Posts: 41
- Liked: 11 times
- Joined: Jan 04, 2023 6:28 pm
- Full Name: Chrispyyy
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
The PowerStore plug-in causes ESXI hosts to crash as it constantly retries incompatible commands without any sort of failure mechanism. Because of this, it’s not officially compatible with 3.X upwards - which after nearly eight months of being released is disappointing.
As per @foggy’s comment in my PowerStore thread, it has now entered QA and should be released soon. You never know, he may be able to shed a light on any potential release date.
As per @foggy’s comment in my PowerStore thread, it has now entered QA and should be released soon. You never know, he may be able to shed a light on any potential release date.
-
- Veteran
- Posts: 251
- Liked: 39 times
- Joined: May 21, 2013 9:08 pm
- Full Name: Alan Wells
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Ah well that makes sense. I thought i read 3.2 was supported. I will wait before retrying. And yes I installed the PowerStore plugin not the SC.
-
- Veteran
- Posts: 251
- Liked: 39 times
- Joined: May 21, 2013 9:08 pm
- Full Name: Alan Wells
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Well, I installed the new PowerStore plugin v2.0.5 because I thought it said it support the 3.2 OS. I am now on the PowerStore 3.5.01 version and after putting in the plug-in many of my backups ran just fine but one of my hosts became unresponsive. I can only assume because of something with the plug-in as it is the only thing that changed. Anyway, I pulled the SAN out of Veeam and won't try this again. Too many issues trying to do storage snapshots in my opinion. I still have several VM reporting issues with the SAN now so I'll be spending my day cleaning up this mess.
-
- Veeam Software
- Posts: 676
- Liked: 149 times
- Joined: Jan 22, 2015 2:39 pm
- Full Name: Stefan Renner
- Location: Germany
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Sorry to hear.
Honestly we, and me personally, would be very interested in the issues you see as of course our main reason for plug-ins is to make your live easier and the backup process more streamlined.
Feel free to open a ticket at Veeam support and send me the support case ID here.
Thanks
Honestly we, and me personally, would be very interested in the issues you see as of course our main reason for plug-ins is to make your live easier and the backup process more streamlined.
Feel free to open a ticket at Veeam support and send me the support case ID here.
Thanks
Stefan Renner
Veeam PMA
Veeam PMA
-
- Veeam Software
- Posts: 21103
- Liked: 2127 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Yes, we'd definitely like to be able to troubleshoot the issues you're seeing as the new plug-in does support all the latest PowerStore OS versions including 3.6. By the 'one of my hosts became unresponsive' do you mean the ESXi host? the proxy? Anything else? What do you mean by 'unresponsive'? Thanks!
-
- Enthusiast
- Posts: 41
- Liked: 11 times
- Joined: Jan 04, 2023 6:28 pm
- Full Name: Chrispyyy
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Running recommended code OS 3.5.0.2 on Veeam v12.1 with v2.0.5, when carrying out a restore - during dismount it’s causing an APD error and thus freezing/crashing the host. Is this what you’re experiencing as well?nunciate wrote: ↑Dec 13, 2023 2:18 pm Well, I installed the new PowerStore plugin v2.0.5 because I thought it said it support the 3.2 OS. I am now on the PowerStore 3.5.01 version and after putting in the plug-in many of my backups ran just fine but one of my hosts became unresponsive. I can only assume because of something with the plug-in as it is the only thing that changed. Anyway, I pulled the SAN out of Veeam and won't try this again. Too many issues trying to do storage snapshots in my opinion. I still have several VM reporting issues with the SAN now so I'll be spending my day cleaning up this mess.
-
- Veeam Software
- Posts: 21103
- Liked: 2127 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Are you probably talking about the issue caused by a host group configuration that was discussed in this thread?
-
- Enthusiast
- Posts: 41
- Liked: 11 times
- Joined: Jan 04, 2023 6:28 pm
- Full Name: Chrispyyy
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Hi Foggy,
No, these are not in host groups - learnt that lesson the hard way.![Smile :)](https://forums.veeam.com/images/smilies/icon_smile.gif)
No, these are not in host groups - learnt that lesson the hard way.
![Smile :)](https://forums.veeam.com/images/smilies/icon_smile.gif)
-
- Enthusiast
- Posts: 41
- Liked: 11 times
- Joined: Jan 04, 2023 6:28 pm
- Full Name: Chrispyyy
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Raised with support w/ logs, support ticket #02844824.foggy wrote: ↑Dec 15, 2023 12:04 am Are you probably talking about the issue caused by a host group configuration that was discussed in this thread?
-
- Veeam Software
- Posts: 21103
- Liked: 2127 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Not a valid support case number - please verify. This one may be your support contract ID.
-
- Enthusiast
- Posts: 41
- Liked: 11 times
- Joined: Jan 04, 2023 6:28 pm
- Full Name: Chrispyyy
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Ah, yes - sorry. Support #07052844.
-
- Enthusiast
- Posts: 41
- Liked: 11 times
- Joined: Jan 04, 2023 6:28 pm
- Full Name: Chrispyyy
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Update: I’ve managed to recreate this issue consistently on an isolated host for testing, got a remote session with Support on Weds.
I’ll have a play with it tomorrow to see if I can see anything in the meantime. As soon as the dismount happens, the storage just drops completely with APD frozen until forced reboot.
Fingers crossed they can get to the bottom of it/gather relevant logs to raise with those who need to see it. Sucks as I was really enjoying the benefits of storage snapshots on my PowerStore.
I’ll have a play with it tomorrow to see if I can see anything in the meantime. As soon as the dismount happens, the storage just drops completely with APD frozen until forced reboot.
Fingers crossed they can get to the bottom of it/gather relevant logs to raise with those who need to see it. Sucks as I was really enjoying the benefits of storage snapshots on my PowerStore.
-
- Enthusiast
- Posts: 41
- Liked: 11 times
- Joined: Jan 04, 2023 6:28 pm
- Full Name: Chrispyyy
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Been playing around, installed the old v1.0.5 plug-in. This is able to unmount/detach without causing APD errors on the host.
In conclusion, this appears to be an issue with v2.0.5 plug-in. I can't provide definitive root cause as I upgraded the PowerStoreOS, Veeam and plug-in at the same time (in that order).
Environment details for validation:
Dell PowerStore 500T running PowerStoreOS 3.5.0.2
Veeam Backup and Replication v12.1
Thanks.
In conclusion, this appears to be an issue with v2.0.5 plug-in. I can't provide definitive root cause as I upgraded the PowerStoreOS, Veeam and plug-in at the same time (in that order).
Environment details for validation:
Dell PowerStore 500T running PowerStoreOS 3.5.0.2
Veeam Backup and Replication v12.1
Thanks.
-
- Veeam Software
- Posts: 676
- Liked: 149 times
- Joined: Jan 22, 2015 2:39 pm
- Full Name: Stefan Renner
- Location: Germany
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Thanks for all your detailed feedback and also for opening the case.
I can see that support is already actively working with you on the ticket for us to get more details.
As soon as there is any explanation for the behaviour or any news we will share it.
I can see that support is already actively working with you on the ticket for us to get more details.
As soon as there is any explanation for the behaviour or any news we will share it.
Stefan Renner
Veeam PMA
Veeam PMA
-
- Enthusiast
- Posts: 41
- Liked: 11 times
- Joined: Jan 04, 2023 6:28 pm
- Full Name: Chrispyyy
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Just to update, I was advised you were able to recreate the issue in the lab - but support said to contact Dell.
Upon contacting Dell, they said they can’t do anything (I’m guessing a support answer, rather than dev) and referred me back to you guys.
Do you have anyone you can reach out to at Dell about this issue in the development/storage division?
I feel like I’m in a ping-pong situation again, where you keep batting the problem between support teams - so a direct contact would be much better IMO.
Thanks,
Chris
Upon contacting Dell, they said they can’t do anything (I’m guessing a support answer, rather than dev) and referred me back to you guys.
Do you have anyone you can reach out to at Dell about this issue in the development/storage division?
I feel like I’m in a ping-pong situation again, where you keep batting the problem between support teams - so a direct contact would be much better IMO.
Thanks,
Chris
-
- Veeam Software
- Posts: 21103
- Liked: 2127 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Hi Chris, I will review the case and reach out to Dell R&D if needed, thanks for the heads up.
-
- Veeam Software
- Posts: 21103
- Liked: 2127 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Hi Chris, I apologize, it was a bad case handling on our side. It should have been escalated internally to confirm the issue is indeed on the plug-in side - so we could contact the plug-in development team directly then, instead of sending you to Dell support. May I ask you to open a new case to be processed properly? If you share the case ID here or via PM with me, I will be able to route it right away. Thanks and sorry for the inconvenience!
-
- Enthusiast
- Posts: 41
- Liked: 11 times
- Joined: Jan 04, 2023 6:28 pm
- Full Name: Chrispyyy
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Hi Alex/Foggy,
No worries, these things happen.
I’m out of office until Monday, I’ll raise a duplicate ticket with all relevant information at the earliest opportunity.
Really appreciate you leading on this.
Thanks,
Chris
No worries, these things happen.
I’m out of office until Monday, I’ll raise a duplicate ticket with all relevant information at the earliest opportunity.
Really appreciate you leading on this.
Thanks,
Chris
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Feb 06, 2024 9:46 pm
- Full Name: Arthur Dogramacian
- Contact:
Re: Dell PowerStore Storage Plug-in Failure
Hi folks, has anyone received an update on this issue? We will be implementing a PowerStore 1200T in a month or 2, and would like to use the PowerStore snapshots feature.
Who is online
Users browsing this forum: No registered users and 12 guests