Host-based backup of VMware vSphere VMs.
DigIT
Novice
Posts: 7
Liked: 2 times
Joined: Mar 14, 2023 2:40 pm
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by DigIT »

I'm on 12. Also, I realized today that I didn't have the network configuration correct -- the backup proxies didn't have network access to the storage, but they now do. I rescanned the storage and now I'm seeing "Successfully determined LUNs supported for backup from storage snapshots on storage..." Still falling back to VM snapshot instead of storage. Will update this thread with anything useful that support finds.
DigIT
Novice
Posts: 7
Liked: 2 times
Joined: Mar 14, 2023 2:40 pm
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by DigIT » 1 person likes this post

I was missing some additional configuration requirements -- such as MPIO on my Veeam Proxies. I've got backups running directly from storage snapshots right now!

Thanks so much, @foggy et al!
pbeebe.lp
Novice
Posts: 5
Liked: 1 time
Joined: Nov 09, 2015 3:52 pm
Full Name: Pete LP
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by pbeebe.lp »

I had tested the recent plugin against our Powerstore 3600T (v3.2.0.1) and created a test snapshot successfully followed by deleting it. Roughly 12 hours later all the ESXi servers in our environment started exhibiting high CPU host alerts and CPU wait alerts at the VM level. After some back and forth with Dell and Veeam they pointed at the Veeam Powerstore plugin as the culprit and referenced a Veeam Nimble article (https://helpcenter.veeam.com/archive/ba ... _name.html). Our Powerstore configuration utilizes a Host Group to consolidate all ESXi hosts and iSCSI initiators as well as Volume Groups to consolidate all iSCSI LUN datastore. The support reps suggested that due to using Powerstore groups the high CPU utilization was a result of the Veeam snapshot removal which caused an All Paths Down (APD) alert on all ESXi hosts in the group which triggered each host to continue retrying the removed LUN over time and slowly driving up CPU utilization until the ESXi hosts were at a crawl with the only recourse to reboot the ESXi hosts.

Has anyone else reported this given the scenario described and if so, is it on the to-do list to remedy? I had a Veeam case open on this topic but had them close it out thinking that this was a Dell developer issue but it appears that it's Veeam developers that would be addressing this issue.
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy »

Hi Pete, what is the support case ID, please?
pbeebe.lp
Novice
Posts: 5
Liked: 1 time
Joined: Nov 09, 2015 3:52 pm
Full Name: Pete LP
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by pbeebe.lp »

@Foggy sorry for the late reply. It would be 06038850.
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy »

It looks like the case is closed and the investigation is being performed on a Dell/VMware side at the moment. This is in line with what we were assuming as based on the described behavior, it looks like the issue is not on the Veeam side here. Let's see how it goes - I'd appreciate it if you could update the thread when you have any news from the vendor.
oliverL
Enthusiast
Posts: 82
Liked: 11 times
Joined: Nov 11, 2016 8:56 am
Full Name: Oliver
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by oliverL »

Did anyone tried to use the storage integration with NVMe over TCP Datastores while the proxy/repository uses iSCSI? Or do we need to attach the PowerStore Volumes over iSCSI to the ESXi hosts, so that storage snapshots will work?
Palmero
Influencer
Posts: 12
Liked: 1 time
Joined: Jul 12, 2021 6:30 pm
Full Name: Juan Palmero
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by Palmero »

Has the new plug-in been released?
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy » 1 person likes this post

oliverL wrote: May 22, 2023 1:16 pm Did anyone tried to use the storage integration with NVMe over TCP Datastores while the proxy/repository uses iSCSI? Or do we need to attach the PowerStore Volumes over iSCSI to the ESXi hosts, so that storage snapshots will work?
Hi Oliver, yes, you cannot have the same volume connected with both technologies.
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy »

Palmero wrote: May 22, 2023 5:38 pm Has the new plug-in been released?
Hi Juan, yes, please see my post above.
Palmero
Influencer
Posts: 12
Liked: 1 time
Joined: Jul 12, 2021 6:30 pm
Full Name: Juan Palmero
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by Palmero »

Thank you. Will give it a try.
oliverL
Enthusiast
Posts: 82
Liked: 11 times
Joined: Nov 11, 2016 8:56 am
Full Name: Oliver
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by oliverL »

foggy wrote: May 22, 2023 5:55 pm Hi Oliver, yes, you cannot have the same volume connected with both technologies.
Thanks for that clarification. Still need to open a case (06078703) :( (customer is currently opening it as i'm not allowed to do so)
The Proxy/repository sees the iSCSI Datastore within his disk management and the storage rescan also says that the iscsi volume is "ok", but the backup job fails with "some volumes are not available for processing through this proxy" (we only enabled san backup for that proxy and disabled failover to nbd for testing purposes).
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy »

Yes, please open a case, our engineers will be able to assist after reviewing the logs.
pbeebe.lp
Novice
Posts: 5
Liked: 1 time
Joined: Nov 09, 2015 3:52 pm
Full Name: Pete LP
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by pbeebe.lp »

foggy wrote: May 19, 2023 12:42 pm It looks like the case is closed and the investigation is being performed on a Dell/VMware side at the moment. This is in line with what we were assuming as based on the described behavior, it looks like the issue is not on the Veeam side here. Let's see how it goes - I'd appreciate it if you could update the thread when you have any news from the vendor.
@Foggy, VMware could only confirm that an APD occurred and was generated by Veeam activity. Dell was only able to point out that they do not develop the plugin and only furnish storage array info to Veeam who in turn develops the plugin and told me that I would need to take this up with Veeam development. VMware ticket is long closed. I've got a contact at Dell if you'd like to compare notes with them on the matter.
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy »

Dell was only able to point out that they do not develop the plugin and only furnish storage array info to Veeam who in turn develops the plugin
Well, this is not true. The plug-in is developed by Dell, Veeam does QA and releases it.

Veeam B&R just issues a snapshot deletion call to the plug-in that then triggers the necessary commands to the storage array. If this causes the issue on the storage, this is either a question to the plug-in for whether it does its part correctly, the storage itself for why it cannot handle the deletion properly, or the improper setup (i.e. host groups configuration).

Anyway, if you want us to investigate, please open a new case with our support for a closer look - we then will be able to report the issue to the Dell development team that produces the plug-in.
pbeebe.lp
Novice
Posts: 5
Liked: 1 time
Joined: Nov 09, 2015 3:52 pm
Full Name: Pete LP
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by pbeebe.lp » 1 person likes this post

@Foggy I'll take what you stated and lob it back over the fence to my contact at Dell. Based on what that stirs up I'll consider opening a new case with Veeam.
oliverL
Enthusiast
Posts: 82
Liked: 11 times
Joined: Nov 11, 2016 8:56 am
Full Name: Oliver
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by oliverL »

Good morning,

just out of curiosity, are there any limitiations regarding the needed PowerStore for the Storage Snapshot integration? As described in my Case 06078703 we could add the PowerStore and trigger the Snapshot of the iSCSI Datastore which also is listed in the PowerStore and in Veeam, but it seems that the Proxy/Repository can't mount the Snapshot, but i also can't manually map the PowerStore Snapshot to the Windows Proxy.
Dell unfortunately isn't a help (i was also told that Dell did not develop the plugin....) as i'm not getting feedback back from them therefore i'm currently struggling to get that done and hope that you guys from here could give me a hint :)

From the PowerStore Documentation i only see that Snapshots aren't used for read/write access but Thinclones are is that something that needs to be changed somewhere?

best regards
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy »

Hi Oliver, all the limitations and known issues are described in the plug-in release notes KB here. I also recommend reviewing the procedure for proxy server configuration.
oliverL
Enthusiast
Posts: 82
Liked: 11 times
Joined: Nov 11, 2016 8:56 am
Full Name: Oliver
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by oliverL » 1 person likes this post

Allright so i wanted to share my findings :-)
If you do security hardening, remember that your backup proxy shouldn't have iSCSI disabled trough GPOs ;-)
Also i wanted to mention that Dell helped outside of a normal support ticket, just to set things straight against my other posts :)

best regards
Oliver
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy » 1 person likes this post

Hi all, to proactively address anticipated questions - the new plug-in version supporting the newest PowerStore OS 3.5 has just been released on the Veeam site. Enjoy. ;)
alex.nhs
Novice
Posts: 3
Liked: never
Joined: Mar 22, 2023 8:31 am
Full Name: Alex Sparks
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by alex.nhs »

pbeebe.lp wrote: May 12, 2023 4:20 pm I had tested the recent plugin against our Powerstore 3600T (v3.2.0.1) and created a test snapshot successfully followed by deleting it. Roughly 12 hours later all the ESXi servers in our environment started exhibiting high CPU host alerts and CPU wait alerts at the VM level. After some back and forth with Dell and Veeam they pointed at the Veeam Powerstore plugin as the culprit and referenced a Veeam Nimble article (https://helpcenter.veeam.com/archive/ba ... _name.html). Our Powerstore configuration utilizes a Host Group to consolidate all ESXi hosts and iSCSI initiators as well as Volume Groups to consolidate all iSCSI LUN datastore. The support reps suggested that due to using Powerstore groups the high CPU utilization was a result of the Veeam snapshot removal which caused an All Paths Down (APD) alert on all ESXi hosts in the group which triggered each host to continue retrying the removed LUN over time and slowly driving up CPU utilization until the ESXi hosts were at a crawl with the only recourse to reboot the ESXi hosts.

Has anyone else reported this given the scenario described and if so, is it on the to-do list to remedy? I had a Veeam case open on this topic but had them close it out thinking that this was a Dell developer issue but it appears that it's Veeam developers that would be addressing this issue.
Hi All,

We just installed the latest plugin yesterday (3000T V3.2.1.0) and experienced similar issues to this. After testing an instant recovery of a VM to our vSphere environment, the mounting of the snapshot and recovery went very well. When it came to dismounting the snap I let Veeam do its thing, it ran the dismount commands on vSphere but they just seemed to get stuck and the volume never dismounted, the job then timed out on veeam and left the mounted snap in a stuck state on vSphere. This then caused most of our esxi hosts to start locking up due to high CPU etc. Is this a known issue with the plugin?
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy »

At least not on the Veeam side. Please open a case with our support to investigate.
Chrispyyy
Enthusiast
Posts: 39
Liked: 9 times
Joined: Jan 04, 2023 6:28 pm
Full Name: Chrispyyy
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by Chrispyyy »

Hi Alex,
I've found this is a known issue when using host groups on the Dell PowerStoreOS platform, you'd be better off removing host groups and mapping individually in the interim. This has been a problem in all releases of the Veeam plug-in, so I guess it's something for Veeam and Dell developers to collaborate on to resolve.

Restoring from storage snapshots are awesome, so I would suggest the interim method - as it's so much faster than restoring from disk-based backups for the most part. Also, something to note is that PowerStoreOS 3.5 introduced immutable snapshots which are definitely worth looking at - although it's not a target code release yet.

P.S. It's nice to see another UK public sector org using the PowerStore platform, they're pretty good! :wink:
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy » 1 person likes this post

I've found this is a known issue when using host groups on the Dell PowerStoreOS platform, you'd be better off removing host groups and mapping individually in the interim. This has been a problem in all releases of the Veeam plug-in, so I guess it's something for Veeam and Dell developers to collaborate on to resolve.
This is actually not a PowerStore plug-in limitation but a general limitation for using storage snapshot integration:

Before you add a storage system to Veeam Backup & Replication, make sure SAN initiator groups for production ESXi hosts and for proxy hosts are isolated and snapshot clones will not be exported to production environment. Otherwise, you may encounter production infrastructure issues such as doubling of the root RGK4IT_VMware_Cluster_Boot Volume to ESXi.
lampshade
Influencer
Posts: 19
Liked: 4 times
Joined: Oct 14, 2022 3:38 pm
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by lampshade »

Hello...

I am afraid to update our plugin! we are currently using plugin version 1.0.47.0 and running powerstore 1000T code 2.X and working flawlessly (using host groups as well on powerstore)... We are going to do a code upgrade on our powerstore soon which we might have no choice to upgrade the plugin... Are there still issues when using host groups on powerstore?
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy »

You will not have any issues provided you keep things separated. This includes not having both ESXi hosts and proxy servers in a single host group and registering host initiators on the array separately (not in a group) - to avoid situations mentioned by Pete above when the restore clone could be mounted to all hosts in a group.
Amistar
Novice
Posts: 6
Liked: never
Joined: May 17, 2021 3:27 pm
Full Name: Mathias Schultz
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by Amistar »

Hey Community,

i was trying to Backup System from a Dell Metro Volume System this weekend and nothing worked.
DELL PowerStorePlugin V2.0.5 from Veeam using 2x Dell PowerStore 500T.
Veeam V12 P20230718
Error Message:
Task failed. Error: Cannot choose between the volume LUNs with the same ID: 'RZ1-SRV-DATA-001'. No synchronous replication relationship was detected for these volumes.
Active-Active PowerStore with Metro Volumes.
No Errors on the PowerStore to see.
I kicked out the PowerStore from Veeam and now its working. When will there be a Plugin for the PowerStore in Connection with Metro Volumes, that will work??

Greeting
Matze
rennerstefan
Veeam Software
Posts: 627
Liked: 146 times
Joined: Jan 22, 2015 2:39 pm
Full Name: Stefan Renner
Location: Germany
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by rennerstefan »

Hi Matze

Are u using a Dell metro node configuration?

Thanks
Stefan Renner

Veeam PMA
Amistar
Novice
Posts: 6
Liked: never
Joined: May 17, 2021 3:27 pm
Full Name: Mathias Schultz
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by Amistar »

Hey @rennerstefan,

No not Metro Node. Metro Volumes. Native since Dell OS 3.X for PowerStore.
i got 2x Dell PowerStore 500T connected with 25GbE and Hosts with FC attached to the PowerStores over Fabric Switches.
So the Config is an Active-Active Host.
Latency is under 5ms
Veeam Plugin does not like that i guess.
Metro Volume on the PowerStore says everything is ok and no errors.
VMware shows all 8 Paths up and 4 of them (due to Active-Active) Actiuve I/O.
So config looks good.
And like i said, once i removed the Storage out of Veeam, Backup worked without any Problems.
foggy
Veeam Software
Posts: 21070
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Dell PowerStore Storage Plug-in (Issues and Future Release?)

Post by foggy »

Hi Mathias, the current version of the Dell PowerStore plug-in supports Native Asynchronous Replication only. Synchronous replication (Metro Volumes) is not supported.
Post Reply

Who is online

Users browsing this forum: civilwar98 and 44 guests