Dell PowerStore Storage Plug-in (Issues and Future Release?)

[DigIT]

I'm on 12. Also, I realized today that I didn't have the network configuration correct -- the backup proxies didn't have network access to the storage, but they now do. I rescanned the storage and now I'm seeing "Successfully determined LUNs supported for backup from storage snapshots on storage..." Still falling back to VM snapshot instead of storage. Will update this thread with anything useful that support finds.

[DigIT]

I was missing some additional configuration requirements -- such as MPIO on my Veeam Proxies. I've got backups running directly from storage snapshots right now!

Thanks so much, @foggy et al!

[pbeebe.lp]

I had tested the recent plugin against our Powerstore 3600T (v3.2.0.1) and created a test snapshot successfully followed by deleting it. Roughly 12 hours later all the ESXi servers in our environment started exhibiting high CPU host alerts and CPU wait alerts at the VM level. After some back and forth with Dell and Veeam they pointed at the Veeam Powerstore plugin as the culprit and referenced a Veeam Nimble article (https://helpcenter.veeam.com/archive/ba ... _name.html). Our Powerstore configuration utilizes a Host Group to consolidate all ESXi hosts and iSCSI initiators as well as Volume Groups to consolidate all iSCSI LUN datastore. The support reps suggested that due to using Powerstore groups the high CPU utilization was a result of the Veeam snapshot removal which caused an All Paths Down (APD) alert on all ESXi hosts in the group which triggered each host to continue retrying the removed LUN over time and slowly driving up CPU utilization until the ESXi hosts were at a crawl with the only recourse to reboot the ESXi hosts.

Has anyone else reported this given the scenario described and if so, is it on the to-do list to remedy? I had a Veeam case open on this topic but had them close it out thinking that this was a Dell developer issue but it appears that it's Veeam developers that would be addressing this issue.

[foggy]

Hi Pete, what is the support case ID, please?

[pbeebe.lp]

@Foggy sorry for the late reply. It would be 06038850.

[foggy]

It looks like the case is closed and the investigation is being performed on a Dell/VMware side at the moment. This is in line with what we were assuming as based on the described behavior, it looks like the issue is not on the Veeam side here. Let's see how it goes - I'd appreciate it if you could update the thread when you have any news from the vendor.

[oliverL]

Did anyone tried to use the storage integration with NVMe over TCP Datastores while the proxy/repository uses iSCSI? Or do we need to attach the PowerStore Volumes over iSCSI to the ESXi hosts, so that storage snapshots will work?

[Palmero]

Has the new plug-in been released?

[foggy]

Did anyone tried to use the storage integration with NVMe over TCP Datastores while the proxy/repository uses iSCSI? Or do we need to attach the PowerStore Volumes over iSCSI to the ESXi hosts, so that storage snapshots will work?

Hi Oliver, yes, you cannot have the same volume connected with both technologies.

[foggy]

Has the new plug-in been released?

Hi Juan, yes, please see my post above.

[Palmero]

Thank you. Will give it a try.

[oliverL]

Hi Oliver, yes, you cannot have the same volume connected with both technologies.

Thanks for that clarification. Still need to open a case (06078703) (customer is currently opening it as i'm not allowed to do so)
The Proxy/repository sees the iSCSI Datastore within his disk management and the storage rescan also says that the iscsi volume is "ok", but the backup job fails with "some volumes are not available for processing through this proxy" (we only enabled san backup for that proxy and disabled failover to nbd for testing purposes).

[foggy]

Yes, please open a case, our engineers will be able to assist after reviewing the logs.

[pbeebe.lp]

It looks like the case is closed and the investigation is being performed on a Dell/VMware side at the moment. This is in line with what we were assuming as based on the described behavior, it looks like the issue is not on the Veeam side here. Let's see how it goes - I'd appreciate it if you could update the thread when you have any news from the vendor.

@Foggy, VMware could only confirm that an APD occurred and was generated by Veeam activity. Dell was only able to point out that they do not develop the plugin and only furnish storage array info to Veeam who in turn develops the plugin and told me that I would need to take this up with Veeam development. VMware ticket is long closed. I've got a contact at Dell if you'd like to compare notes with them on the matter.

[foggy]

Dell was only able to point out that they do not develop the plugin and only furnish storage array info to Veeam who in turn develops the plugin

Well, this is not true. The plug-in is developed by Dell, Veeam does QA and releases it.

Veeam B&R just issues a snapshot deletion call to the plug-in that then triggers the necessary commands to the storage array. If this causes the issue on the storage, this is either a question to the plug-in for whether it does its part correctly, the storage itself for why it cannot handle the deletion properly, or the improper setup (i.e. host groups configuration).

Anyway, if you want us to investigate, please open a new case with our support for a closer look - we then will be able to report the issue to the Dell development team that produces the plug-in.

[pbeebe.lp]

@Foggy I'll take what you stated and lob it back over the fence to my contact at Dell. Based on what that stirs up I'll consider opening a new case with Veeam.

[oliverL]

Good morning,

just out of curiosity, are there any limitiations regarding the needed PowerStore for the Storage Snapshot integration? As described in my Case 06078703 we could add the PowerStore and trigger the Snapshot of the iSCSI Datastore which also is listed in the PowerStore and in Veeam, but it seems that the Proxy/Repository can't mount the Snapshot, but i also can't manually map the PowerStore Snapshot to the Windows Proxy.
Dell unfortunately isn't a help (i was also told that Dell did not develop the plugin....) as i'm not getting feedback back from them therefore i'm currently struggling to get that done and hope that you guys from here could give me a hint

From the PowerStore Documentation i only see that Snapshots aren't used for read/write access but Thinclones are is that something that needs to be changed somewhere?

best regards

[foggy]

Hi Oliver, all the limitations and known issues are described in the plug-in release notes KB here. I also recommend reviewing the procedure for proxy server configuration.

[oliverL]

Allright so i wanted to share my findings
If you do security hardening, remember that your backup proxy shouldn't have iSCSI disabled trough GPOs
Also i wanted to mention that Dell helped outside of a normal support ticket, just to set things straight against my other posts

best regards
Oliver

[foggy]

Hi all, to proactively address anticipated questions - the new plug-in version supporting the newest PowerStore OS 3.5 has just been released on the Veeam site. Enjoy.

[alex.nhs]

I had tested the recent plugin against our Powerstore 3600T (v3.2.0.1) and created a test snapshot successfully followed by deleting it. Roughly 12 hours later all the ESXi servers in our environment started exhibiting high CPU host alerts and CPU wait alerts at the VM level. After some back and forth with Dell and Veeam they pointed at the Veeam Powerstore plugin as the culprit and referenced a Veeam Nimble article (https://helpcenter.veeam.com/archive/ba ... _name.html). Our Powerstore configuration utilizes a Host Group to consolidate all ESXi hosts and iSCSI initiators as well as Volume Groups to consolidate all iSCSI LUN datastore. The support reps suggested that due to using Powerstore groups the high CPU utilization was a result of the Veeam snapshot removal which caused an All Paths Down (APD) alert on all ESXi hosts in the group which triggered each host to continue retrying the removed LUN over time and slowly driving up CPU utilization until the ESXi hosts were at a crawl with the only recourse to reboot the ESXi hosts.

Has anyone else reported this given the scenario described and if so, is it on the to-do list to remedy? I had a Veeam case open on this topic but had them close it out thinking that this was a Dell developer issue but it appears that it's Veeam developers that would be addressing this issue.

Hi All,

We just installed the latest plugin yesterday (3000T V3.2.1.0) and experienced similar issues to this. After testing an instant recovery of a VM to our vSphere environment, the mounting of the snapshot and recovery went very well. When it came to dismounting the snap I let Veeam do its thing, it ran the dismount commands on vSphere but they just seemed to get stuck and the volume never dismounted, the job then timed out on veeam and left the mounted snap in a stuck state on vSphere. This then caused most of our esxi hosts to start locking up due to high CPU etc. Is this a known issue with the plugin?

[foggy]

At least not on the Veeam side. Please open a case with our support to investigate.

[Chrispyyy]

Hi Alex,
I've found this is a known issue when using host groups on the Dell PowerStoreOS platform, you'd be better off removing host groups and mapping individually in the interim. This has been a problem in all releases of the Veeam plug-in, so I guess it's something for Veeam and Dell developers to collaborate on to resolve.

Restoring from storage snapshots are awesome, so I would suggest the interim method - as it's so much faster than restoring from disk-based backups for the most part. Also, something to note is that PowerStoreOS 3.5 introduced immutable snapshots which are definitely worth looking at - although it's not a target code release yet.

P.S. It's nice to see another UK public sector org using the PowerStore platform, they're pretty good!

[foggy]

I've found this is a known issue when using host groups on the Dell PowerStoreOS platform, you'd be better off removing host groups and mapping individually in the interim. This has been a problem in all releases of the Veeam plug-in, so I guess it's something for Veeam and Dell developers to collaborate on to resolve.

This is actually not a PowerStore plug-in limitation but a general limitation for using storage snapshot integration:

Before you add a storage system to Veeam Backup & Replication, make sure SAN initiator groups for production ESXi hosts and for proxy hosts are isolated and snapshot clones will not be exported to production environment. Otherwise, you may encounter production infrastructure issues such as doubling of the root RGK4IT_VMware_Cluster_Boot Volume to ESXi.

[lampshade]

Hello...

I am afraid to update our plugin! we are currently using plugin version 1.0.47.0 and running powerstore 1000T code 2.X and working flawlessly (using host groups as well on powerstore)... We are going to do a code upgrade on our powerstore soon which we might have no choice to upgrade the plugin... Are there still issues when using host groups on powerstore?

[foggy]

You will not have any issues provided you keep things separated. This includes not having both ESXi hosts and proxy servers in a single host group and registering host initiators on the array separately (not in a group) - to avoid situations mentioned by Pete above when the restore clone could be mounted to all hosts in a group.

[Amistar]

Hey Community,

i was trying to Backup System from a Dell Metro Volume System this weekend and nothing worked.
DELL PowerStorePlugin V2.0.5 from Veeam using 2x Dell PowerStore 500T.
Veeam V12 P20230718
Error Message:
Task failed. Error: Cannot choose between the volume LUNs with the same ID: 'RZ1-SRV-DATA-001'. No synchronous replication relationship was detected for these volumes.
Active-Active PowerStore with Metro Volumes.
No Errors on the PowerStore to see.
I kicked out the PowerStore from Veeam and now its working. When will there be a Plugin for the PowerStore in Connection with Metro Volumes, that will work??

Greeting
Matze

[rennerstefan]

Hi Matze

Are u using a Dell metro node configuration?

Thanks

[Amistar]

Hey @rennerstefan,

No not Metro Node. Metro Volumes. Native since Dell OS 3.X for PowerStore.
i got 2x Dell PowerStore 500T connected with 25GbE and Hosts with FC attached to the PowerStores over Fabric Switches.
So the Config is an Active-Active Host.
Latency is under 5ms
Veeam Plugin does not like that i guess.
Metro Volume on the PowerStore says everything is ok and no errors.
VMware shows all 8 Paths up and 4 of them (due to Active-Active) Actiuve I/O.
So config looks good.
And like i said, once i removed the Storage out of Veeam, Backup worked without any Problems.

[foggy]

Hi Mathias, the current version of the Dell PowerStore plug-in supports Native Asynchronous Replication only. Synchronous replication (Metro Volumes) is not supported.