Veeam is using deprecated method of connecting to EMC Unity

[FrancWest]

Hi,

Case # 02599583.

It appears that Veeam is using an older, deprecated, method of connecting to a EMC Unity array. This causes all kinds of strange issues regarding management of the Unity. For example you constantly get a warning message about a host that doesn't have any connected initiators. It also causes so called 'ghost' luns to be presented to ESXi hosts. When you create a new datastore, this 'ghost' lun shows up as a free lun which you can format. I haven't tried what happens when you format it though, since it's not an actual lun that is created on the Unity. It's a container which is being used to attach a snapshot to, according to EMC support.

I contacted veeam support and according to them this behavior won't be changed any time soon. To me that makes no sense, EMC Unity is a relative new product line (less than a year old) and for some reason Veeam decided to use an old API to manage these devices from Veeam B&R.

Franc.

[Gostev]

To me that makes no sense, EMC Unity is a relative new product line (less than a year old) and for some reason Veeam decided to use an old API to manage these devices from Veeam B&R.

Hi, Franc. It does not make sense because this is now what actually happened.

We had the existing integration in our product for EMC VNXe, which was built using the only available API at the time. When EMC Unity was released, it mostly "just worked" because Unity is identical to VNXe under the hood (frankly, they are so similar it seemed to me like a simple rebranding). Anyway, because everything just worked, we started to officially support Unity as well.

Now, of course things we chose to use a few years ago for VNXe gradually become more and more legacy... but it was not at the Unity release time when we chose to use those APIs. Hope this makes better sense now! We rarely do things which don't make sense

Thanks!

[FrancWest]

Hi,

I understand, but this older API causes several issues now so it has side effects. Any plans on updating the veeam logic to the new API in the near future?

Franc.

[foggy]

Hi Franc, we will consider existing options to address these issues in one of the future updates. Thanks for your feedback.

[kwinsor]

Hi,
We are having similar issues with our EMC Unity arrays. Case # 02484119. We see warnings of single path hosts that get created when Veeam does a rescan using it's own initiator despite the proxy host having MPIO configured using the Microsoft Initiator. We end up with multiple hosts in Unity for the same proxy server. One for Veeam Initiator and one for Microsoft Initiator. This leads to lowered CloudIQ scores for the arrays which we don't like. Also, so far, myself nor EMC can remove the host created with the Veeam Initiator because it's mapped to the so called ghost LUNs mentioned previously. An even bigger issue here is that this method allows for only a single snap mount at a time. Therefore, jobs following other jobs with VM's on the same VMware datastore have to wait until the first job is finished and the snapshot is unmounted before the next job starts which is slowing down our backups.
We would like to see this fixed sooner rather than later if possible.

Thanks,
Kent

[NightBird]

+1 same problem at many customers...

You can try to remove ghost "host access" with :
uemcli /stor/prov/vmware/vmfs show -detail (to see vmfs volume)

uemcli -U user -P password -d x.y.z.w -sslPolicy accept /stor/prov/vmware/vmfs -id res_1 set -snapHosts "" (to remove ghost host access)

[FrancWest]

EMC support already did this but the next day the issue is back again and the veeam initiator host is being recreated again.

Another issue we are having is that veeam is unable to restore from a storage snapshot created on the unity. It creates the wrong iscsi target on the esxi hosts. It doesn’t use the existing iscsi targets on the esxi host, but instead creates a new, wrong, one with an IP address of the storage processor which doesn’t own the lun. As a result, the snapshot mount on the esxi host fails since the host cannot connect to the snapshot of the iscsi lun using this wrong IP. Case number for this issue is 02556472.

[NightBird]

I have always FC, any update about your case ?

[Gostev]

I have actually discussed this with the devs back in February, but forgot to post an update, sorry.

1. 02556472 is not related to deprecated API usage at all. It is currently on the highest level of escalation, being investigated by R&D.

2. We confirmed with EMC support that the API call we're using is NOT deprecated (they actually specifically suggested us using the API call that we're already using).

3. Warning message mentioned in the first post has no consequences, but fixing it is quite dangerous (may break concurrent backup/restore/rescan) so we decided not to attemp to hot fix this. It would make sense to approach this differently, as the whole complexity comes from the fact that Unity did not support exporting multiple snapshots of the same LUN at the time. But judging on the current, now they do and we may be able to start working with them like with any other storage - without this extra code. This will be investigated for the next updates.

Thanks!

[FrancWest]

Hi,

any news on this? It seems that after update 3a the warnings in the eventlog of the Unity about single paths and a not connected iscsi initiatior (the one created by veeam) has increased significantly. Every our we get 2 of these warnings (when the backup is taking place). After 3 days we already have 90 of these warnings in the unity alert log. This makes managing the unity more difficult since it's being cluttered by the events caused by Veeam.

Host VeeamAUX_Veeam_VeeamInitiator does not have any initiators logged into the storage system.
Host VeeamAUX_Veeam_VeeamInitiator is only configured with one path to the storage system. Configure multiple paths to establish redundancy.

[foggy]

Hi Franc, this has been investigated and the plan is to address this in one of the future updates. I cannot provide any particular ETA though, the amount of changes required is significant.

[NightBird]

Any chance for U4 ?

[foggy]

Low chances.

[NightBird]

From U4 what's new

Seems that changes were made

"DELL EMC Unity integration received some changes to its snapshot export approach to prevent storage from logging warnings and other issues and allow more than one LUN snapshots to be exported to backup proxy."

[foggy]

Yes, we were able to include this into U4.

[FrancWest]

Well, even after update 4 I still see the following warnings in the EMC Unity alerts:

Host VeeamAUX_Veeam_VeeamInitiator does not have any initiators logged into the storage system.

And

VeeamAUX_VeeamInitiator is only configured with one path to the storage system. Configure multiple paths to establish redundancy.

[Andreas Neufert]

I think the first message is by design of Veeam and nothing we can do about on our side. We will reach out to DellEMC to discuss. Our rescan engine connects only temporary to the cloned LUNs to scan for new VMs on the Snapshots. After that we do not keep an open connection and it would not make sense to do so at all. That said, we need to speak with DellEMC about it, so that they add a filter to ignore "VeeamAUX..." at this error message.

Second part is as well by design of our architecture as we only use it for temporary rescans nothing else. Will speak with DellEMC here as well to ask for a filter. If they can´t we could do something on our side in one of the upcoming versions.

[FrancWest]

Ok, but what warnings are the release notes talking about then that were fixed in U4?

[Gostev]

There are a number of issues fixed around EMC primary storage support, for example this one as well.

[FrancWest]

ok, but that thread talks about a bug with NFS, not about the warnings I'm referring to mentioned in the release notes.

[Gostev]

I am the one who wrote the release notes, and I can tell you this snippet may very well relate to the fixed bug with NFS, because snapshot export functionality is used there as well.

As far as I remember, the raw data I had mentioned multiple improvements as a result of changes around storage snapshot management functionality, including fixing some storage-side warnings... but it did not specify which ones, so I left it at the same level of vagueness in the document. It would be too much to include this level of details on specific warnings for a What's New document anyway, as then it will not be consumable.

We will confirm with developers for sure.

[foggy]

There were specific changes made in U4 to prevent those initiator warnings (I can see the corresponding issue logged in our bugtracking system, which is marked as resolved). The fact that you still see them might be explained by some old dependencies left. I recommend you to contact support again for assistance with volume access options cleanup.

[FrancWest]

ok, will do. Thanks.