Storage snapshot fail on v11 after Unity fw v5.1

[nobrell]

Hi,

I ran into a problem after updating our Dell EMC Unity 350F SAN from 5.0.7.0.5.008 to the new firmware 5.1.0.0.5.394.
Storage snapshot fail on all jobs and LUN.

Failed to create snapshot for LUN NNN Error: ere is a syntax error in the command. Please recheck the command syntax. (Error Code:0x1000017)

Running Veeam Backup & Replication v11.0.0.837 P20210525 and vSphere 7.0U2
Have anybody else run in to the same problem yet?
Is it a know issue att this time?

There isn't much to go on, its seems like the command syntax for creating a snapshot have changed in 5.1.
Thanks

[foggy]

Hi Rikard, 5.1 has just been released a couple of weeks ago and we haven't yet tested against it (hence it is not mentioned in the system requirements as the supported one). We will schedule the tests and get back when we have something to share. Thanks!

[vesak]

We have the same issue. Had to upgrade to Unity fw v5.1 due to security reasons. Now our Veeam v11 storage snapshots (and hence all our backups) started failing. Please post here as soon as there is a resolution.

[KaySeverin]

Hi,
we have the same issue after the OS update to OS 5.1.0.0.5.394. Error Code:0x1000017 but we are running diffenrent config to the config above.
Failed to create snapshot for NFS export 121_vmstore_win01 Details: ere is a syntax error in the command. Please recheck the command syntax. (Error Code:0x1000017)

Running Veeam Backup & Replication v10.0.1.4854 P20201202 and vSphere Client Version 7.0.1.00200
VMWare ESXi (VMKernel Release 6.7.0 buil 16773714)
Dell Unity 480 with OS 5.1.0.0.5.394 since 03.jul.2021

Regards

[foggy]

May I ask you guys to open cases with Veeam support so we could review the logs and see if you're all seeing the same issue?

[KaySeverin]

Did it now! Case # 04897321

[nobrell]

May I ask you guys to open cases with Veeam support so we could review the logs and see if you're all seeing the same issue?

I did that first actually: Case # 04884072

[nobrell]

Hi Rikard, 5.1 has just been released a couple of weeks ago and we haven't yet tested against it (hence it is not mentioned in the system requirements as the supported one). We will schedule the tests and get back when we have something to share. Thanks!

Thanks for your quick reply!
I understand the the situation, but its difficult in todays world to hold off an update for 90d.
I have been updating this Unity SAN for 3y, without any issues so I guess my luck ran out.

I find it very strange that Dell EMC changed the syntax, I will open a case and ask them as well.

If at all possible I´m thankful if you can post your finds.
Thanks!

[antonk]

Hi guys,

The same issue is here. The machine had arrived already with a new 5.1.0 firmware. With 5.0.7 everything worked like a charm, but with the new machine we are simply unable to add it to storages, receiving the following error:

Response received from server does not contain a valid XML
Output error returned from target: 10.0.0.26
CLI commands are not supported by target system

As per my understanding, something got failed with Unity CLI and API on a new firmware.
Advises are welcome. We are running latest v.11 Veeam with latest patch.

For reference a case to EMC has been raised, with ID: 116637627

[Gostev]

My only advise to you would not be very helpful in the current situation, and that is to respect the official System Requirements for Veeam Backup & Replication and don't knowingly put yourself into an unsupported configuration.

Whereas in this case, there's a hope this is something that would be relatively easy for us to hotfix from our end by adjusting our product to the changed CLI output, I can tell you this is not always the case and we had situations before where returning official support after some change by the vendor was taking far more than our target 90 days. In one case the change required major re-architecture on our end, and in at least a few other cases we had to wait for a vendor for nearly a year to fix an API regression on their end as the situation had no workarounds. And we can only hope this won't be the case here.

Given that, it is never a good idea to take a leap of faith and upgrade to an unsupported firmware version just because you were lucky enough with this approach for the past 3 years.

[antonk]

Hi Anton,

In our case, for instance, we had no choice. A brand new machine had already arrived with 5.1.0 firmware so we are checking right now with EMC, if a downgrade can be performed and what's the impact of such a downgrade.

[KaySeverin]

I find the advice not to update in a timely manner not helpful. Rather, Veeam should be closer to the storage manufacturer in order to react preventively before a new OS is released.

[Andreas Neufert]

We are working very closely with all our vendors that we have integrations with. In case of DellEMC we have multiple calls with their storage teams every week!
You can be as close as possible to the vendors, there is always a chance that a change/bug that was not in any pre release will break the integration. In this case we were not notified of any planned CLI changes that could affect us.

[nobrell]

I have opened a case with Dell EMC as well and they are basically saying that they haven't changed anything in the syntax and that the issue is related to the Uemcli package installed on Veaam v11.

The uemcli tool installed with the UnisphereCLI package but can also be integrated in other tools or applications as for example in Veeam Backup and Replication.
This issue can also been seen with the default UEMCLI installation on the Unity array, but this is not a Unity side issue.
The issue will be wherever UEMCLI is being run from when this error is displayed.

In the current SR I could see a different error, however from our side we have not changed any syntax so can you please advise the
exact syntax that you are currently using to double check it?

I'm not using emcli and the syntax used by Veeam is as far as I can tell not visible.
Are you able to reveal what syntax is used so I can pass that on to the Dell EMC support?
I can also try that

Code: Select all

 uemcli /prot/snap 

command manually and see if there is an error message.
https://www.delltechnologies.com/en-lt/ ... t-data.htm

They are also saying that Veeam support are saying that previous similar errors was solved with substituting the CLI version shipped with Veeam v10, is that something worth trying?

We have been using the workaround substituting the CLI version shipped with Veeam v10 since then.

I feel a bit like a pinball, but I would like to try to help out to get this matter resolved.

[foggy]

According to our investigation so far, it doesn't look like Dell EMC haven't changed anything in the syntax - we see a different output for one of the commands and the issue is that one of the parameters that we expect is not returned. Investigation continues.

[Gostev]

Shoot, a syntax change would have been better... while this looks more like these cases then

in at least a few other cases we had to wait for a vendor for nearly a year to fix an API regression on their end as the situation had no workarounds

We can only hope the fix won't take Dell too much time to implement.

[foggy]

There's a chance we can get along without that parameter.

[nobrell]

That sounds promising, thanks for the update foggy!

[foggy]

R&D confirmed that we do not need the missing parameter and will adapt our response processing logic to the new output format in the next update.

[nobrell]

Great news, thanks foggy!

[sfaits]

We are looking at having to upgrade to the Unity 5.1.0.0.5 patch soon.
We do not use storage level snapshots, is there any other issues with backups?

Just want to get some more information as Dell is being very vague.

Thanks much!!

[foggy]

Hi William, in case storage integration is not used, Veeam B&R is not affected by the API changes on the storage side.

[sfaits]

Thank you foggy!

[Eblue]

We are experiencing the same problem. Can't wait for the next update.
Any ETA?

[Andreas Neufert]

DellEMC has developed a bugfix for this issue.
If you are affected please open a support case with DellEMC and reference to DellEMC KB535472.
UPDATE from Dell: New reference link is: https://www.dell.com/support/kbdoc/en-us/000189458
The KB speaks about other things, but DellEMC attached an internal note to the bugfix. If you reference this KB, then they are able to give your the hotfix.

We are planning as well to add workarounds on our side but the DellEMC bugfix is working today.

Thanks to the DellEMC team who addressed this situation quickly on the storage side.

[FrancWest]

Are you sure that’s the correct KB? That one is about Veeam 9.5 and Unity 4.5. It’s from 2019.

[Andreas Neufert]

Hi Franc,
I updated my statement. The KB is the correct one, they just added an internal reference to the bugfix to it.

[FrancWest]

Ok, thanks!

[Andreas Neufert]

New link from DellEMC as reference: https://www.dell.com/support/kbdoc/en-us/000189458

[nathano]

Brand new unity and can't add it to Veeam, same problem, on the phone to Dell now. fingers crossed