Data Domain 3300 with DD OS 7.5

[fdietze]

Hello!

i have a customar that upgraded by himself his data domain 3300 from DD OS 7.1 to DD OS 7.5 (recently launched 23/02/2021).

this customer is already with veeam 11 before this data domain upgrade that was yesterday.

yes i know, the doc says "Dell EMC Data Domain (DD OS version 6.1 to 7.3) with DDBoost license. Both Ethernet and Fibre Channel (FC) connectivity is supported." on https://helpcenter.veeam.com/docs/backu ... 110#target

after upgrade i have the error:
"Error Failed to save Backup Repository: Failed to call RPC function 'DDBoostFindUnitInfo': Failed to connect to storage. Storage: 172.26.5.1, User: veeam. the user has insufficient access rights. Err: 5075."

yes i saw the thread veeam-backup-replication-f2/veeam-fails ... 57064.html and the thread veeam-backup-replication-f2/adding-data ... 57862.html

its not a user situation since i teste everything, i tryied new storage unit, new user, changed clients acceptance and every single option, i compared with other customer running 7.1 and 7.3 and everything looks the same. really appear to be something changed in the ddboost client or connection.

yes, i opened a case to see if has any workaround because data domain os does not offer downgrade or rollback (you need to reinstall from scratch and lost eveything). any comment or workaround this time until full support will be appreciated.
Case #04689950

thanks in advance!

[foggy]

Hi Flavio, that's why we don't claim support for a particular version before running tests internally. So far we haven't performed any verification of 7.5 so cannot comment. Have you also contacted Dell EMC for any information on a possible related change in the latest version?

[fdietze]

Foggy,

yes i have contact because was emc support from india that upgraded DD without asking nothing to customer. the DD3300 of the customer was with problem in the ssd cache. Support made the upgrade for resolve the ssd, the upgrade did not solve and they exchanged the disk but all the backup problem came on table.

they didnt know anything related to say, their work was pretty amateur and now i have this situation and asked for help in the case number mentioned.

any help appreciated.

[foggy]

I'd ask for the case escalation inside Dell EMC support org then. We will definitely do our best to find what've caused the issue, but since it is likely the change on the DDOS side, I'd continue working with Dell EMC as well.

[fdietze]

Foggy,

i know, i already raised and talked to escalation but first they asked for sorry after upgrading without asking and then checked ddboost cnnectivity with the ddboot conectivity check and all pass. they said now the problem is at client side (ddboost client inside veeam).

i´m lost.

[fdietze]

Just to let know that tests trought ddbostfs version 7.5 in the os where the console resides can send ddboost traffic do datadomain.

[soncscy]

Surely they must have logging that explains exactly what access is not allowed?

AFAIK for any application, not just Veeam, you use a library to connect to the data domains, you don't get to customize this client, you just use the ddboostlib.dll library provided and that's it. It's their own native client returning this, and they should have logging which says specifically which permission was wrong and why or why the access was denied.

I know it can be tough with some support teams, but you need to be persistent and direct. It's non-sense to suggest that an application could some how force and produce a DDboost access error...

As a workaround, couldn't you expose the share temporarily as CIFS and remap the backups? (disable the DDBoost repo in Veeam)

[fdietze]

Hello, i agree with soncsy. doesnt make sense for me at all the point of the problem is in veeam but i was needed to ask for something in the case. because veeam could be verifying or analysing in the dev labs the dd os 7.5 and could be a know issue even in dd.

im true case with Dell because the problem is in dd.

after the test ok with ddboostfs, was too little tests. i made other tests without veeam, with just accessing cifs and ddboostfs. initial testes ok and after some moment the dd start to denied read and write in storage units.
now the case is escalated with dell and have two dd engineers looking, one from europe and one from usa.

even with dd os in 7.5 version out of the veeam matrix i was not with confidence that the problem was in veeam.

when dell confirm something and tell something i will update here.

[foggy]

We will investigate on our side, but since it is a communication issue between two Dell EMC components, involving them is the right step. Could you please share the Dell case number? I can also say that you're not the only one - we already have other similar cases of imprudent upgrades followed by the same error.

[fdietze]

hello foggy, news and solved!

for sure the problem was at dd side, i already closed veeam case.

the engineer from europe inditified the probleam and made a new patch for dd version 7.5.0.006-680172.rpm even with number lower than dd (7.5.0.5) this patch solved my case and in be inside the next release 7.5.0.10. they reported internally as know issue with dd and will include in the next release notes.

[adurelli]

Hi fdietze!

I´m having the same problem as you. Can you send me the patch, so I can test here too?

Tks!

[Andreas Neufert]

Please do not share patches outside of support cases. It is important to know which customer is on what patch so that we can follow up if we face issues at updates or overall.

[fdietze]

Andreas, the patch i am saying is a patch for Dell EMC Data Domain...

[fdietze]

adurelli, sent you dell case number so they can provide you the patch.

[foggy]

Flavio, thanks for the update and please also PM me the case number for reference as I'd like to get the patch for our internal DDOS testing as well.

[stevekarra]

It's a good practice to adhere to the target code revisions to minimize the chances of disruption. 7.2.0.50 is the current stable and well-tested version.

As long as you have a Dell support contract this information can be obtained from https://dl.dell.com/content/docu64492_D ... uage=en_US

[Trelor]

Not trying to post jack, but please note that there is also a bug in DDOS 6.2 -> 7.4 which we are facing and DD support has advised to go to 7.5.0.5 due to bug 213735 where a BTree causes MTree issues and file system crashes.

[stevekarra]

That's unfortunate. You're running out of runway with DDOS 7.5.0.5 - hopefully, all will be good.

DDOS cannot be downgraded which is why I like to follow the target code recommendations. It also gives you breathing room in case you do run into a problem fixed in a later version.

[oan]

the engineer from europe inditified the probleam and made a new patch for dd version 7.5.0.006-680172.rpm even with number lower than dd (7.5.0.5) this patch solved my case and in be inside the next release 7.5.0.10. they reported internally as know issue with dd and will include in the next release notes.

Hi, do you have details on the patch? I am sharing your problem and would like to share the solution.
Oliver

[foggy]

Hi Oliver, please contact Dell EMC support to confirm the issue and get the patch.

[oan]

Update: Dell EMC has released DDOS 7.5.0.10 which solves the issue.

[DRuben]

a customer of mine has upgraded to 7.5.0.10 and has the same issue

[Trelor]

Hi Oliver, please contact Dell EMC support to confirm the issue and get the patch.

This is the link to DDOS 7.5.0.10, you need to login first: https://dl.dell.com/content/docu103371

Also from the release notes: DDOS-92082 - Backup applications using DD Boost versions prior to 7.1 may not be able to complete operations like backups, restore, etc

[foggy]

a customer of mine has upgraded to 7.5.0.10 and has the same issue

Hi Ruben, do you have a case open with either Dell EMC or Veeam regarding that?

[JTT]

Hi Flavio, that's why we don't claim support for a particular version before running tests internally. So far we haven't performed any verification of 7.5 so cannot comment. Have you also contacted Dell EMC for any information on a possible related change in the latest version?

Offtopic, but when will Veeam support it then?? Next VBR version isnt a acceptable solution,

[foggy]

We have planned the tests but ETA will depend on whether any changes are required on our side to support it - in this case, it inevitably will be one of the next Veeam B&R updates (not necessarily a major version, if that is your concern).

[JTT]

Well my only concern is that, how can Veeam claim itself to be a enterprise product, but at the same time cant keep up with the hardware vendors software updates!?

[Andreas Neufert]

I would argue that the opposit is the case. We carefully testing new versions deeply within our QA so that our user install base can trust in our code and reliability. This means we have less code releases than the vendors (Dell releases new major DDOS versions every quarter now and there are updates and patches as well in between) available for the testing and necessary code changes. As well we need to find the right balance with the SDK/DDOS version support as newer SDK means as well that we cap exisiting customers that run on older DDOS versions. Overall our new version adaption process worked reliably in the last 8 years with DataDomain.

To compare this with VMware vSphere Version releases. We support new versions there usually within 90 days of the GA. This is needed because of the massive API and code changes VMware introduces. Even last minute changes in the APIs.

DDOS 7.5.0.10 was released March 16th, 2021

[foggy]

FYI, we have completed v11 verification against DDOS 7.6 and now all the latest DDOS versions (7.4/7.5/7.6) are officially supported. We will update the documentation to reflect that fact shortly. Thanks!

[JTT]

Veeam support said: Veeam doesn't have any limitations in the documentation about the Firmware,
This means all the versions are supported.

Is this true, so we can apply latest update?