v10 RTM DataDomain 7.1.0.5?

[Trelor]

Hello,

We have had issues in the past with our Veeam and DataDomain. I had to upgrade from V7.0.07 to 7.1.0.5 to fix other issues that we had however after upgrading Veeam 9.5U4b stopped talking to the Datadomain. I got an error 5040, Veeam support said it is not their problem, Dell EMC siad it is not their problem. Who can assist me with this. I though the upgrade to v10 RTM fixed it but it did not, it look like it worked then on a re-scan it was no longer working.

[HannesK]

Hello,
can you please share your case number?

I know that DD-OS 7.0 is supported for V10, but need to check for 7.1

Best regards,
Hannes

[foggy]

This might be the DD Boost SDK/DD OS compatibility issue. We definitely need case numbers on both sides to investigate. Veeam B&R currently uses Boost SDK 3.3 and based on the -2/+4 compatibility rule from Dell EMC, DD OS 7.1 might not be supported by this SDK version.

[Trelor]

I have a Veeam case and an DellEMC case how should I proceeded in passing off the DellEMC case Veeam case # is 03974819

[foggy]

Thanks! Please share the Dell EMC case number here or PM it to me.

[Trelor]

PMed thank you foggy and Hannes

[Trelor]

Has there been anyone looking at this I have not heard a thing since I logged it here and passed off the support case numbers

[foggy]

Yes, I'm in discussion with Dell EMC regarding the supportability of 7.1. Meanwhile, could you please check in the storage management console what are your DD Boost Global Security Settings?

[Trelor]

Thank you, and is this what you are looking for?

[foggy]

Ok, thanks. I will get back once I have a definitive answer. Chances are you would need to re-open the support case with Veeam (I can see that it was closed after you admitted that upgrade to v10 RTM helped) to proceed with the investigation.

[Trelor]

OK, I will wait before I open a new case; thanks.

[foggy]

According to Dell EMC, our current DDBoost SDK 3.3 is compatible with DDOS 7.1, so this is not the compatibility issue. Please re-open the support case with Veeam for a deeper investigation of connectivity problems (it might be something not version related). I also must say that we haven't specifically tested Veeam B&R v10 against DDOS 7.1 so far and cannot claim it is fully supported with this release. Thanks!

[Trelor]

I reopened it and the support ID is #03998591

[Trelor]

This is definitely a DataDomain bug....The "fix" was to change the DDBoost interface group from enabled to disabled and then it magically worked. I am not sure if it is corrupted but I only have one interface group and now Veeam is talking to the DataDomain on multiple interfaces as it did before I have my Dell EMC Case open still but I will update when they respond. FWIW, Veeam 9.5 U4b / Veeam 10 work with DDOS 7.1.0.5; just be warned of interface groups.

[foggy]

Good news, indeed. I appreciate the update, thanks!

[hsayil]

We had the same problem and dell engineers couldn't find a solution.

Our problem was solved with the information you provided.

Thank you very much

[AndyGDIT]

Can confirm issue with ifgroups. Once we disabled them everything started working again. We have a DD9500 we just installed 7.1.0.5 on and got failures. We are on Veeam 9.5U4b.

Has OP heard anything from Dell/EMC on the issue? I have my Dell technical managers looking at submitting the bug. Id hate to keep this like this since most of the traffic is going down my management interface.

[Trelor]

Dell says it is a Veeam bug Veeam says it is a Dell bug. I have another ticket escalated with Veeam, Dell told me that they did all they can do and it's all green on their end (I am not buying this). The only "fix" that I have since I have MGMT / Backups on different vlans is that I have dns setup like this now (Could also do local hosts file I guess):
Backup Network (Quad 10g)
dd6300-01.blah.com -> 10.0.12.101
dd6300-01.blah.com -> 10.0.12.102
dd6300-01.blah.com -> 10.0.12.103
dd6300-01.blah.com -> 10.0.12.104

MGMT Network (Single 1g)
dd6300-01-MGMT.blah.com -> 10.23.0.100

At least now it kind of randomly picks what interfaces to use. However even the MFR features of Boost with Interface groups do not work.

I have my Veeam pointed to: dd6300-01.blah.com where previously it was dd6300-01-MGMT.blah.com, it is not ideal but at least I have backups running at 10g now. I also have multiple network cards in the server with the vlans for the Backup network / Server network so they are separate so I am not sure that my setup will help you.

[AndyGDIT]

I am using hosts files on my proxies to point to separate 10GB NICs to my DD9500. It seems to be working good so far. Though I dont think its going to be a long term solution.
I am raising this issue with my reps with Dell and I also have a Veeam case (04003509) open with it.

[AndyGDIT]

I have a ticket open with Dell/EMC and they wont help me until we move to V10. I dont think it will help since it technically works now with 9.5.4b. I guess Ill upgrade to V10 once it goes GA this week.

[AndyGDIT]

I finally got Dell/EMC to open a bug case with DDOS 7.1.0.5 and DDBoost.

I upgraded Veeam to V10 GA, and long story short, backups were still going over my management interface with ifgroups disabled. I get error code 5040 access errors with the ifgroups enabled.

I still have the workaround with hosts file entries on all of my proxies to my 10GB interfaces which works fine.

[Trelor]

Thanks just let me know what they say as I would be interested into a fix for this too.

[AndyGDIT]

Its bug 261830 for me.

I will post what I hear from Dell.

[Pamik]

Hello,
Same problem here. We have Veeam 10.0.0.4461 and DataDomain 7.1.0.10. After DD upgrade to 7.1.0.10 Veeam jobs end up wit error 5040. After disabling corresponding Interface Group everything started working again. This is workaround, so I would be interested into a permanent fix. Any updates on this?

[AndyGDIT]

My recommendation is to open a dell service request and reference the bug in my previous post. The more we get on it the faster they may get a fix.

I have been working with support for about a week now since I upgraded to V10 and it didn’t fix the issue. I have given them all they wanted in regards to logs. If I get anything I will be sure to post it.

[cfizz34]

Anyone seen this error?


3/7/2020 7:43:11 AM :: Processing FILESHARE01 Error: Failed to call RPC function 'DDBoostFcWriteFile': Failed to write data to file [[ddbunk-backups.fqdn.com] veeam-su:/Backup-Job-Fileshare01/Backup-Job-Fileshare.vbm_5_tmp ( 1584844802 )]. Offset: [0]. calling system(), returns nonzero. Err: 5040.
3/7/2020 7:43:16 AM :: All VMs have been queued for processing
3/7/2020 7:44:46 AM :: Unable to create synthetic full backup: all OIBs in the current restore point are corrupted and previous restore point is not an increment
3/7/2020 7:44:50 AM :: Job finished with error at 3/7/2020 7:44:50 AM

[cfizz34]

Should I be using the management interface IP (DNS name of the DD) and let VEEAM and DD handle the traffic over the ddboost network or should I be using the ddboost IP (DNS IP for backup traffic/ddboost traffic)?

[AndyGDIT]

What DDOS are you on? There are three people (myself included) on 7.1.0.5 have the same issue. There is a post that may be on the second or third page now show what you should do to get up and running again.

I have an active service request open with Dell. It’s with engineering with an active bug ticket.

[cfizz34]

I just recently upgraded to OS: 7.1.0.10-646016

[cfizz34]

Can you point me in the right direction to where you mentioned to get this running again?