Comprehensive data protection for all workloads
cfizz34
Expert
Posts: 128
Liked: 14 times
Joined: Jul 02, 2010 2:57 pm
Full Name: Chad
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by cfizz34 »

Thanks for posting this. I too am having this issue when trying to use the management DNS name and if I have more than on IP/network in the default ddboost interface group then it fails.
cfizz34
Expert
Posts: 128
Liked: 14 times
Joined: Jul 02, 2010 2:57 pm
Full Name: Chad
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by cfizz34 »

Newly created ddboost storage units do not work at all. My original storage unit only works if I have one of the backup network interface IP's added to the interface default group.
cfizz34
Expert
Posts: 128
Liked: 14 times
Joined: Jul 02, 2010 2:57 pm
Full Name: Chad
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by cfizz34 »

For now, the only option is to disable the interface group.
AndyGDIT
Enthusiast
Posts: 68
Liked: 9 times
Joined: Nov 14, 2011 3:15 pm
Full Name: Andrew Frye
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by AndyGDIT »

cifzz34

The only way me and the other person got things working again is to use hosts files on the proxies.

Your DataDomain is likely added to Veeam by the management IP so use the hosts file on your proxy(ies) to point the datadomain to one of your 10GB interfaces.

I have multiple proxies in multiple vCenters so I spread out the load over 4 10GB interfaces

Keep the ifgroup disabled when you do this as well.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by foggy »

All, thanks for the feedback so far. We're discussing the situation with Dell EMC and the issue comes down to DD Boost SDK 3.3/DDOS 7.1 combination. We're currently considering the ways of addressing it in our future updates.
AndyGDIT
Enthusiast
Posts: 68
Liked: 9 times
Joined: Nov 14, 2011 3:15 pm
Full Name: Andrew Frye
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by AndyGDIT »

So after Dell/EMC engineering reviewed my logs, it seems that Veeam is using an outdated version of DDBoost. Current installed DDBoost libraries are at 3.3 with V10 and according to Dell support DDOS 7.1.0.5 (now up to 7.1.0.10 and soon to be 7.1.0.11) minimum supported DDBoost is 3.5.

I am working with my Veeam support engineer to see if those libraries can be updated so we can test.

I will update once I hear back from them.

Andy
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by foggy »

Hi Andrew, in general, 3.3 is compatible with DDOS 7.1. There's the -2/+4 SDK/firmware compatibility rule according to which 7.1 is the supported release for SDK 3.3 (which is the base for DDOS 6.0 so the +4 compatible versions are: 6.0 – 6.1 – 6.2 – 7.0 - 7.1). Replacing the lib on the gateway might be possible but is something we wouldn't recommend doing in production.
AndyGDIT
Enthusiast
Posts: 68
Liked: 9 times
Joined: Nov 14, 2011 3:15 pm
Full Name: Andrew Frye
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by AndyGDIT »

Ok fair enough, but would Veeam possibly look to update the libraries say in V10 U1? My workaround is working fine right now and I can wait.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by foggy »

We're discussing the options at the moment.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by Gostev »

AndyGDIT wrote: Mar 12, 2020 5:29 pmSo after Dell/EMC engineering reviewed my logs, it seems that Veeam is using an outdated version of DDBoost.
I just want to address this comment in more details.

There are two reasons why we've always been using earlier versions of DDBoost SDK:

1. According to the phone home metrics that DELL EMC shared with us, their customer base tends to have a long tail of devices using old DD OS versions, and this tail is significant in volume. Given that DDBoost SDK compatibility rule is -2 to +4 from the matching DD OS version, for maximum coverage we must remain on the lower end of the SDK versions' spectrum.

2. Here at Veeam, we're super conservative about jumping on the "latest and greatest" 3rd party SDK versions, as they often bring major regressions (which can include data corruptions), and usually need time to stabilize. Plus, we have a history of running into such issues with newly released DDBoost SDKs specifically in early days of this integration. So this is another reason why these days, we stick to proven SDK versions.

Our current plan for this issue is to issue a patch that is basically DDBoost SDK 3.4, and provide it to all customers who upgraded to DD OS 7.1.

Please also note that DD OS 7.1 is not officially supported by v10 as per our System Requirements, simply because it was made generally available one month after v10 was shipped. In particular, this means that we obviously did not have a chance to do any testing against DD OS 7.1 prior to shipping v10.

Thanks!
AndyGDIT
Enthusiast
Posts: 68
Liked: 9 times
Joined: Nov 14, 2011 3:15 pm
Full Name: Andrew Frye
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by AndyGDIT »

Anton:

Thanks for the update on this! I will await the patch.

Thanks!
cfizz34
Expert
Posts: 128
Liked: 14 times
Joined: Jul 02, 2010 2:57 pm
Full Name: Chad
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by cfizz34 »

Any expected date for the patch?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by Gostev »

Hard to say because the required QC team is currently busy with shipping the new major release.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by foggy » 1 person likes this post

All, the DD OS 7.1 support patch is now available. Please contact our Customer Support to get it. Thanks!
AndyGDIT
Enthusiast
Posts: 68
Liked: 9 times
Joined: Nov 14, 2011 3:15 pm
Full Name: Andrew Frye
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by AndyGDIT »

Thanks Foggy! My support tech has already reached out to me about it. I will test it later this afternoon.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by foggy »

We will appreciate the feedback since we're planning to include the patch into the upcoming Veeam B&R update depending on its success.
cfizz34
Expert
Posts: 128
Liked: 14 times
Joined: Jul 02, 2010 2:57 pm
Full Name: Chad
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by cfizz34 »

DELL EMC is stating this is a VEEAM related issue and not a Data Domain issue and to contact VEEAM for the patch. Is that correct?
cfizz34
Expert
Posts: 128
Liked: 14 times
Joined: Jul 02, 2010 2:57 pm
Full Name: Chad
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by cfizz34 »

per Dell....
The fix for this bug doesn't solely depend on us as Veeam 9.5 update 4 and 10 are using a boost library 3.3 that is old and therefore not compatible with DDOS 7.1 when used with if-group. Dell EMC Data Domain Engineering team is working in collaboration with Veeam Engineering team.

I spoke to my customer who got the issue resolved. As per the customer, it was not a Veeam upgrade file but he was provided with files to upgrade just ddboost plugin on the Veeam version 10.0.0.4461.
When he installed the plugin files, ddboost plugin version changed from 3.3.1.3-597934 to 3.4.2.3-615962
AndyGDIT
Enthusiast
Posts: 68
Liked: 9 times
Joined: Nov 14, 2011 3:15 pm
Full Name: Andrew Frye
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by AndyGDIT » 1 person likes this post

Foggy

Initial tests are successful, we had to make some minor changes to avoid having to do this fix to all of our proxies which my team is planning to do this week so we can keep the automatic selection on the gateway for each of our mtree backup repos.

I will let you know how that goes after we get that done.

Andy
cfizz34
Expert
Posts: 128
Liked: 14 times
Joined: Jul 02, 2010 2:57 pm
Full Name: Chad
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by cfizz34 »

I just applied the new CU1 to v10, and I'm sitting now at 10.0.0.4461 but my .dll version did not change.
"C:\Program Files (x86)\Veeam\Backup Transport\x86\ddboost\libDDBoost.dll"
"C:\Program Files (x86)\Veeam\Backup Transport\x64\ddboost\libDDBoost.dll"
Both are still at 3.3.1.3
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by foggy »

Hi Chad, CU1 doesn't contain this DDboost SDK patch. As mentioned by both Veeam and Dell EMC, you should contact Veeam support directly to get it. Thanks!
Pamik
Service Provider
Posts: 4
Liked: 1 time
Joined: Jul 19, 2016 9:10 am
Full Name: Pawel Mikulicz
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by Pamik » 1 person likes this post

foggy wrote: Apr 10, 2020 12:23 pm We will appreciate the feedback since we're planning to include the patch into the upcoming Veeam B&R update depending on its success.
The fix is valid and our issue has been resolved. Thank you.
AndyGDIT
Enthusiast
Posts: 68
Liked: 9 times
Joined: Nov 14, 2011 3:15 pm
Full Name: Andrew Frye
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by AndyGDIT »

We had some issues with the fix. Something in the process is still ignoring the network ifgroups that we have set up for the DD. The fix worked if we made the Backup server the DD Gateway, but that crashed our Veeam application when we tried to backup our entire backup set.

Once I put the workaround back (disable ifgroups and use hosts files on my proxies) everything worked again

I am have been in touch with my tier 2 tech and we are going to discuss it some more later today.
AndyGDIT
Enthusiast
Posts: 68
Liked: 9 times
Joined: Nov 14, 2011 3:15 pm
Full Name: Andrew Frye
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by AndyGDIT »

cfizz34

there should be 3 files in two separate directories (6 total files) that you need to move
cfizz34
Expert
Posts: 128
Liked: 14 times
Joined: Jul 02, 2010 2:57 pm
Full Name: Chad
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by cfizz34 »

I applied the files (3 in each folder) but once I enabled the ifgroup on the DD side the re-scan of the repository failed.
cfizz34
Expert
Posts: 128
Liked: 14 times
Joined: Jul 02, 2010 2:57 pm
Full Name: Chad
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by cfizz34 »

rescan error: 4/15/2020 10:36:09 AM Warning Failed to import backup path ddboost://name-su01@/|Backup-Job-Fileshare|Backup-Job-Fileshare.vbm Details: Failed to call RPC function 'DDBoostFcReadFile': Failed to read data from the file '[name] veeam-su01:/Backup-Job-Fileshare/Backup-Job-Fileshare.vbm'. Offset: '0'. calling system(), returns nonzero. Err: 5040.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by foggy »

Hi Chad, please investigate with the support engineer who has provided you the patch. I'd check if the appropriate redistributable packages are installed - if not, you can obtain the proper ones along with the hotfix. I would also appreciate if your share your case ID number.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by foggy »

AndyGDIT wrote: Apr 15, 2020 11:55 am We had some issues with the fix. Something in the process is still ignoring the network ifgroups that we have set up for the DD. The fix worked if we made the Backup server the DD Gateway, but that crashed our Veeam application when we tried to backup our entire backup set.
Andrew, have you replaced the files on all servers that might serve as DDBoost gateways?
Pamik
Service Provider
Posts: 4
Liked: 1 time
Joined: Jul 19, 2016 9:10 am
Full Name: Pawel Mikulicz
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by Pamik »

cfizz34 wrote: Apr 15, 2020 3:28 pm I applied the files (3 in each folder) but once I enabled the ifgroup on the DD side the re-scan of the repository failed.
I had to restart our Veeam B&R machine after installing the hotfix. Please, try rebooting.
Trelor
Enthusiast
Posts: 47
Liked: 15 times
Joined: Apr 27, 2015 6:02 pm
Contact:

Re: v10 RTM DataDomain 7.1.0.5?

Post by Trelor »

I just got the hotfix myself.
Post Reply

Who is online

Users browsing this forum: No registered users and 214 guests