v10 RTM DataDomain 7.1.0.5?

[cfizz34]

Thanks for posting this. I too am having this issue when trying to use the management DNS name and if I have more than on IP/network in the default ddboost interface group then it fails.

[cfizz34]

Newly created ddboost storage units do not work at all. My original storage unit only works if I have one of the backup network interface IP's added to the interface default group.

[cfizz34]

For now, the only option is to disable the interface group.

[AndyGDIT]

cifzz34

The only way me and the other person got things working again is to use hosts files on the proxies.

Your DataDomain is likely added to Veeam by the management IP so use the hosts file on your proxy(ies) to point the datadomain to one of your 10GB interfaces.

I have multiple proxies in multiple vCenters so I spread out the load over 4 10GB interfaces

Keep the ifgroup disabled when you do this as well.

[foggy]

All, thanks for the feedback so far. We're discussing the situation with Dell EMC and the issue comes down to DD Boost SDK 3.3/DDOS 7.1 combination. We're currently considering the ways of addressing it in our future updates.

[AndyGDIT]

So after Dell/EMC engineering reviewed my logs, it seems that Veeam is using an outdated version of DDBoost. Current installed DDBoost libraries are at 3.3 with V10 and according to Dell support DDOS 7.1.0.5 (now up to 7.1.0.10 and soon to be 7.1.0.11) minimum supported DDBoost is 3.5.

I am working with my Veeam support engineer to see if those libraries can be updated so we can test.

I will update once I hear back from them.

Andy

[foggy]

Hi Andrew, in general, 3.3 is compatible with DDOS 7.1. There's the -2/+4 SDK/firmware compatibility rule according to which 7.1 is the supported release for SDK 3.3 (which is the base for DDOS 6.0 so the +4 compatible versions are: 6.0 – 6.1 – 6.2 – 7.0 - 7.1). Replacing the lib on the gateway might be possible but is something we wouldn't recommend doing in production.

[AndyGDIT]

Ok fair enough, but would Veeam possibly look to update the libraries say in V10 U1? My workaround is working fine right now and I can wait.

[foggy]

We're discussing the options at the moment.

[Gostev]

So after Dell/EMC engineering reviewed my logs, it seems that Veeam is using an outdated version of DDBoost.

I just want to address this comment in more details.

There are two reasons why we've always been using earlier versions of DDBoost SDK:

1. According to the phone home metrics that DELL EMC shared with us, their customer base tends to have a long tail of devices using old DD OS versions, and this tail is significant in volume. Given that DDBoost SDK compatibility rule is -2 to +4 from the matching DD OS version, for maximum coverage we must remain on the lower end of the SDK versions' spectrum.

2. Here at Veeam, we're super conservative about jumping on the "latest and greatest" 3rd party SDK versions, as they often bring major regressions (which can include data corruptions), and usually need time to stabilize. Plus, we have a history of running into such issues with newly released DDBoost SDKs specifically in early days of this integration. So this is another reason why these days, we stick to proven SDK versions.

Our current plan for this issue is to issue a patch that is basically DDBoost SDK 3.4, and provide it to all customers who upgraded to DD OS 7.1.

Please also note that DD OS 7.1 is not officially supported by v10 as per our System Requirements, simply because it was made generally available one month after v10 was shipped. In particular, this means that we obviously did not have a chance to do any testing against DD OS 7.1 prior to shipping v10.

Thanks!

[AndyGDIT]

Anton:

Thanks for the update on this! I will await the patch.

Thanks!

[cfizz34]

Any expected date for the patch?

[Gostev]

Hard to say because the required QC team is currently busy with shipping the new major release.

[foggy]

All, the DD OS 7.1 support patch is now available. Please contact our Customer Support to get it. Thanks!

[AndyGDIT]

Thanks Foggy! My support tech has already reached out to me about it. I will test it later this afternoon.

[foggy]

We will appreciate the feedback since we're planning to include the patch into the upcoming Veeam B&R update depending on its success.

[cfizz34]

DELL EMC is stating this is a VEEAM related issue and not a Data Domain issue and to contact VEEAM for the patch. Is that correct?

[cfizz34]

per Dell....
The fix for this bug doesn't solely depend on us as Veeam 9.5 update 4 and 10 are using a boost library 3.3 that is old and therefore not compatible with DDOS 7.1 when used with if-group. Dell EMC Data Domain Engineering team is working in collaboration with Veeam Engineering team.

I spoke to my customer who got the issue resolved. As per the customer, it was not a Veeam upgrade file but he was provided with files to upgrade just ddboost plugin on the Veeam version 10.0.0.4461.
When he installed the plugin files, ddboost plugin version changed from 3.3.1.3-597934 to 3.4.2.3-615962

[AndyGDIT]

Foggy

Initial tests are successful, we had to make some minor changes to avoid having to do this fix to all of our proxies which my team is planning to do this week so we can keep the automatic selection on the gateway for each of our mtree backup repos.

I will let you know how that goes after we get that done.

Andy

[cfizz34]

I just applied the new CU1 to v10, and I'm sitting now at 10.0.0.4461 but my .dll version did not change.
"C:\Program Files (x86)\Veeam\Backup Transport\x86\ddboost\libDDBoost.dll"
"C:\Program Files (x86)\Veeam\Backup Transport\x64\ddboost\libDDBoost.dll"
Both are still at 3.3.1.3

[foggy]

Hi Chad, CU1 doesn't contain this DDboost SDK patch. As mentioned by both Veeam and Dell EMC, you should contact Veeam support directly to get it. Thanks!

[Pamik]

We will appreciate the feedback since we're planning to include the patch into the upcoming Veeam B&R update depending on its success.

The fix is valid and our issue has been resolved. Thank you.

[AndyGDIT]

We had some issues with the fix. Something in the process is still ignoring the network ifgroups that we have set up for the DD. The fix worked if we made the Backup server the DD Gateway, but that crashed our Veeam application when we tried to backup our entire backup set.

Once I put the workaround back (disable ifgroups and use hosts files on my proxies) everything worked again

I am have been in touch with my tier 2 tech and we are going to discuss it some more later today.

[AndyGDIT]

cfizz34

there should be 3 files in two separate directories (6 total files) that you need to move

[cfizz34]

I applied the files (3 in each folder) but once I enabled the ifgroup on the DD side the re-scan of the repository failed.

[cfizz34]

rescan error: 4/15/2020 10:36:09 AM Warning Failed to import backup path ddboost://name-su01@/|Backup-Job-Fileshare|Backup-Job-Fileshare.vbm Details: Failed to call RPC function 'DDBoostFcReadFile': Failed to read data from the file '[name] veeam-su01:/Backup-Job-Fileshare/Backup-Job-Fileshare.vbm'. Offset: '0'. calling system(), returns nonzero. Err: 5040.

[foggy]

Hi Chad, please investigate with the support engineer who has provided you the patch. I'd check if the appropriate redistributable packages are installed - if not, you can obtain the proper ones along with the hotfix. I would also appreciate if your share your case ID number.

[foggy]

We had some issues with the fix. Something in the process is still ignoring the network ifgroups that we have set up for the DD. The fix worked if we made the Backup server the DD Gateway, but that crashed our Veeam application when we tried to backup our entire backup set.

Andrew, have you replaced the files on all servers that might serve as DDBoost gateways?

[Pamik]

I applied the files (3 in each folder) but once I enabled the ifgroup on the DD side the re-scan of the repository failed.

I had to restart our Veeam B&R machine after installing the hotfix. Please, try rebooting.

[Trelor]

I just got the hotfix myself.