Issue with libDDBoost.dll on Windows 2019 & 2022

[mrholm]

Hi all
This is a heads up for anyone who wants to refresh their Data Domain Gateway servers, do NOT upgrade to Windows 2019 or Windows 2022.
We have an incident Case #05236920 after we did a hardware refresh of our gateway proxy servers and on new servers we install Windows 2019 (our standard OS) but when starting to use these new servers VeeamAgent crashes when using the DDboost library.

The answer we get from support yesterday is this:
"This is indeed unintended behavior for the Veeam transport Agent using DDboost libraries on Windows Server 2019 and 2022. Our RnD department is aware of this issue and working on the fix.
In the meanwhile we can only recommend to use earlier versions of Windows Server for connection to the DDboost storage."


To me this is a BIG surprise that Veeam is not able to use any OS later then Windows 2016 as gateway towards Data Domain and to us this means we have to restart our old Gateway servers (soon out of support) and run them with 30% less performance than the new ones have.
How can Veeam missed to upgrade the DDboost DLL to be able to run on Windows 2019? If there was problem with running it on Windows 2022 I would understand but 2019 has been out since 2018 and I guess that Veeam has a lot of customers using Data Domain as target for backups/copies.

This MUST be fixed as soon as possible but in the meanwhile do NOT upgrade your Gateway servers to anything beyond Windows 2016.

[Andreas Neufert]

This topic of adding always latest SDK/libraries is not that simple to answer, as you have to drag support for older DDBoost OS firmware versions along as a significant number of customers run older DataDomain firmwares and cuting support for those is problematic.

But anyway in this case it is a bit different as DDBoost library is not supporting yet Windows 2022 from DellEMC side and we found that the same issue was introduced with later Windows 2019 patch levels. Usually newer OS are not an issue and Veeam is supporting them anyway. We are working with DellEMC on both topics (the issue and Win2022 support).

[mrholm]

Thanks Andreas, does Veeam say this somewhere? We were looking for supported platforms prior to installing new servers but did not find anything for this issue?
It could be good to mention in documentation like https://helpcenter.veeam.com/docs/backu ... ml?ver=110

As now it says in requirements that Windows 2019 is supported as Proxy server and for DD it says:
Dell EMC Data Domain (DD OS version 6.2 to 7.6) with DDBoost license. Both Ethernet and Fibre Channel (FC) connectivity is supported.

Maybe a not saying that it only runs on for Windows 2016 could be good?

[Andreas Neufert]

I agree, we are waiting for an official statement from Dell EMC to got this path. I appologize for the situation for you. Let me share an update here soon.

[foggy]

Hi Mats, indeed, Windows 2022 is not yet supported by DDBoost. Windows 2019, however, was thoroughly tested against Veeam B&R v11 and we haven't experienced any issues like that internally. Also, other customers are successfully using Windows 2019 as gateway servers for DDBoost and haven't reported any issues, which makes us consider the fact that some recent Windows 2019 update introduces the issue with the DDBoost DLL we're using in Veeam B&R v11. The issue itself was acknowledged by Dell EMC as an issue with the OpenSSL library that is integrated with this DDBoost version (7.0) and it is already addressed in later DDBoost versions. We're switching to a newer DDBoost SDK version in v12 to address this situation and are working on verifying it as a hotfix for v11 as well.

[mrholm]

I agree to that Foggy, as seen in our case 05236920 we tried to setup this with Windows 2016 (fully patched) on the new servers instead of 2019 but it fails the same, so a Windows cahange seems to be a valid source of the error. We are a bit worried that the nex upcoming patches (for 2016) that are to be installed in February on our current 2016 servers facing DD will be affected.
So we are glad to test out a hotfica for v11, we are on v11 not v11a yet and are not in plan to upgrade yet.

[foggy]

Thanks, you will be notified of its availability by your support engineer.

[mrholm]

Hi
Some update, we installed a new virtual Windows 2019, patched for December patches and set it up as DD gateway and it works just fine.
We are thinking if there's an issue with the DDboost library and the new generation of CPU we have in new servers. The new CPU are:
Intel Xeon Gold 6342 2.8G, 24C/48T, 11.2GT/s, 36M Cache, Turbo, HT (230W) DDR4-3200
Intel Xeon Gold 6326 2.9G, 16C/32T, 11.2GT/s, 24M Cache, Turbo, HT (185W) DDR4-3200

[mrholm]

Update
We added a 3:rd physical server as DD gateway and that also fails with DDboost, that is also a new server on windows 2019 with following CPU:
Intel Xeon Gold 6334 3.6G, 8C/16T, 11.2GT/s, 18M Cache, Turbo, HT (165W)

The other two servers have same new generation of CPU:
Intel Xeon Gold 6342 2.8G, 24C/48T, 11.2GT/s, 36M Cache, Turbo, HT (230W) DDR4-3200
Intel Xeon Gold 6326 2.9G, 16C/32T, 11.2GT/s, 24M Cache, Turbo, HT (185W) DDR4-3200

The DDboost is only failing in our environment for these new servers, older servers works fine on 2016 and 2019 and also newly installed virtual servers, so maybe start with Dell to investigate CPU/Chipset

[foggy]

Hi Mats, I will pass this info to the Dell EMC eng team but what we've been told was that the issue is caused by an older version of OpenSSL library, without any indication of the hardware specification.

[mrholm]

foggy, I opened a case with Dell today and shared with Veeam support. We have not been able to recreate the issue on any other server then the new servers and the issue is there for windos 2016 and 2019 on these servers. It does not matter what WIndows patches either. The same DDboost library works fine for now 10 other servers, older Dell physicals win2016 and virtual win2016 and 2019.
Let see what Veeam and Dell can do together.

[Andreas Neufert]

Can you pelase share the Dell case number with us here or send it directly to Foggy per PN?

[foggy]

Hi Mats, I have your support case ID from our support team and have already shared it and the considerations you provided with Dell for their investigation. Thanks!

[APerkins182]

Hello

May be slightly unrelated to post but I was curious if there was any issue expected. We are setting up a new data domain which is going to use DDBoost 7.6, but we are going to use the existing Windows 2019 gateway server which currently connects to the old DD running DDBoost 6.2 . Plan is to put old DD in maintenance mode and evactuate backups to new DD

[foggy]

Hi Andy, shouldn't cause any issues unless your gateway also has that recent Windows update affecting DDBoost operation (the issue being discussed in this thread). If it is currently working fine with your existing DDOS 6.2 repository, then you should be fine.

[foggy]

The issue itself was acknowledged by Dell EMC as an issue with the OpenSSL library that is integrated with this DDBoost version (7.0) and it is already addressed in later DDBoost versions. We're switching to a newer DDBoost SDK version in v12 to address this situation and are working on verifying it as a hotfix for v11 as well.

The hotfix for Veeam B&R v11a should be available through our support.

[mrholm]

Yes, we downloaded ythe fic and has been runnning succesfully since Thursday with even better perfomance than before. Thanks a lot for all effeort

[JTShine]

All my backups were failing after migrating VMs to AHV from ESXi. Once I applied this hotfix for Veeam 11a on the Veeam Backup server everything works like a charm now!
Thanks so much for this post - been trying to figure this one out for 3 months

[bbuchanan]

I also recently encountered the issue of VeeamAgent crashing with new hardware as the gateway server for our Data Domain. Support was able to provide me with a hot fix.

I wondered how a change in CPU could cause this problem, and today I learned of Intel’s Data Operand Independent Timing
https://www.intel.com/content/www/us/e ... dance.html, which is discussed on the SecurityNow! Podcast #908, and that does change the timing of some instructions. Maybe not, though, I don’t know.