Case-Nr.04761690 -Crash of Veeam Backup Service with EMC Data Domain as Repository

[Virtuollie]

Hi folks,

does anybody reading here and using DDBoost with a Data Domain as backup repository ever noticed entries in the event log like these:

Code: Select all

Log Name: Application
Source: EMC DataDomain
Date: 03.05.2021 03:38:10
Event ID: 0
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: P3ITINP-vbrc01
Description:
The description for Event ID 0 from source EMC DataDomain cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

libDataDomainoncrpc PANIC:
xdrrec_create: out of memory 

Code: Select all

Log Name: Application
Source: EMC DataDomain
Date: 03.05.2021 03:38:10
Event ID: 0
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: P3ITINP-vbrc01
Description:
The description for Event ID 0 from source EMC DataDomain cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

libDDBoost PANIC:
..ddclddcl_init.c: ddcl_destroy_ddcl: 1646: !(ddcl->marker_auto_detection_subctx_pool != NULL)

Those are followed by a crash of the Veeam Backup Service, which can not be restarted without rebooting the server.
It happens even after all backup jobs finished, with only periodic backup copy jobs running at that time.

Support case has been opened, but I am curious if someone here ever noticed similar issues.
The funny thing is, I have another Veeam Backup server configured identically without these issues.

Regards,
VirtuOllie

[Dima P.]

Hello Oliver,

Thanks for sharing the case! I have a feeling that you might also need to raise a ticket with Dell EMC Data Domain team (at least to investigate these event creation from EMC Data Domain). Regarding the the backup service problems, please keep working with our support team. Cheers!

[Virtuollie]

Yes, Dima,
this might be necessary - although I can not find any issue or alert in the DD logs in DD System Manager.

Thanks for your reply,
Ollie

[foggy]

It happens even after all backup jobs finished, with only periodic backup copy jobs running at that time.

In case this job is targeted to a Data Domain repository, this might be well expected. Might be a valid reaction of the DDBoost client residing on the gateway server when everything crashes.

[Virtuollie]

Support sent me some new DDBoost DLLs to try, that will be included in a future update to Veeam B&R - unfortunately they did not solve my issues.
I will keep you all posted.

[foggy]

Seems I've confused the sequence of events in your post and it looks like the library reaction is what causes our backup service to crash. Have you contacted Dell EMC support for a closer look?

[Virtuollie]

Sorry for answering with delay.
The issue could be solved without the help of Dell EMC.

Unfortunately, the DDBoost DLLs provided by Veeam Support did not help.
I noticed the issue to occurr when only a Backup Copy Job was running, copying VM backups from Repository on Data Domain to an SOBR on offsite Windows Server.
The extents of the target SOBR had no task limits, which caused memory congestion on the backup server, which is also mount server for source repository.

As a result, the MSSQL service failed and eventually the Veeam backup service also.

Setting maximum concurrent tasks to 8 for all extents on the offsite server solved the issue.

Thanks for all answers,
Ollie

[foggy]

Hi Oliver, appreciate the update. Setting the concurrent tasks limits is indeed the best practice to avoid such issues.