EMC DD After upgrade to latest V12 - Error: Failed to call RPC function 'DDBoostFcWriteFile': Failed to open file '

[JTT]

Has anybody encountered this kind of issue?
14:34:35 Failed to pre-process the job Error: Failed to call RPC function 'DDBoostFcWriteFile': Failed to open file 'xxx.vbm_1_tmp'. nothing matched. Err: 5004.
14:34:43 Error: Failed to call RPC function 'DDBoostFcWriteFile': Failed to open file xxx.vbm_1_tmp'. nothing matched. Err: 5004.

This issue seems to be random and not for all backup jobs.

EMC DD is upgraded to latest Veeam supported firmware - 7.11
Case 06196276

[foggy]

Thanks for opening a support case - log review is definitely required here.

[JTT]

What logs - support hasnt asked it. Also is thery any way to escalate the case to higher tier support?

[systemCGV]

Hi,

We have Upgraded our Veeam in 12.0.0.1420 P20230718 Yesterday.
And we have the same behaviour than you.

After the Upgrade I have making Backup/Restore and Replication Test, all was OK.
This Night during our Backup Windows, We had half of our Backup who had Failed with the error :
Failed to pre-process the job Error: Failed to call RPC function 'DDBoostFcWriteFile': Failed to open file 'xxx.vbm_1_tmp'. nothing matched. Err: 5004.

This Morning I have making another Backup Test All was Ok.
So I have run one of our Failed Job and the Job run Fine.

I have Opened a Case : 06259108 (and I have transfered logs)

I hope our Backup will run fine...

[JTT]

Thanks, at least i am not alone with this So maybe Veeam - can you give some more attention to this issue now?

Weird thing is, that if backups fail at night, even after three retrys, i retry in the morning and all are success now.

[foggy]

What logs - support hasnt asked it.

You have already uploaded them upon opening a case.

Also is thery any way to escalate the case to higher tier support?

Sure, you can either ask the engineer or use the Talk to a manager button within the support portal to escalate the case.

[pmichelli]

What DDOS version are you both running?

I have 2 Data Domains with latest V12 patch running on DDOS 7.10 and have not had any issues at all (so far).

[JTT]

I do not know what version we were on, when the issue started, but currently 7.11 as Veeam supported version.

[systemCGV]

Latest Veeam Patch and DDOS 7.10.0.10-1023227

[mboelsch]

Same issue here.

Latest Veeam Patch and DDOS 7.7.5.11-1046187.

[mboelsch]

Case #06259859

I´ve got a fixed DLL from support. I´m going to deploy it tomorrow. Will keep you updated.

[systemCGV]

Yesterday the Support tell us to change our gateway server or to reboot our Datadomain.
We have reboot our datadomain and our veeam server (who is the gateway server).
This night half of our backup failed with the same error.
We have relaunch failed jobs at 5h00 this morning and they successfully finish.

This night while Jobs failing I have tested the Datadomain connectivity/accessbility and all was alright !
And the CPU/ RAM of our veeam server was at 10%.

[KayaTR]

After the Upgrade I have making Backup/Restore and Replication Test, all was OK.
This Night during our Backup Windows, We had half of our Backup who had Failed with the error :
Failed to pre-process the job Error: Failed to call RPC function 'DDBoostFcWriteFile': Failed to open file 'xxx.vbm_1_tmp'. nothing matched. Err: 5004.

We are having the same problem with our client. Did you solve it?

[JTT]

Case 06196276 and today i also got the updated DLL file. Tonights backups will show, if it is fixed or not.

[mboelsch]

First tests with a replaced VeeamDeploymentDll.dll looking good so far.

Hence we only noticed the errors in scheduled jobs, I´ll have to wait for tonights backups to be clear.

[systemCGV]

We are having the same problem with our client. Did you solve it?

Support give us a "fixed" VeeamDeploymentDll.dll. We have replaced it this morning.
Like mboelsch our tests are fine but errors seems to appear the night so we will see tonight if all is fine.

[JTT]

Well i still get the same error, not for all jobs, but rather random, also a new error line is showing up:
25.08.2023 18:33:49 :: Unable to apply retention policy: failed to delete backup files Error: Failed to call RPC function 'DDBoostFcWriteFile': Failed to open file '[xxx] xxx:/ka129lis - xxx.vbm_3_tmp'. nothing matched. Err: 5004.

[ae_group_ag]

We have the same issue and have opened a support case (case #06261582).

We restarted our DataDomains, updated to the latest version of DDOS without luck.
For testing we disabled all backup copy jobs and enabled only one to synchronize the restore points with success.
So as next test we changed the maximum concurrent tasks to 1 in the backup repository settings and enabled all jobs and it works. No more errors.
But for now it takes a lot of time to process all jobs.

For us It looks like a resource management problem with the latest patch.

Hope it will help

[systemCGV]

We have update the DLL and the last backup set has successfully finish.
We had only one backup since friday so we will keeping an eye on our backup.

[KayaTR]

Hello, As a workaround for users who are experiencing the problem, right-clicking on the repository where you get the error, Gateway server, then using the following gateway servers only option, and choosing a suitable gateway server option will ensure that jobs are completed without any problems.
and you need to perform this operation on all repositroys of your error name. By choosing different gateway servers

[simo77]

Hi,
FYI:
we also ran into DDBoost Error 5004. Veeam Support gives us a fix that seems working (just a single DLL file to replace in a certain directory), but we noticed that when a server rescan occurs the original bugged DLL is restored and the fixing DLL is renamed with prefix "old_12.0.0.1420_".
We have just asked support to investigate this.

[systemCGV]

Hi,

We have replaced the DLL one week ago and we don't have any issue since.
Hope all of you will find a solution.

Cordialy,

[fgiannardi]

Hi,
We have the same problem, how can we access the DLL ?
Cordialy,

[Mildur]

Hello Cordialy

Private hotfixes are provided after opening a support case.
Please open a case and let us know the case number.

Best,
Fabian

[tinto1970]

same problem here (one of our customers): opened Case # 06283169

[tinto1970]

.dll recieved and installed: works fine now

[NicHOK]

Same problem here: opened Case: 06288954

[frit0]

Good morning everyone,

Always in the morning I checked mi backups from the last and I had many time errors calls "Failed to pre-process the job Error: Failed to call RPC function 'DDBoostFcWriteFile' Failed to open the file... nothing matched. Err: 5004"

I searched fo this error but I neved found something abou it in the forums.

Do anybody here got this error in the past?

how to solve.

Thanks a lot

[Mildur]

Hi Robert

how to solve.

A support case solves technical issues. We cannot do it over a forum post.
I moved your topic to the existing topic with a similar issue.

Please open a case with our customer support team. You may be affected by a known issue where we already have a hotfix available.
Don't forget to provide the case number with us.

Best,
Fabian

[ibarizz]

Hello!

Just wondering if this hotfix is gonna be available for all the public or a case support is really needed to get it. A public KB would be really appreciated

Cheers!