NFC server is busy - Restoring individual VM files

[SAG]

Hello,

We are desperately trying to restore invidual files (Vmx...) from a DELL Data Domain VE backup to our vSphere infrastructure but whenever the file is big (like all VMDK files), it stops saying :
- " NFC Server is busy - Delivery of a FILE_PUT message has failed".

- Case number is #05594260
- vSphere with 3x 6.7 ESXi hosts
- Dell Data domain VE 7.1.0.5 OS as repository
- Dell Equalogic 4100 & 4210 as ISCI storage for our vSphere infrastructure
- Veeam Backup & Replication v11 freshly installed (case opened with 9.5 but the upgrade done today didn't change anything)
- Backup server already rebooted
- Antivirus files & folder whitelist already done respecting Veeam specific #KB
- Management Agent rebooted on 3 hosts
- Mutiple VM tested with same results
- Multiple different restore from different backup has same results
- Only failing using individual VM file restore method, every other methods works flawlessly (FLR, Full VM rest., individual disk rest.)

We never tried to restore individual files from a VM such as VMX file before. But we need it for a specific project.
When restoring using this method, everything works until large VMDK file. So it is not a complete fail, it is only failing when processing larger files. The actual VMDK we're testing is "only" 40Gb.

We have opened a ticket with VMWare but they didn't find anything and asked us to open a ticket with Veeam. However we are repying to each others questions once a day and nothing seems to go forward or getting a more technical aspect beside log analysis. This has began since August now

We have read all Google results about NFC server is busy but nothing helped us. Can anyone give us hints or some help ?
Many thanks
Adrien

[PetrM]

Hello,

As far as I remember, this error may appear if ESXi is overloaded and does not reply to a specific NFC call to put a file to a datastore. I guess more information about the root cause can be found in VMware logs, for example in hostd or vpxa (I assume you have vCenter). I guess VMware support engineer asked to open a case with Veeam as he needs to get more technical insight about how does our application interact with vSphere. I suggest to continue working with our and VMware support teams to find the solution. I see the case with us was opened just one week ago and from my perspective more time is needed to investigate this infrastructural issue.

Thanks!

[virtuallaborer]

Were you able to resolve this issue? I am running into the same issue. Running 6.7 and Veaam 11 as well.

[PetrM]

Hi Alex,

I doubt that we can fix the issue at the level of our code. Most likely it comes from vSphere infrastructure. As stated in the previous post, it makes sense to check hostd and vpxa logs to obtain more information regarding the error. However, I'd let our support team investigate this issue deeper and look for potential workarounds. Maybe they will need to involve VMware support as well. Please share a support case ID for our reference.

Thanks!