NFS proxy method causes nfs server to disconnect?

[danswartz]

So I am running a CentOS 7.5 storage server. Serves up vsphere 6.x datastore using NFS over 1gb LAN. If I specify the proxy transfer mode to allow NFS, I see frequent messages like this in the CentOS server log:

Oct 21 11:13:50 nas kernel: rpc-srv/tcp: nfsd: sent only 699876 when sending 1048708 bytes - shutting down socket

It reconnects okay, but this happens dozens of times during a backup, which screws the performance to hell and gone. I never see this message for hypervisor driven reads. It's not the 1gb pipe, since I was seeing the same messages using a Mellanox 40gb enet link (I took those out in preparation for newer Mellanox cards on order.) It almost seems like the TCP send window is closing, and because the kernel NFS server uses non-blocking I/O, the result is a short write, causing the message and closed down socket. Has anyone seen anything like this? I'll open a case if need be, but it's working okay in NBD mode for now...

[nielsengelen]

I would suggest opening a case to dig into this performance issue as we'll need more insight on the full setup and maybe additional logs where things may be mentioned. Do post the support case ID for future reference if possible. Thanks!

[danswartz]

Case 03256298 opened.