nfsd: sent only ... shutting down socket

[musv]

Found [topic=1030102]this thread[/topic] today. Since the last kernel update I'm suffering this problem too. But the circumstances are quite different.

It does seem to be during times of "reasonably" heavy loads

Mostly yes. But the last freeze occurred without any noticeable load.

Code: Select all

Sep 18 15:16:53 nas systemd-logind[174]: System is powering down.
Sep 18 15:16:53 nas systemd-logind[174]: Powering Off...
Sep 18 15:16:53 nas systemd-logind[174]: Power key pressed.
Sep 18 15:16:19 nas kernel: sysrq: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memor>
Sep 18 15:16:19 nas kernel: sysrq: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memor>
Sep 18 15:16:14 nas kernel: sysrq: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memor>
Sep 18 15:15:35 nas kernel: rpc-srv/tcp: nfsd: sent only 108 when sending 32876 bytes - shutting down socket

At least the powerbutton still works to bring the system down in a clean way.

Now the difference:

Code: Select all

Linux nas 4.12.8-gentoo #3 SMP Fri Aug 25 11:28:33 CEST 2017 armv7l ARMv7 Processor rev 1 (v7l) Marvell Armada 380/385 (Device Tree) GNU/Linux

It's a Zyxel NSA326. So, surely it's not related to Intel or AMD IOMMU.

Any ideas? With the previous kernel, this freezes never occurred. So I think, the problem is related to a special kernel option or the changed device tree.


EDIT: Added "this thread" link and split from that topic. Due to thread age and differences, a new thread is more useful. Plus, having 0 replies in this thread will likely get some additional attention. --pjp

[musv]

I'm a step forward:

Code: Select all

Sep 18 15:15:35 nas kernel: rpc-srv/tcp: nfsd: sent only 108 when sending 32876 bytes - shutting down socket

This seems to be the source of the problems. Found some older bugs from Redhat. But they did use nfs-utils-1.x and something else from Ubuntu.

Strange.

[musv]

Some more investigation. Got again:

Code: Select all

Sep 27 17:35:37 nas kernel: rpc-srv/tcp: nfsd: sent only 36488 when sending 65644 bytes - shutting down socket

This time the network kept alive. So I could restart the nfs-server. Redhat has an entry to this topic: https://access.redhat.com/solutions/28104. They recommend to update to nfs-utils-1.0.9 or higher. Ok regarding to the fact, Enterprise Linux is always 5 years behind, this solution isn't really useful:

Code: Select all

[I] net-fs/nfs-utils
     Verfügbare Versionen:   1.3.1-r5^t 1.3.4-r1^t (~)2.1.1-r1^t **2.1.2_rc3^t **2.1.2_rc5^t {caps ipv6 kerberos +libmount nfsdcld +nfsidmap +nfsv4 nfsv41 selinux tcpd +uuid}
     Installierte Versionen: 2.1.1-r1^t(12:58:52 25.05.2017)(caps libmount nfsidmap nfsv4 nfsv41 tcpd uuid -ipv6 -kerberos -nfsdcld -selinux)

At least version 2.1.1 was already installed in May of this year. And the problems started only after the last kernel upgrade from 4.10.10 to 4.12.8 and continue with 4.13.2.

The next proposal is to adjust the read buffer values, which are set by the system to:

Code: Select all

cat /proc/sys/net/core/rmem_default
163840
cat /proc/sys/net/core/rmem_max
163840

I don't have any idea, if 160 kb is to low or to high.

My kernel config regarding NFS:

Code: Select all

CONFIG_KERNFS=y
CONFIG_NFS_FS=y
# CONFIG_NFS_V2 is not set
# CONFIG_NFS_V3 is not set
CONFIG_NFS_V4=y
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_PNFS_FILE_LAYOUT=y
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
# CONFIG_NFS_V4_1_MIGRATION is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
# CONFIG_NFSD_BLOCKLAYOUT is not set
# CONFIG_NFSD_SCSILAYOUT is not set
# CONFIG_NFSD_FLEXFILELAYOUT is not set
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_COMMON=y

[musv]

Update:

It seems, as I've at least identified the problem part. Due to the fact, that this error only occurs on my Zyxel NAS it has something to do with this special configuration.

I changed the device tree file (dts) to the previous version and so far there are no error messages anymore. I'll investigate this some time further.

works:

Code: Select all

…
		internal-regs {
			ethernet@70000 {
				status = "okay";
				phy-mode = "rgmii";
                mac-address = [ 1a 07 a8 2c 04 76 ];
				fixed-link {
					speed = <1000>;
					full-duplex;
				};
			};
…

doesn't work:

Code: Select all

…
		internal-regs {
			ethernet@70000 {
				pinctrl-0 = <&ge0_rgmii_pins>;
				pinctrl-names = "default";
				status = "okay";
				phy = <&phy1>;
				phy-mode = "rgmii-id";
                mac-address = [ 1a 07 a8 2c 04 76 ];
			};

But I guess, this is to specific to describe this problem in this forum. We've hacked this device in this thread. So it's better to continue there.

[musv]

Wrote a bug-report to the developers of the mvneta driver:

https://www.spinics.net/lists/netdev/msg461420.html