Bug state set to urgent according to condition "update broke productionsystems" Version-Release number of selected component (if applicable): 2.5.4 Description of problem: The following update: 021-07-07T04:01:45+0200 SUBDEBUG Upgrade: libnfsidmap-1:2.5.4-0.fc33.x86_64 2021-07-07T04:01:45+0200 SUBDEBUG Upgrade: nfs-utils-1:2.5.4-0.fc33.x86_64 2021-07-07T04:01:46+0200 SUBDEBUG Upgraded: nfs-utils-1:2.5.3-2.fc33.x86_64 2021-07-07T04:01:46+0200 SUBDEBUG Upgraded: libnfsidmap-1:2.5.3-2.fc33.x86_64 broke nfs: # systemctl status nfs-mountd ● nfs-mountd.service - NFS Mount Daemon Loaded: loaded (/usr/lib/systemd/system/nfs-mountd.service; static) Active: failed (Result: exit-code) since Wed 2021-07-07 09:20:22 CEST; 9s ago Process: 4925 ExecStart=/usr/sbin/rpc.mountd (code=exited, status=0/SUCCESS) Main PID: 4928 (code=exited, status=1/FAILURE) CPU: 43ms Jul 07 09:20:22 XXX systemd[1]: Starting NFS Mount Daemon... Jul 07 09:20:22 xxx rpc.mountd[4928]: Unable to watch /proc/fs/nfsd/clients: No such file or directory Jul 07 09:20:22 XXX systemd[1]: Started NFS Mount Daemon. Jul 07 09:20:22 xxx systemd[1]: nfs-mountd.service: Main process exited, code=exited, status=1/FAILURE Jul 07 09:20:22 XXX systemd[1]: nfs-mountd.service: Failed with result 'exit-code'. Server could no longer mount shares on this device. Instantly after the downgrade to 2.5.3 nfs started to work again, as mountd started successfully.
Hmm... it appears the proc-fs-nfsd.mount service was not run or failed which is called by nfs-mountd.service How reproducible is this.. because it seems to be working every else
It's 100 reproduceable as thsi test from 5 minutes ago shows: [root@XXXXX ~]# dnf update nfs-utils Letzte Prüfung auf abgelaufene Metadaten: vor 0:12:10 am Di 03 Aug 2021 12:01:09 CEST. Abhängigkeiten sind aufgelöst. ================================================================================================================================================================================================================================================================================ Package Architecture Version Repository Size ================================================================================================================================================================================================================================================================================ Aktualisieren: nfs-utils x86_64 1:2.5.4-0.fc33 updates 421 k Transaktionsübersicht ================================================================================================================================================================================================================================================================================ Aktualisieren 1 Paket Gesamte Downloadgröße: 421 k Ist dies in Ordnung? [j/N]: j Pakete werden heruntergeladen: nfs-utils-2.5.4-0.fc33.x86_64.rpm 2.5 MB/s | 421 kB 00:00 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Gesamt 1.2 MB/s | 421 kB 00:00 Transaktionsüberprüfung wird ausgeführt Transaktionsprüfung war erfolgreich. Transaktion wird getestet Transaktionstest war erfolgreich. Transaktion wird ausgeführt Vorbereitung läuft : 1/1 Ausgeführtes Scriptlet: nfs-utils-1:2.5.4-0.fc33.x86_64 1/1 Ausgeführtes Scriptlet: nfs-utils-1:2.5.4-0.fc33.x86_64 1/2 Aktualisieren : nfs-utils-1:2.5.4-0.fc33.x86_64 1/2 Ausgeführtes Scriptlet: nfs-utils-1:2.5.4-0.fc33.x86_64 1/2 Ausgeführtes Scriptlet: nfs-utils-1:2.5.3-2.fc33.x86_64 2/2 Aufräumen : nfs-utils-1:2.5.3-2.fc33.x86_64 2/2 Ausgeführtes Scriptlet: nfs-utils-1:2.5.3-2.fc33.x86_64 2/2 Überprüfung läuft : nfs-utils-1:2.5.4-0.fc33.x86_64 1/2 Überprüfung läuft : nfs-utils-1:2.5.3-2.fc33.x86_64 2/2 Aktualisiert: nfs-utils-1:2.5.4-0.fc33.x86_64 Fertig. [root@XXXXX ~]# systemctl restart nfs Failed to restart nfs.service: Unit nfs.service not found. [root@XXXXX ~]# systemctl restart nfs-mountd.service [root@XXXXX ~]# systemctl status nfs-mountd.service ● nfs-mountd.service - NFS Mount Daemon Loaded: loaded (/usr/lib/systemd/system/nfs-mountd.service; static) Active: failed (thawing) (Result: exit-code) since Tue 2021-08-03 12:13:37 CEST; 5s ago Process: 519291 ExecStart=/usr/sbin/rpc.mountd (code=exited, status=0/SUCCESS) Main PID: 519294 (code=exited, status=1/FAILURE) CPU: 44ms Aug 03 12:13:37 XXXXX systemd[1]: Starting NFS Mount Daemon... Aug 03 12:13:37 XXXXX rpc.mountd[519294]: Unable to watch /proc/fs/nfsd/clients: No such file or directory Aug 03 12:13:37 XXXXX systemd[1]: Started NFS Mount Daemon. Aug 03 12:13:37 XXXXX systemd[1]: nfs-mountd.service: Main process exited, code=exited, status=1/FAILURE Aug 03 12:13:37 XXXXX systemd[1]: nfs-mountd.service: Failed with result 'exit-code'. downgrading to 2.5.3 and the services starts again. who creates /proc/fs/nfsd/clients? Is it something the kernel needs to supply?
What's the output of rpc.nfsd? And is the "nfsd" filesystem mounted at /proc/fs/nfsd/? Yes, that directory is created by the kernel (if the nfsd filesystem is mounted); /proc/fs/nfsd/clients was added to kernel 5.3. For compatibility with older kernels, rpc.mountd should be falling back gracefully when it doesn't find /proc/fs/nfsd/clients, rather than exiting with an error. That sounds like an rpc.mountd bug. Until that's fixed, either a kernel upgrade or an nfs-utils downgrade should work around this problem for anyone hitting it.
a) with : 2.5.4 # systemctl status nfs-mountd.service ● nfs-mountd.service - NFS Mount Daemon Loaded: loaded (/usr/lib/systemd/system/nfs-mountd.service; static) Active: failed (thawing) (Result: exit-code) since Tue 2021-08-03 18:12:26 CEST; 2min 20s ago Process: 669483 ExecStart=/usr/sbin/rpc.mountd (code=exited, status=0/SUCCESS) Main PID: 669486 (code=exited, status=1/FAILURE) CPU: 46ms Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: Starting NFS Mount Daemon... Aug 03 18:12:26 s120.resellerdesktop.de rpc.mountd[669486]: Unable to watch /proc/fs/nfsd/clients: No such file or directory Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: Started NFS Mount Daemon. Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: nfs-mountd.service: Main process exited, code=exited, status=1/FAILURE Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: nfs-mountd.service: Failed with result 'exit-code'. # rpc.nfsd -d rpc.nfsd: knfsd is currently up b) no, kernel to old. c) then pls inform upstream. d) fine for me, as long as we don't have a security problem in mountd
This just came up in Gentoo as we upgraded to nfs-utils 2.5.4. A user running an older kernel (4.19.198 in this case) found that they couldn't start NFS, manifesting in the same way as this bug (see https://bugs.gentoo.org/808183). This commit seems to be the cause: https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=b5aa8378e1a100898d8e0dd628eb704c68667027. Noting the errors given above (and in the linked bug): >Aug 03 18:12:26 s120.resellerdesktop.de rpc.mountd[669486]: Unable to watch /proc/fs/nfsd/clients: No such file or directory /proc/fs/nfsd/clients was introduced in kernel 5.3 according to https://man7.org/linux/man-pages/man8/nfsdclnts.8.html. Is there a roadmap for how to handle older kernels in nfs-utils here?
(In reply to sam from comment #5) > [...] > This commit seems to be the cause: > https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff; > h=b5aa8378e1a100898d8e0dd628eb704c68667027. > I meant https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=edeb381593d65a4519c71568c969d3e2937ad911, of course!
I am not seeing this problem.... kernel: 4.18.16-300.fc29.x86_64 nfs-utils: nfs-utils-2.5.4-2.rc2.fc35.x86_64 # systemctl start nfs-utils # systemctl status nfs-utils ● nfs-utils.service - NFS server and client services Loaded: loaded (/usr/lib/systemd/system/nfs-utils.service; static; vendor preset: disabled) Active: active (exited) since Tue 2021-09-21 14:27:42 EDT; 47s ago Process: 990 ExecStart=/bin/true (code=exited, status=0/SUCCESS) Main PID: 990 (code=exited, status=0/SUCCESS) Sep 21 14:27:42 steved-f29 systemd[1]: Starting NFS server and client services... Sep 21 14:27:42 steved-f29 systemd[1]: Started NFS server and client services. What am I missing?
Try : 5.1.20-300.fc30.x86_64 Thats the kernel running at the time, the bug emerged.
This the version causing trouble: nfs-utils-1:2.5.4-0.fc33.x86_64 yours is a newer version, which could have solved the issue. How knows.
(In reply to customercare from comment #9) > This the version causing trouble: nfs-utils-1:2.5.4-0.fc33.x86_64 > > yours is a newer version, which could have solved the issue. How knows. Well, I spun up a Fedora 30 vm with a kernel of kernel-5.0.9-301.fc30.x86_64 and installed nfs-utils-2.5.4-0.fc33.x86_64... Added '/home *(rw)' to /etc/exports... and the server came up just fine... # rpm -q nfs-utils nfs-utils-2.5.4-0.fc33.x86_64 # uname -r 5.0.9-301.fc30.x86_64 # cat /etc/exports /home *(rw) systemctl start nfs-server systemctl status nfs-server ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled) Active: active (exited) since Wed 2021-09-22 10:10:08 EDT; 2min 19s ago Process: 1941 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) Process: 1942 ExecStart=/usr/sbin/rpc.nfsd (code=exited, status=0/SUCCESS) Process: 1961 ExecStart=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi (code=exited, status=0/SUCCESS) Main PID: 1961 (code=exited, status=0/SUCCESS) Sep 22 10:10:08 steved-f30.fs.lab.eng.bos.redhat.com systemd[1]: Starting NFS server and services... Sep 22 10:10:08 steved-f30.fs.lab.eng.bos.redhat.com systemd[1]: Started NFS server and services. So I'm a bit stumped... Maybe it is an SElinux issue? I do have SELINUX=enforcing in /etc/selinux/config
Never mind... I see the problem now... # rpc.mountd -F -d all rpc.mountd: Unable to watch /proc/fs/nfsd/clients: No such file or directory
FEDORA-2021-21c3211d85 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-21c3211d85
FEDORA-2021-bc48d66739 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-bc48d66739
FEDORA-2021-7e9ae4b0fe has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-7e9ae4b0fe
FEDORA-2021-21c3211d85 has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-21c3211d85` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-21c3211d85 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2021-7e9ae4b0fe has been pushed to the Fedora 33 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-7e9ae4b0fe` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-7e9ae4b0fe See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2021-bc48d66739 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-bc48d66739` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-bc48d66739 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
FEDORA-2021-21c3211d85 has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report.
FEDORA-2021-bc48d66739 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report.
confirmed: works.
FEDORA-2021-7e9ae4b0fe has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report.