Bug 1979816
| Summary: | update to 2.5.4 broke nfs-mountd | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | customercare |
| Component: | nfs-utils | Assignee: | Steve Dickson <steved> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 33 | CC: | bfields, luk.claes, mmatsuya, sam, soap, steved |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | nfs-utils-2.5.4-2.rc3.fc33 nfs-utils-2.5.4-2.rc3.fc34 nfs-utils-2.5.4-2.rc3.fc35 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-09-24 20:32:38 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Hmm... it appears the proc-fs-nfsd.mount service was not run or failed which is called by nfs-mountd.service How reproducible is this.. because it seems to be working every else It's 100 reproduceable as thsi test from 5 minutes ago shows:
[root@XXXXX ~]# dnf update nfs-utils
Letzte Prüfung auf abgelaufene Metadaten: vor 0:12:10 am Di 03 Aug 2021 12:01:09 CEST.
Abhängigkeiten sind aufgelöst.
================================================================================================================================================================================================================================================================================
Package Architecture Version Repository Size
================================================================================================================================================================================================================================================================================
Aktualisieren:
nfs-utils x86_64 1:2.5.4-0.fc33 updates 421 k
Transaktionsübersicht
================================================================================================================================================================================================================================================================================
Aktualisieren 1 Paket
Gesamte Downloadgröße: 421 k
Ist dies in Ordnung? [j/N]: j
Pakete werden heruntergeladen:
nfs-utils-2.5.4-0.fc33.x86_64.rpm 2.5 MB/s | 421 kB 00:00
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Gesamt 1.2 MB/s | 421 kB 00:00
Transaktionsüberprüfung wird ausgeführt
Transaktionsprüfung war erfolgreich.
Transaktion wird getestet
Transaktionstest war erfolgreich.
Transaktion wird ausgeführt
Vorbereitung läuft : 1/1
Ausgeführtes Scriptlet: nfs-utils-1:2.5.4-0.fc33.x86_64 1/1
Ausgeführtes Scriptlet: nfs-utils-1:2.5.4-0.fc33.x86_64 1/2
Aktualisieren : nfs-utils-1:2.5.4-0.fc33.x86_64 1/2
Ausgeführtes Scriptlet: nfs-utils-1:2.5.4-0.fc33.x86_64 1/2
Ausgeführtes Scriptlet: nfs-utils-1:2.5.3-2.fc33.x86_64 2/2
Aufräumen : nfs-utils-1:2.5.3-2.fc33.x86_64 2/2
Ausgeführtes Scriptlet: nfs-utils-1:2.5.3-2.fc33.x86_64 2/2
Überprüfung läuft : nfs-utils-1:2.5.4-0.fc33.x86_64 1/2
Überprüfung läuft : nfs-utils-1:2.5.3-2.fc33.x86_64 2/2
Aktualisiert:
nfs-utils-1:2.5.4-0.fc33.x86_64
Fertig.
[root@XXXXX ~]# systemctl restart nfs
Failed to restart nfs.service: Unit nfs.service not found.
[root@XXXXX ~]# systemctl restart nfs-mountd.service
[root@XXXXX ~]# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
Loaded: loaded (/usr/lib/systemd/system/nfs-mountd.service; static)
Active: failed (thawing) (Result: exit-code) since Tue 2021-08-03 12:13:37 CEST; 5s ago
Process: 519291 ExecStart=/usr/sbin/rpc.mountd (code=exited, status=0/SUCCESS)
Main PID: 519294 (code=exited, status=1/FAILURE)
CPU: 44ms
Aug 03 12:13:37 XXXXX systemd[1]: Starting NFS Mount Daemon...
Aug 03 12:13:37 XXXXX rpc.mountd[519294]: Unable to watch /proc/fs/nfsd/clients: No such file or directory
Aug 03 12:13:37 XXXXX systemd[1]: Started NFS Mount Daemon.
Aug 03 12:13:37 XXXXX systemd[1]: nfs-mountd.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 12:13:37 XXXXX systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.
downgrading to 2.5.3 and the services starts again.
who creates /proc/fs/nfsd/clients?
Is it something the kernel needs to supply?
What's the output of rpc.nfsd? And is the "nfsd" filesystem mounted at /proc/fs/nfsd/? Yes, that directory is created by the kernel (if the nfsd filesystem is mounted); /proc/fs/nfsd/clients was added to kernel 5.3. For compatibility with older kernels, rpc.mountd should be falling back gracefully when it doesn't find /proc/fs/nfsd/clients, rather than exiting with an error. That sounds like an rpc.mountd bug. Until that's fixed, either a kernel upgrade or an nfs-utils downgrade should work around this problem for anyone hitting it. a)
with : 2.5.4
# systemctl status nfs-mountd.service
● nfs-mountd.service - NFS Mount Daemon
Loaded: loaded (/usr/lib/systemd/system/nfs-mountd.service; static)
Active: failed (thawing) (Result: exit-code) since Tue 2021-08-03 18:12:26 CEST; 2min 20s ago
Process: 669483 ExecStart=/usr/sbin/rpc.mountd (code=exited, status=0/SUCCESS)
Main PID: 669486 (code=exited, status=1/FAILURE)
CPU: 46ms
Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: Starting NFS Mount Daemon...
Aug 03 18:12:26 s120.resellerdesktop.de rpc.mountd[669486]: Unable to watch /proc/fs/nfsd/clients: No such file or directory
Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: Started NFS Mount Daemon.
Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: nfs-mountd.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 18:12:26 s120.resellerdesktop.de systemd[1]: nfs-mountd.service: Failed with result 'exit-code'.
# rpc.nfsd -d
rpc.nfsd: knfsd is currently up
b)
no, kernel to old.
c) then pls inform upstream.
d) fine for me, as long as we don't have a security problem in mountd
This just came up in Gentoo as we upgraded to nfs-utils 2.5.4. A user running an older kernel (4.19.198 in this case) found that they couldn't start NFS, manifesting in the same way as this bug (see https://bugs.gentoo.org/808183). This commit seems to be the cause: https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=b5aa8378e1a100898d8e0dd628eb704c68667027. Noting the errors given above (and in the linked bug): >Aug 03 18:12:26 s120.resellerdesktop.de rpc.mountd[669486]: Unable to watch /proc/fs/nfsd/clients: No such file or directory /proc/fs/nfsd/clients was introduced in kernel 5.3 according to https://man7.org/linux/man-pages/man8/nfsdclnts.8.html. Is there a roadmap for how to handle older kernels in nfs-utils here? (In reply to sam from comment #5) > [...] > This commit seems to be the cause: > https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff; > h=b5aa8378e1a100898d8e0dd628eb704c68667027. > I meant https://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=edeb381593d65a4519c71568c969d3e2937ad911, of course! I am not seeing this problem.... kernel: 4.18.16-300.fc29.x86_64 nfs-utils: nfs-utils-2.5.4-2.rc2.fc35.x86_64 # systemctl start nfs-utils # systemctl status nfs-utils ● nfs-utils.service - NFS server and client services Loaded: loaded (/usr/lib/systemd/system/nfs-utils.service; static; vendor preset: disabled) Active: active (exited) since Tue 2021-09-21 14:27:42 EDT; 47s ago Process: 990 ExecStart=/bin/true (code=exited, status=0/SUCCESS) Main PID: 990 (code=exited, status=0/SUCCESS) Sep 21 14:27:42 steved-f29 systemd[1]: Starting NFS server and client services... Sep 21 14:27:42 steved-f29 systemd[1]: Started NFS server and client services. What am I missing? Try : 5.1.20-300.fc30.x86_64 Thats the kernel running at the time, the bug emerged. This the version causing trouble: nfs-utils-1:2.5.4-0.fc33.x86_64 yours is a newer version, which could have solved the issue. How knows. (In reply to customercare from comment #9) > This the version causing trouble: nfs-utils-1:2.5.4-0.fc33.x86_64 > > yours is a newer version, which could have solved the issue. How knows. Well, I spun up a Fedora 30 vm with a kernel of kernel-5.0.9-301.fc30.x86_64 and installed nfs-utils-2.5.4-0.fc33.x86_64... Added '/home *(rw)' to /etc/exports... and the server came up just fine... # rpm -q nfs-utils nfs-utils-2.5.4-0.fc33.x86_64 # uname -r 5.0.9-301.fc30.x86_64 # cat /etc/exports /home *(rw) systemctl start nfs-server systemctl status nfs-server ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled) Active: active (exited) since Wed 2021-09-22 10:10:08 EDT; 2min 19s ago Process: 1941 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) Process: 1942 ExecStart=/usr/sbin/rpc.nfsd (code=exited, status=0/SUCCESS) Process: 1961 ExecStart=/bin/sh -c if systemctl -q is-active gssproxy; then systemctl reload gssproxy ; fi (code=exited, status=0/SUCCESS) Main PID: 1961 (code=exited, status=0/SUCCESS) Sep 22 10:10:08 steved-f30.fs.lab.eng.bos.redhat.com systemd[1]: Starting NFS server and services... Sep 22 10:10:08 steved-f30.fs.lab.eng.bos.redhat.com systemd[1]: Started NFS server and services. So I'm a bit stumped... Maybe it is an SElinux issue? I do have SELINUX=enforcing in /etc/selinux/config Never mind... I see the problem now... # rpc.mountd -F -d all rpc.mountd: Unable to watch /proc/fs/nfsd/clients: No such file or directory FEDORA-2021-21c3211d85 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-21c3211d85 FEDORA-2021-bc48d66739 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2021-bc48d66739 FEDORA-2021-7e9ae4b0fe has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-7e9ae4b0fe FEDORA-2021-21c3211d85 has been pushed to the Fedora 34 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-21c3211d85` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-21c3211d85 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-2021-7e9ae4b0fe has been pushed to the Fedora 33 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-7e9ae4b0fe` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-7e9ae4b0fe See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-2021-bc48d66739 has been pushed to the Fedora 35 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-bc48d66739` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-bc48d66739 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-2021-21c3211d85 has been pushed to the Fedora 34 stable repository. If problem still persists, please make note of it in this bug report. FEDORA-2021-bc48d66739 has been pushed to the Fedora 35 stable repository. If problem still persists, please make note of it in this bug report. confirmed: works. FEDORA-2021-7e9ae4b0fe has been pushed to the Fedora 33 stable repository. If problem still persists, please make note of it in this bug report. |
Bug state set to urgent according to condition "update broke productionsystems" Version-Release number of selected component (if applicable): 2.5.4 Description of problem: The following update: 021-07-07T04:01:45+0200 SUBDEBUG Upgrade: libnfsidmap-1:2.5.4-0.fc33.x86_64 2021-07-07T04:01:45+0200 SUBDEBUG Upgrade: nfs-utils-1:2.5.4-0.fc33.x86_64 2021-07-07T04:01:46+0200 SUBDEBUG Upgraded: nfs-utils-1:2.5.3-2.fc33.x86_64 2021-07-07T04:01:46+0200 SUBDEBUG Upgraded: libnfsidmap-1:2.5.3-2.fc33.x86_64 broke nfs: # systemctl status nfs-mountd ● nfs-mountd.service - NFS Mount Daemon Loaded: loaded (/usr/lib/systemd/system/nfs-mountd.service; static) Active: failed (Result: exit-code) since Wed 2021-07-07 09:20:22 CEST; 9s ago Process: 4925 ExecStart=/usr/sbin/rpc.mountd (code=exited, status=0/SUCCESS) Main PID: 4928 (code=exited, status=1/FAILURE) CPU: 43ms Jul 07 09:20:22 XXX systemd[1]: Starting NFS Mount Daemon... Jul 07 09:20:22 xxx rpc.mountd[4928]: Unable to watch /proc/fs/nfsd/clients: No such file or directory Jul 07 09:20:22 XXX systemd[1]: Started NFS Mount Daemon. Jul 07 09:20:22 xxx systemd[1]: nfs-mountd.service: Main process exited, code=exited, status=1/FAILURE Jul 07 09:20:22 XXX systemd[1]: nfs-mountd.service: Failed with result 'exit-code'. Server could no longer mount shares on this device. Instantly after the downgrade to 2.5.3 nfs started to work again, as mountd started successfully.