Bug 2099256
| Summary: | kdump dump to nfs fails | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Matej Marušák <mmarusak> |
| Component: | dracut | Assignee: | Pavel Valena <pvalena> |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | high | ||
| Version: | 36 | CC: | bhe, coxu, dracut-maint-list, jamacku, jonathan, lnykryn, luk.claes, pvalena, ruyang, ryncsn, steved |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | Flags: | ruyang:
needinfo-
ltao: needinfo- fedora-admin-xmlrpc: mirror+ |
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-07-01 01:07:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Matej Marušák
2022-06-20 12:08:42 UTC
Forgot to mention, while booting X1 we see:
[ 2.203647] systemd[1]: Reached target remote-fs-pre.target - Preparation for Remote File Systems.
[ 2.204734] systemd[1]: Mounting kdumproot.mount - /kdumproot...
[ 2.205713] systemd[1]: dracut-pre-mount.service - dracut pre-mount hook was skipped because all trigger condition checks failed.
[ 2.207193] systemd[1]: Reached target initrd-root-fs.target - Initrd Root File System.
[ OK ] Reached target initrd-root…get - Initrd Root File System.
[ 2.210848] systemd[1]: Starting initrd-parse-etc.service - Reload Configuration from the Real Root...
Starting initrd-parse-etc.…onfiguration from the Real Root...
[ 2.217995] systemd[1]: Reloading.
[ 2.323573] FS-Cache: Loaded
[ 2.281620] mount[417]: mount.nfs: No such device
[ 2.288337] systemd[1]: /usr/lib/systemd/system/kdump-capture.service:23: Standard output type syslog is obsolete, automatically updating to journal. Please update your unit file, and consider removing the setting altogether.
[ 2.290485] systemd[1]: /usr/lib/systemd/system/kdump-capture.service:24: Standard output type syslog+console is obsolete, automatically updating to journal+console. Please update your unit file, and consider removing the setting altogether.
[ 2.330516] systemd[1]: kdumproot.mount: Mount process exited, code=exited, status=32/n/a
[ 2.331437] systemd[1]: kdumproot.mount: Failed with result 'exit-code'.
[ 2.334214] systemd[1]: Failed to mount kdumproot.mount - /kdumproot.
[FAILED] Failed to mount kdumproot.mount - /kdumproot.
See 'systemctl status kdumproot.mount' for details.
Hi Matej, the comment #1 looks like the kdump kernel bootup log. Could you attach the whole kernel log? It could be some networking issue, either in the scripts or some device driver issue, anyway kernel log will be helpful. Moved to kexec-tools component, we can move back to nfs if it is a nfs bug later. nfs-utils added a new file "/usr/lib/modprobe.d/50-nfs.conf" in rc6 patch [1], which contains lines as: install sunrpc /sbin/modprobe --ignore-install sunrpc $CMDLINE_OPTS && /sbin/sysctl -q --pattern sunrpc --system However /sbin/sysctl is not exist in kdump initramfs image, which will fail during dracut-pre-udev: [ 8.834385] dracut-pre-udev[366]: sh: line 1: /sbin/sysctl: No such file or directory [ 8.844594] dracut-pre-udev[365]: modprobe: ERROR: libkmod/libkmod-module.c:990 command_do() Error running install command '/sbin/modprobe --ignore-install sunrpc && /sbin/sysctl -q --pattern sunrpc --system' for module sunrpc: retcode 127 [ 8.867523] dracut-pre-udev[365]: modprobe: ERROR: could not insert 'sunrpc': Invalid argument Thus nfs modules are not loaded before kdump-capture service starts. Here is the result of lsmod before kdump, we can see no nfs modules presents: [ 27.506599] kdump.sh[599]: Module Size Used by [ 27.514522] kdump.sh[599]: lockd 122880 0 [ 27.521524] kdump.sh[599]: grace 16384 1 lockd [ 27.529523] kdump.sh[599]: fscache 372736 0 [ 27.536530] kdump.sh[599]: netfs 57344 1 fscache [ 27.544533] kdump.sh[599]: crct10dif_pclmul 16384 1 [ 27.551530] kdump.sh[599]: crc32_pclmul 16384 0 [ 27.558531] kdump.sh[599]: crc32c_intel 24576 0 [ 27.565536] kdump.sh[599]: ghash_clmulni_intel 16384 0 [ 27.572534] kdump.sh[599]: ice 851968 0 [ 27.579550] kdump.sh[599]: tg3 192512 0 [ 27.586546] kdump.sh[599]: mgag200 40960 0 [ 27.593544] kdump.sh[599]: sunrpc 651264 2 lockd [ 27.601550] kdump.sh[599]: ipmi_devintf 20480 0 [ 27.608545] kdump.sh[599]: ipmi_msghandler 122880 1 ipmi_devintf [ 27.616546] kdump.sh[599]: overlay 151552 1 [ 27.623529] kdump.sh[599]: squashfs 69632 1 [ 27.630515] kdump.sh[599]: loop 32768 2 As a result, when mount.nfs without nfs modules, it will report errors as: mount.nfs: No such device A quick fix is to append the following line to kdump.conf, then kdump works fine: extra_bins /sbin/sysctl I will work out a better way for formal fix. [1]: https://src.fedoraproject.org/rpms/nfs-utils/c/d6281e4f6ed7560f723a9fbba5ecae7f329078f9?branch=rawhide Hi Tao, Thanks! good finding. Sounds like sysctl is needed in dracut nfs module, so not only a kdump issue if people use nfs in initramfs they will have this bug, so probably the right component should be "dracut"? (In reply to Dave Young from comment #2) > Hi Matej, the comment #1 looks like the kdump kernel bootup log. Could you > attach the whole kernel log? It could be some networking issue, either in > the scripts or some device driver issue, anyway kernel log will be helpful. > > Moved to kexec-tools component, we can move back to nfs if it is a nfs bug > later. Matej, since Tao has identified the root cause, so please ignore the above request. (In reply to Dave Young from comment #4) > Hi Tao, > > Thanks! good finding. Sounds like sysctl is needed in dracut nfs module, > so not only a kdump issue if people use nfs in initramfs they will have this > bug, so probably the right component should be "dracut"? Yes, I agree it's better to be fixed in dracut, should the bz be assigned to dracut team? Thanks, Tao Liu Sorry, un-needinfo from bhe (In reply to ltao from comment #6) > (In reply to Dave Young from comment #4) > > Hi Tao, > > > > Thanks! good finding. Sounds like sysctl is needed in dracut nfs module, > > so not only a kdump issue if people use nfs in initramfs they will have this > > bug, so probably the right component should be "dracut"? > > Yes, I agree it's better to be fixed in dracut, should the bz be assigned to > dracut team? > Yes, just reassigned. thanks! Possibly related to: https://github.com/dracutdevs/dracut/issues/1857 FEDORA-2022-38325154c4 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-38325154c4 This should be fixed in https://bodhi.fedoraproject.org/updates/FEDORA-2022-38325154c4 . See https://bugzilla.redhat.com/show_bug.cgi?id=2100668 FEDORA-2022-38325154c4 has been pushed to the Fedora 36 testing repository. Soon you'll be able to install the update with the following command: `sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2022-38325154c4` You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-38325154c4 See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates. FEDORA-2022-38325154c4 has been pushed to the Fedora 36 stable repository. If problem still persists, please make note of it in this bug report. |