Description of problem: /bin/bash missing after starting kdump, might be more. Version-Release number of selected component (if applicable): # uname -r 5.10.0-0.rc4.20201117git9c87c9f41245.79.fc34.x86_64 How reproducible: Steps to Reproduce: 1.yum install kexec-tools 2.systemctl start kdump 3.ls -l /bin/bash Actual results: # systemctl start kdump # ls -l /bin/bash bash: bash: command not found Expected results: Additional info:
I'm seeing this in journalctl after reproducing. Mar 09 15:47:34 localhost.localdomain dracut[16140]: *** Generating early-microcode cpio image *** Mar 09 15:47:34 localhost.localdomain dracut[16140]: *** Store current command line parameters *** Mar 09 15:47:34 localhost.localdomain dracut[16140]: Stored kernel commandline: Mar 09 15:47:34 localhost.localdomain dracut[16140]: No dracut internal kernel commandline stored in the initramfs Mar 09 15:47:34 localhost.localdomain dracut[16140]: *** Install squash loader *** Mar 09 15:47:34 localhost.localdomain audit[18531]: AVC avc: denied { unlink } for pid=18531 comm="rm" name="modules.builtin" dev="vda1" ino=657461 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file permissive=1 Mar 09 15:47:34 localhost.localdomain audit[18637]: AVC avc: denied { read } for pid=18637 comm="dracut-install" name="modules.softdep" dev="vda1" ino=268211 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=root:object_r:modules_dep_t:s0 tclass=file permissive=1 Mar 09 15:47:34 localhost.localdomain audit[18637]: AVC avc: denied { open } for pid=18637 comm="dracut-install" path="/usr/lib/modules/5.12.0-0.rc2.165.fc35.x86_64/modules.softdep" dev="vda1" ino=268211 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=root:object_r:modul> Mar 09 15:47:34 localhost.localdomain audit[18637]: AVC avc: denied { getattr } for pid=18637 comm="dracut-install" path="/usr/lib/modules/5.12.0-0.rc2.165.fc35.x86_64/modules.softdep" dev="vda1" ino=268211 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=root:object_r:mo> Mar 09 15:47:34 localhost.localdomain audit[18642]: AVC avc: denied { write } for pid=18642 comm="mv" name="sbin" dev="vda1" ino=131086 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=dir permissive=1 Mar 09 15:47:34 localhost.localdomain audit[18642]: AVC avc: denied { remove_name } for pid=18642 comm="mv" name="modprobe" dev="vda1" ino=146280 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=dir permissive=1 Mar 09 15:47:34 localhost.localdomain audit[18642]: AVC avc: denied { rename } for pid=18642 comm="mv" name="modprobe" dev="vda1" ino=146280 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=lnk_file permissive=1 Mar 09 15:47:34 localhost.localdomain audit[18644]: AVC avc: denied { rename } for pid=18644 comm="mv" name="bash" dev="vda1" ino=138113 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=1 Mar 09 15:47:34 localhost.localdomain kdumpctl[18645]: find: 'bash': No such file or directory <repeats-many-times> Mar 09 15:47:34 localhost.localdomain kdumpctl[18709]: find: 'bash': No such file or directory Mar 09 15:47:34 localhost.localdomain audit[18710]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 pid=18710 comm="dracut-install" exe="/usr/lib/dracut/dracut-install" sig=11 res=1 Mar 09 15:47:34 localhost.localdomain kernel: show_signal_msg: 13 callbacks suppressed Mar 09 15:47:34 localhost.localdomain kernel: dracut-install[18710]: segfault at 0 ip 00007f0cd0b0dca4 sp 00007ffc8e49a778 error 4 in libc-2.33.9000.so[7f0cd0ab6000+14d000] Mar 09 15:47:34 localhost.localdomain kernel: Code: 84 00 00 eb 84 90 87 07 83 f8 01 7e ce 45 31 d2 ba 01 00 00 00 be 81 00 00 00 b8 ca 00 00 00 0f 05 eb b8 0f 1f 00 f3 0f 1e fa <8b> 17 f6 47 74 80 0f 84 90 00 00 00 89 d1 81 e1 00 80 00 00 0f 85 Mar 09 15:47:34 localhost.localdomain audit[18712]: AVC avc: denied { write } for pid=18712 comm="systemd-coredum" name="coredump" dev="tmpfs" ino=487 scontext=system_u:system_r:systemd_coredump_t:s15:c0.c1023 tcontext=system_u:object_r:init_var_run_t:s0 tclass=sock_file permissive=1 Mar 09 15:47:34 localhost.localdomain audit[18712]: AVC avc: denied { connectto } for pid=18712 comm="systemd-coredum" path="/run/systemd/coredump" scontext=system_u:system_r:systemd_coredump_t:s15:c0.c1023 tcontext=system_u:system_r:systemd_coredump_t:s0-s15:c0.c1023 tclass=unix_strea> Mar 09 15:47:34 localhost.localdomain systemd[1]: Created slice system-systemd\x2dcoredump.slice. Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=100 op=LOAD Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=101 op=LOAD Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=102 op=LOAD Mar 09 15:47:34 localhost.localdomain systemd[1]: Started Process Core Dump (PID 18712/UID 0). Mar 09 15:47:34 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0-s15:c0.c1023 msg='unit=systemd-coredump@0-18712-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Mar 09 15:47:34 localhost.localdomain audit[18713]: AVC avc: denied { getattr } for pid=18713 comm="systemd-coredum" path="socket:[38942]" dev="sockfs" ino=38942 scontext=system_u:system_r:systemd_coredump_t:s0-s15:c0.c1023 tcontext=system_u:system_r:systemd_coredump_t:s15:c0.c1023 tcl> Mar 09 15:47:34 localhost.localdomain audit[18713]: AVC avc: denied { write } for pid=18713 comm="systemd-coredum" path=2F7661722F6C69622F73797374656D642F636F726564756D702F23333330202864656C6574656429 dev="vda1" ino=330 scontext=system_u:system_r:systemd_coredump_t:s0-s15:c0.c1023 tcon> Mar 09 15:47:34 localhost.localdomain audit[18713]: AVC avc: denied { setattr } for pid=18713 comm="systemd-coredum" name="#7893" dev="vda1" ino=7893 scontext=system_u:system_r:systemd_coredump_t:s0-s15:c0.c1023 tcontext=system_u:object_r:init_var_lib_t:s0 tclass=file permissive=1 Mar 09 15:47:34 localhost.localdomain audit[18713]: AVC avc: denied { link } for pid=18713 comm="systemd-coredum" name="#7893" dev="vda1" ino=7893 scontext=system_u:system_r:systemd_coredump_t:s0-s15:c0.c1023 tcontext=system_u:object_r:init_var_lib_t:s0 tclass=file permissive=1 Mar 09 15:47:34 localhost.localdomain systemd-coredump[18713]: [🡕] Process 18710 (dracut-install) of user 0 dumped core. Stack trace of thread 18710: #0 0x00007f0cd0b0dca4 _IO_feof (libc.so.6 + 0x7dca4) #1 0x00005609f4d94788 resolve_deps (dracut-install + 0x8788) #2 0x00005609f4d93b59 dracut_install (dracut-install + 0x7b59) #3 0x00005609f4d90c42 main (dracut-install + 0x4c42) #4 0x00007f0cd0ab7b75 __libc_start_main (libc.so.6 + 0x27b75) #5 0x00005609f4d9114e _start (dracut-install + 0x514e) Mar 09 15:47:34 localhost.localdomain kdumpctl[16115]: /usr/lib/dracut/dracut-init.sh: line 259: 18710 Segmentation fault (core dumped) "$DRACUT_INSTALL" ${dracutsysrootdir:+-r "$dracutsysrootdir"} ${initdir:+-D "$initdir"} -a ${loginstall:+-L "$loginstall"} ${DRACUT_RESOLVE_DEPS:+-l> Mar 09 15:47:34 localhost.localdomain kdumpctl[16115]: dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.JXbeAR/initramfs -a -l /lib64/libgcc_s.so.1 /usr/lib64/libgcc_s.so.1 Mar 09 15:47:34 localhost.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0-s15:c0.c1023 msg='unit=systemd-coredump@0-18712-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Mar 09 15:47:34 localhost.localdomain systemd[1]: systemd-coredump: Deactivated successfully. Mar 09 15:47:34 localhost.localdomain dracut[16140]: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.JXbeAR/initramfs -a -l /lib64/libgcc_s.so.1 /usr/lib64/libgcc_s.so.1 Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=102 op=UNLOAD Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=101 op=UNLOAD Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=100 op=UNLOAD Mar 09 15:47:34 localhost.localdomain dracut[16140]: *** Stripping files ***
Note that SELinux is in permissive mode during my reproduction.
When this happens, not only /bin/bash disappear, /bin/sh link is not there either. Reboot will not be successful at this time. Also, I tried to replace /bin/bash after it disappear, but I ran into some other missing files. It seems the problem happens after kdump has started. /bin/bash was still there after I installed kexec-tools.
It's known issue now. We met this in CKI since it failed kdump sysrq-c test case, have opened below bug to track it. Bug 1936781 - Kdump service restarting causes /bin/bash being lost Kairui found it's caused by below dracut commit. Then when kdump service trying to re-generating initramfs of kdump, it will cause the bash disappearing. commit 41cfdfc4d37b3075fef02b1775e903ebe5a15283 Author: Harald Hoyer <harald> Date: Fri Feb 12 13:50:47 2021 +0100 fix: shellcheck for dracut.sh Thanks Baoquan
Closing, latest dracut in rawhide should have fixes this, confirmed with f34.