Bug 1937105

Summary:	/bin/bash missing after kdump started in RAWHIDE
Product:	[Fedora] Fedora	Reporter:	Chung <cchung>
Component:	kexec-tools	Assignee:	Kairui Song <kasong>
Status:	CLOSED RAWHIDE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	rawhide	CC:	awalsh, bhe, kasong, ruyang, sweettea
Target Milestone:	---	Flags:	fedora-admin-xmlrpc: mirror+
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-04-28 20:43:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Chung 2021-03-09 20:35:55 UTC

Description of problem:

/bin/bash missing after starting kdump, might be more.

Version-Release number of selected component (if applicable):

# uname -r
5.10.0-0.rc4.20201117git9c87c9f41245.79.fc34.x86_64


How reproducible:

Steps to Reproduce:
1.yum install kexec-tools
2.systemctl start kdump
3.ls -l /bin/bash

Actual results:

# systemctl start kdump
# ls -l /bin/bash
bash: bash: command not found


Expected results:


Additional info:

Comment 1 Andy Walsh 2021-03-09 20:56:36 UTC

I'm seeing this in journalctl after reproducing.

Mar 09 15:47:34 localhost.localdomain dracut[16140]: *** Generating early-microcode cpio image ***
Mar 09 15:47:34 localhost.localdomain dracut[16140]: *** Store current command line parameters ***
Mar 09 15:47:34 localhost.localdomain dracut[16140]: Stored kernel commandline:
Mar 09 15:47:34 localhost.localdomain dracut[16140]: No dracut internal kernel commandline stored in the initramfs
Mar 09 15:47:34 localhost.localdomain dracut[16140]: *** Install squash loader ***
Mar 09 15:47:34 localhost.localdomain audit[18531]: AVC avc:  denied  { unlink } for  pid=18531 comm="rm" name="modules.builtin" dev="vda1" ino=657461 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=system_u:object_r:modules_dep_t:s0 tclass=file permissive=1
Mar 09 15:47:34 localhost.localdomain audit[18637]: AVC avc:  denied  { read } for  pid=18637 comm="dracut-install" name="modules.softdep" dev="vda1" ino=268211 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=root:object_r:modules_dep_t:s0 tclass=file permissive=1
Mar 09 15:47:34 localhost.localdomain audit[18637]: AVC avc:  denied  { open } for  pid=18637 comm="dracut-install" path="/usr/lib/modules/5.12.0-0.rc2.165.fc35.x86_64/modules.softdep" dev="vda1" ino=268211 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=root:object_r:modul>
Mar 09 15:47:34 localhost.localdomain audit[18637]: AVC avc:  denied  { getattr } for  pid=18637 comm="dracut-install" path="/usr/lib/modules/5.12.0-0.rc2.165.fc35.x86_64/modules.softdep" dev="vda1" ino=268211 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=root:object_r:mo>
Mar 09 15:47:34 localhost.localdomain audit[18642]: AVC avc:  denied  { write } for  pid=18642 comm="mv" name="sbin" dev="vda1" ino=131086 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=dir permissive=1
Mar 09 15:47:34 localhost.localdomain audit[18642]: AVC avc:  denied  { remove_name } for  pid=18642 comm="mv" name="modprobe" dev="vda1" ino=146280 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=dir permissive=1
Mar 09 15:47:34 localhost.localdomain audit[18642]: AVC avc:  denied  { rename } for  pid=18642 comm="mv" name="modprobe" dev="vda1" ino=146280 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=system_u:object_r:bin_t:s0 tclass=lnk_file permissive=1
Mar 09 15:47:34 localhost.localdomain audit[18644]: AVC avc:  denied  { rename } for  pid=18644 comm="mv" name="bash" dev="vda1" ino=138113 scontext=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file permissive=1
Mar 09 15:47:34 localhost.localdomain kdumpctl[18645]: find: 'bash': No such file or directory

<repeats-many-times>

Mar 09 15:47:34 localhost.localdomain kdumpctl[18709]: find: 'bash': No such file or directory
Mar 09 15:47:34 localhost.localdomain audit[18710]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:kdumpctl_t:s0-s15:c0.c1023 pid=18710 comm="dracut-install" exe="/usr/lib/dracut/dracut-install" sig=11 res=1
Mar 09 15:47:34 localhost.localdomain kernel: show_signal_msg: 13 callbacks suppressed
Mar 09 15:47:34 localhost.localdomain kernel: dracut-install[18710]: segfault at 0 ip 00007f0cd0b0dca4 sp 00007ffc8e49a778 error 4 in libc-2.33.9000.so[7f0cd0ab6000+14d000]
Mar 09 15:47:34 localhost.localdomain kernel: Code: 84 00 00 eb 84 90 87 07 83 f8 01 7e ce 45 31 d2 ba 01 00 00 00 be 81 00 00 00 b8 ca 00 00 00 0f 05 eb b8 0f 1f 00 f3 0f 1e fa <8b> 17 f6 47 74 80 0f 84 90 00 00 00 89 d1 81 e1 00 80 00 00 0f 85
Mar 09 15:47:34 localhost.localdomain audit[18712]: AVC avc:  denied  { write } for  pid=18712 comm="systemd-coredum" name="coredump" dev="tmpfs" ino=487 scontext=system_u:system_r:systemd_coredump_t:s15:c0.c1023 tcontext=system_u:object_r:init_var_run_t:s0 tclass=sock_file permissive=1
Mar 09 15:47:34 localhost.localdomain audit[18712]: AVC avc:  denied  { connectto } for  pid=18712 comm="systemd-coredum" path="/run/systemd/coredump" scontext=system_u:system_r:systemd_coredump_t:s15:c0.c1023 tcontext=system_u:system_r:systemd_coredump_t:s0-s15:c0.c1023 tclass=unix_strea>
Mar 09 15:47:34 localhost.localdomain systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=100 op=LOAD
Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=101 op=LOAD
Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=102 op=LOAD
Mar 09 15:47:34 localhost.localdomain systemd[1]: Started Process Core Dump (PID 18712/UID 0).
Mar 09 15:47:34 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0-s15:c0.c1023 msg='unit=systemd-coredump@0-18712-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 09 15:47:34 localhost.localdomain audit[18713]: AVC avc:  denied  { getattr } for  pid=18713 comm="systemd-coredum" path="socket:[38942]" dev="sockfs" ino=38942 scontext=system_u:system_r:systemd_coredump_t:s0-s15:c0.c1023 tcontext=system_u:system_r:systemd_coredump_t:s15:c0.c1023 tcl>
Mar 09 15:47:34 localhost.localdomain audit[18713]: AVC avc:  denied  { write } for  pid=18713 comm="systemd-coredum" path=2F7661722F6C69622F73797374656D642F636F726564756D702F23333330202864656C6574656429 dev="vda1" ino=330 scontext=system_u:system_r:systemd_coredump_t:s0-s15:c0.c1023 tcon>
Mar 09 15:47:34 localhost.localdomain audit[18713]: AVC avc:  denied  { setattr } for  pid=18713 comm="systemd-coredum" name="#7893" dev="vda1" ino=7893 scontext=system_u:system_r:systemd_coredump_t:s0-s15:c0.c1023 tcontext=system_u:object_r:init_var_lib_t:s0 tclass=file permissive=1
Mar 09 15:47:34 localhost.localdomain audit[18713]: AVC avc:  denied  { link } for  pid=18713 comm="systemd-coredum" name="#7893" dev="vda1" ino=7893 scontext=system_u:system_r:systemd_coredump_t:s0-s15:c0.c1023 tcontext=system_u:object_r:init_var_lib_t:s0 tclass=file permissive=1
Mar 09 15:47:34 localhost.localdomain systemd-coredump[18713]: [🡕] Process 18710 (dracut-install) of user 0 dumped core.
                                                                                           
                                                                                           Stack trace of thread 18710:
                                                                                           #0  0x00007f0cd0b0dca4 _IO_feof (libc.so.6 + 0x7dca4)
                                                                                           #1  0x00005609f4d94788 resolve_deps (dracut-install + 0x8788)
                                                                                           #2  0x00005609f4d93b59 dracut_install (dracut-install + 0x7b59)
                                                                                           #3  0x00005609f4d90c42 main (dracut-install + 0x4c42)
                                                                                           #4  0x00007f0cd0ab7b75 __libc_start_main (libc.so.6 + 0x27b75)
                                                                                           #5  0x00005609f4d9114e _start (dracut-install + 0x514e)
Mar 09 15:47:34 localhost.localdomain kdumpctl[16115]: /usr/lib/dracut/dracut-init.sh: line 259: 18710 Segmentation fault      (core dumped) "$DRACUT_INSTALL" ${dracutsysrootdir:+-r "$dracutsysrootdir"} ${initdir:+-D "$initdir"} -a ${loginstall:+-L "$loginstall"} ${DRACUT_RESOLVE_DEPS:+-l>
Mar 09 15:47:34 localhost.localdomain kdumpctl[16115]: dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.JXbeAR/initramfs -a -l /lib64/libgcc_s.so.1 /usr/lib64/libgcc_s.so.1
Mar 09 15:47:34 localhost.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0-s15:c0.c1023 msg='unit=systemd-coredump@0-18712-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 09 15:47:34 localhost.localdomain systemd[1]: systemd-coredump: Deactivated successfully.
Mar 09 15:47:34 localhost.localdomain dracut[16140]: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.JXbeAR/initramfs -a -l /lib64/libgcc_s.so.1 /usr/lib64/libgcc_s.so.1
Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=102 op=UNLOAD
Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=101 op=UNLOAD
Mar 09 15:47:34 localhost.localdomain audit: BPF prog-id=100 op=UNLOAD
Mar 09 15:47:34 localhost.localdomain dracut[16140]: *** Stripping files ***

Comment 2 Andy Walsh 2021-03-09 20:57:45 UTC

Note that SELinux is in permissive mode during my reproduction.

Comment 3 Chung 2021-03-09 21:36:03 UTC

When this happens, not only /bin/bash disappear, /bin/sh link is not there either.
Reboot will not be successful at this time.  Also, I tried to replace /bin/bash
after it disappear, but I ran into some other missing files.  

It seems the problem happens after kdump has started.  /bin/bash was still there 
after I installed kexec-tools.

Comment 4 Baoquan He 2021-03-10 01:11:40 UTC

It's known issue now. We met this in CKI since it failed kdump sysrq-c test case, have opened below bug to track it. 

Bug 1936781 - Kdump service restarting causes /bin/bash being lost

Kairui found it's caused by below dracut commit. Then when kdump service trying to re-generating initramfs of kdump, it will cause the bash disappearing.

commit 41cfdfc4d37b3075fef02b1775e903ebe5a15283
Author: Harald Hoyer <harald>
Date:   Fri Feb 12 13:50:47 2021 +0100

    fix: shellcheck for dracut.sh

Thanks
Baoquan

Comment 5 Kairui Song 2021-04-28 20:43:00 UTC

Closing, latest dracut in rawhide should have fixes this, confirmed with f34.