Description of problem: ******************************** When smbd crashes and tries to dump core it gets refused to dump core in /var/log/core and there is no core file generated. The setup is RHEL7 based , the expected location from systemd to dump core is /var/log/systemd/coredump/ and samba's coredump location is /var/log/samba/cores/smbd. The issue is why smbd is trying to dump core in /var/log/core and SELinux is preventing it? We need to first make sure that smbd dumps core in systemd's default location then it will not have SELinux AVC's. The AVC could be present because of the location /var/log/core where smbd doesn't have proper context set to have write access to that location to dump core. the /var/log/messages shows following: Mar 29 09:40:58 dhcp47-91 smbd[19960]: #10 /usr/sbin/smbd(+0x84a1) [0x7f608a2c14a1] Mar 29 09:40:58 dhcp47-91 smbd[19960]: [2016/03/29 09:40:58.794953, 0] ../source3/lib/dumpcore.c:313(dump_core) Mar 29 09:40:58 dhcp47-91 smbd[19960]: unable to change to /var/log/core Mar 29 09:40:58 dhcp47-91 smbd[19960]: refusing to dump core Also observed SELinux AVC's : type=AVC msg=audit(1459244458.801:1506): avc: denied { search } for pid=19961 comm="smbd" name="core" dev="dm-0" ino=68515998 scontext=system_u:system_r:smbd_t:s0 tcontext=system_u:object_r:virt_cache_t:s0 tclass=dir Version-Release number of selected component (if applicable): ******************************************** samba-4.4.0-1.el7rhgs How reproducible: ******************** Always Steps to Reproduce: 1.Setup rhgs+samba4.4 systems 2.Try steps to produce a crash from smbd 3.check for the core file being generated, and /var/log/messages. Actual results: smbd trying to dump core in /var/log/core and is refused to dump core in this location. Expected results: smbd should be able to dump core, also the location of core could be either systemd's default location(/var/log/systemd/coredump) or smbd's default core location(/var/log/samba/cores/smbd) Additional info: *************** With SELinux in permissive mode and changing the location of core , found the core file. On the fedora systems smbd was always able to dump core in /var/log/systemd/coredump without any issues.
Initial analysis: First of all a correction to the bug description. Default systemd coredump location is /var/lib/systemd/coredump. From man sysctl.d(5): . . . /etc/sysctl.d/*.conf /run/sysctl.d/*.conf /usr/lib/sysctl.d/*.conf At boot, systemd-sysctl.service(8) reads configuration files from the above directories to configure sysctl(8) kernel parameters. . . . kernel.core_pattern is one among those parameters. Why it works in Fedora? ----------------------- In fedora, systemd-coredump is used as a helper binary by the kernel when a user space program receives a fatal signal and dumps core. kernel.core_pattern is modified to contain the following with a special conf file named 50-coredump.conf under /usr/lib/sysctl.d/. kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e This will be loaded at boot so as to dump cores @ /var/lib/systemd/coredump. Why it is not working in RHEL 7? ------------------------------- Simple answer: We don't have 50-coredump.conf under /usr/lib/sysctl.d/ with systemd v219 which makes systemd-coredump to process coredumps accordingly. Reason for SELinux AVC in RHEL ------------------------------ Since we don't have the above mentioned setting with RHEL 7, /proc/sys/kernel/core_pattern will contain the text 'core' by default. This just means that dump the core as 'core' in the current working directory. When crash occurred back trace will be written to /var/log/messages. So current working directory will be /var/log and /proc/sys/kernel/core_pattern says to dump the core using the file name 'core' @ /var/log/. Finally SELinux will not allow to dump core directly inside /var/log and the AVC is seen. To confirm the above analysis, I remember Surabhi saying that she was able to get cores under /tmp when she changed /proc/sys/kernel/core_pattern to contain /tmp/core.<pid> or something instead of just 'core'.
Vivek, as discussed, could you please verify (or falsify) anoop's analysis: Are you able to get cores under /tmp when changing /proc/sys/kernel/core_pattern to contain /tmp/core.<pid> or something instead of just 'core'.
Hi all, Small update/correction to my previous analysis on how /proc/sys/kernel/core_pattern is honoured: In order to reproduce the issue I setup a RHEL 7 vm and tried out some exercises. SELinux is enabled with mode as enforcing. [1] Brought up samba via systemctl and issued SIGABRT to smbd process(CIFS mount of a local share). Core was generated under /var/log/samba/cores/smbd/. This is expected. Due to the absence of 50-coredump.conf file in RHEL, /proc/sys/kernel/core_pattern contains the text 'core'. Under this situation dumping core appends '.<PID>' to whatever is given inside /proc/sys/kernel/core_pattern. But in samba source code while setting up the corepath it checks whether /proc/sys/kernel/core_pattern content starts with '/' or not(https://github.com/samba-team/samba/blob/master/source3/lib/dumpcore.c#L150). Since text 'core' does not start with '/' during abort it changes cwd to /var/log/samba/cores/smbd and system dumps as core.<PID> inside it. Note:- If somehow we change /proc/sys/kernel/core_pattern to contain something which includes '/' samba will honour it by changing cwd to that path and system dumps the core. [2] I modified /proc/sys/kernel/core_pattern to contain '/tmp' Now issuing SIGABRT to smbd process resulted in following AVC(which is not similar to what is given in bug description): type=AVC msg=audit(1460552288.236:456): avc: denied { write } for pid=3772 comm="smbd" name="/" dev="dm-1" ino=128 scontext=system_u:system_r:smbd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=dir Reason for this AVC is because smbd is trying to write core file as /tmp.3772(directly under root) which will be denied. [3] Repeated the above by modifying /proc/sys/kernel/core_pattern to contain '/tmp/', an extra slash. Now core was dumped inside /tmp with file name '.<PID>'. Conclusion: ----------- SELinux is doing the right thing here. AVC was jsut a symptom of something that goes wrong in setting up corepaths. /var/log/messages from bug description says the following Mar 29 09:40:58 dhcp47-91 smbd[19960]: unable to change to /var/log/core Mar 29 09:40:58 dhcp47-91 smbd[19960]: refusing to dump core This means that /proc/sys/kernel/core_pattern file on that system during crash might have contained atleast '/var/log/core/' and smbd was denied search permission in that path resulting in the AVC described in the bug.
Hi all, We found the culprit as vdsm package which changes the system-wide /proc/sys/kernel/core_pattern to /var/log/core/core.%p.%t.dump and sets a particular context on /var/log/core in an RHGS installation. Please see the procedure I followed to verify the same. Thanks to Rejy for helping me out with debugging the AVC. [1] Installed RHGS-3.1.2-RHEL-7 iso. [2] # cat /proc/sys/kernel/core_pattern /var/log/core/core.%p.%t.dump [3] Installed samba and did a CIFS mount of a local XFS partition. [4] # kill -6 <mount-smbd-pid> [5] cat /var/log/messages | grep smbd | tail -3 Apr 14 02:42:01 localhost smbd[13215]: [2016/04/14 02:42:01.554528, 0] ../source3/lib/dumpcore.c:313(dump_core) Apr 14 02:42:01 localhost smbd[13215]: unable to change to /var/log/core Apr 14 02:42:01 localhost smbd[13215]: refusing to dump core [6] less /var/log/audit/audit.log | grep smbd | grep AVC type=AVC msg=audit(1460616121.553:618): avc: denied { search } for pid=13215 comm="smbd" name="core" dev="dm-0" ino=35063774 scontext=system_u:system_r:smbd_t:s0 tcontext=system_u:object_r:virt_cache_t:s0 tclass=dir This is the same AVC from bug description. Here in this AVC, tcontext(for /var/log/core) is set as virt_cache_t. [7] In order to find out who creates and sets context to /var/log/core: # rpm -qf /var/log/core vdsm-4.16.30-1.3.el7rhgs.x86_64 # rpm -q --scripts vdsm | less These scripts contain the following snippet. # hack until we replace core dump with abrt if /usr/sbin/selinuxenabled; then /usr/sbin/semanage fcontext -a -t virt_cache_t '/var/log/core(/.*)?' fi /sbin/restorecon -R /var/log/core >/dev/null 2>&1 # hack until we replace core dump with abrt [8] Finally who writes '/var/log/core/core.%p.%t.dump' to /proc/sys/kernel/core_pattern. Further investigation led us to the following init script from vdsm package. # cat /usr/libexec/vdsm/vdsmd_init_common.sh | grep core_pattern echo "/var/log/core/core.%p.%t.dump" > /proc/sys/kernel/core_pattern
This is just a tracker for the vdsm bug https://bugzilla.redhat.com/show_bug.cgi?id=1259402. It has a recent comment that it might be fixed by a rebase of the vdsm component in RHGS.
Noticed that this bug got closed by accident when we were acting on bugs with PM_Score < 0. Opening it back.
Dependent Bug has been closed as wontfix, we still have to check for properly documenting it as a known issue.