Bug 1322672 - SMB: smbd is not able to dump core file in default location set by vdsm when SELinux is enabled
Summary: SMB: smbd is not able to dump core file in default location set by vdsm when ...
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: samba
Version: rhgs-3.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Anoop C S
QA Contact: Vivek Das
URL:
Whiteboard:
Depends On: 1259402
Blocks: 1311843
TreeView+ depends on / blocked
 
Reported: 2016-03-31 05:27 UTC by surabhi
Modified: 2020-01-11 02:23 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Known Issue
Doc Text:
When vdsm and abrt's ccpp addon are installed alongside each other, vdsmd overwrites abrt's core dump configuration in /proc/sys/kernel/core_pattern. This prevents Samba from generating core dumps due to SELinux search denial on new coredump location set by vdsmd. To work around this issue, execute the following steps: 1. Disable core dumps in /etc/vdsm/vdsm.conf: core_dump_enable = false 2. Restart the abrt-ccpp and smb services: # systemctl restart abrt-ccpp # systemctl restart smb
Clone Of:
Environment:
Last Closed: 2018-04-16 18:18:07 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description surabhi 2016-03-31 05:27:05 UTC
Description of problem:
********************************

When smbd crashes and tries to dump core it gets refused to dump core in /var/log/core and there is no core file generated. 

The setup is RHEL7 based , the expected location from systemd to dump core is 
/var/log/systemd/coredump/ and samba's coredump location is /var/log/samba/cores/smbd.

The issue is why smbd is trying to dump core in /var/log/core and SELinux is preventing it?

We need to first make sure that smbd dumps core in systemd's default location then it will not have SELinux AVC's. 
The AVC could be present because of the location /var/log/core where smbd doesn't have proper context set to have write access to that location to dump core.

the /var/log/messages shows following:

Mar 29 09:40:58 dhcp47-91 smbd[19960]:   #10 /usr/sbin/smbd(+0x84a1) [0x7f608a2c14a1]
Mar 29 09:40:58 dhcp47-91 smbd[19960]: [2016/03/29 09:40:58.794953, 0] ../source3/lib/dumpcore.c:313(dump_core)
Mar 29 09:40:58 dhcp47-91 smbd[19960]:  unable to change to /var/log/core
Mar 29 09:40:58 dhcp47-91 smbd[19960]:  refusing to dump core 

Also observed SELinux AVC's :


type=AVC msg=audit(1459244458.801:1506): avc:  denied  { search } for  pid=19961 comm="smbd" name="core" dev="dm-0" ino=68515998 scontext=system_u:system_r:smbd_t:s0 tcontext=system_u:object_r:virt_cache_t:s0 tclass=dir


Version-Release number of selected component (if applicable):
********************************************
samba-4.4.0-1.el7rhgs

How reproducible:
********************
Always


Steps to Reproduce:
1.Setup rhgs+samba4.4 systems
2.Try steps to produce a crash from smbd
3.check for the core file being generated, and /var/log/messages.

Actual results:

smbd trying to dump core in /var/log/core and is refused to dump core in this location.


Expected results:
smbd should be able to dump core, also the location of core could be either systemd's default location(/var/log/systemd/coredump) or smbd's default core location(/var/log/samba/cores/smbd)

Additional info:
***************
With SELinux in permissive mode and changing the location of core , found the core file.
On the fedora systems smbd was always able to dump core in /var/log/systemd/coredump without any issues.

Comment 3 Anoop C S 2016-04-13 10:09:34 UTC
Initial analysis:

First of all a correction to the bug description. Default systemd coredump location is /var/lib/systemd/coredump.

From man sysctl.d(5):
. . .
/etc/sysctl.d/*.conf
/run/sysctl.d/*.conf
/usr/lib/sysctl.d/*.conf

At boot, systemd-sysctl.service(8) reads configuration files from the above directories to configure sysctl(8) kernel parameters.
. . .

kernel.core_pattern is one among those parameters.

Why it works in Fedora?
-----------------------
In fedora, systemd-coredump is used as a helper binary by the kernel when a user space program receives a fatal signal and dumps core. kernel.core_pattern is modified to contain the following with a special conf file named 50-coredump.conf under /usr/lib/sysctl.d/.

kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e

This will be loaded at boot so as to dump cores @ /var/lib/systemd/coredump.

Why it is not working in RHEL 7?
-------------------------------
Simple answer: We don't have 50-coredump.conf under /usr/lib/sysctl.d/ with systemd v219 which makes systemd-coredump to process coredumps accordingly.

Reason for SELinux AVC in RHEL
------------------------------
Since we don't have the above mentioned setting with RHEL 7, /proc/sys/kernel/core_pattern will contain the text 'core' by default. This just means that dump the core as 'core' in the current working directory. When crash occurred back trace will be written to /var/log/messages. So current working directory will be /var/log and /proc/sys/kernel/core_pattern says to dump the core using the file name 'core' @ /var/log/. Finally SELinux will not allow to dump core directly inside /var/log and the AVC is seen.

To confirm the above analysis, I remember Surabhi saying that she was able to get cores under /tmp when she changed /proc/sys/kernel/core_pattern to contain /tmp/core.<pid> or something instead of just 'core'.

Comment 4 Michael Adam 2016-04-13 10:30:07 UTC
Vivek, as discussed, could you please verify (or falsify) anoop's analysis:

Are you able to get cores under /tmp when changing /proc/sys/kernel/core_pattern to contain /tmp/core.<pid> or something instead of just 'core'.

Comment 5 Anoop C S 2016-04-13 16:16:29 UTC
Hi all,

Small update/correction to my previous analysis on how /proc/sys/kernel/core_pattern is honoured:

In order to reproduce the issue I setup a RHEL 7 vm and tried out some exercises. SELinux is enabled with mode as enforcing.

[1] Brought up samba via systemctl and issued SIGABRT to smbd process(CIFS mount of a local share). Core was generated under /var/log/samba/cores/smbd/.

This is expected. Due to the absence of 50-coredump.conf file in RHEL, /proc/sys/kernel/core_pattern contains the text 'core'. Under this situation dumping core appends '.<PID>' to whatever is given inside /proc/sys/kernel/core_pattern. But in samba source code while setting up the corepath it checks whether /proc/sys/kernel/core_pattern content starts with '/' or not(https://github.com/samba-team/samba/blob/master/source3/lib/dumpcore.c#L150). Since text 'core' does not start with '/' during abort it changes cwd to /var/log/samba/cores/smbd and system dumps as core.<PID> inside it.

Note:-
If somehow we change /proc/sys/kernel/core_pattern to contain something which includes '/' samba will honour it by changing cwd to that path and system dumps the core.

[2] I modified /proc/sys/kernel/core_pattern to contain '/tmp'

Now issuing SIGABRT to smbd process resulted in following AVC(which is not similar to what is given in bug description):

type=AVC msg=audit(1460552288.236:456): avc:  denied  { write } for  pid=3772 comm="smbd" name="/" dev="dm-1" ino=128 scontext=system_u:system_r:smbd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=dir

Reason for this AVC is because smbd is trying to write core file as /tmp.3772(directly under root) which will be denied.

[3] Repeated the above by modifying /proc/sys/kernel/core_pattern to contain '/tmp/', an extra slash.

Now core was dumped inside /tmp with file name '.<PID>'.


Conclusion:
-----------
SELinux is doing the right thing here. AVC was jsut a symptom of something that goes wrong in setting up corepaths.

/var/log/messages from bug description says the following

Mar 29 09:40:58 dhcp47-91 smbd[19960]:  unable to change to /var/log/core
Mar 29 09:40:58 dhcp47-91 smbd[19960]:  refusing to dump core

This means that /proc/sys/kernel/core_pattern file on that system during crash might have contained atleast '/var/log/core/' and smbd was denied search permission in that path resulting in the AVC described in the bug.

Comment 6 Anoop C S 2016-04-14 06:58:00 UTC
Hi all,

We found the culprit as vdsm package which changes the system-wide /proc/sys/kernel/core_pattern to /var/log/core/core.%p.%t.dump and sets a particular context on /var/log/core in an RHGS installation. Please see the procedure I followed to verify the same.

Thanks to Rejy for helping me out with debugging the AVC.

[1] Installed RHGS-3.1.2-RHEL-7 iso.

[2] # cat /proc/sys/kernel/core_pattern
/var/log/core/core.%p.%t.dump

[3] Installed samba and did a CIFS mount of a local XFS partition.

[4] # kill -6 <mount-smbd-pid>

[5] cat /var/log/messages | grep smbd | tail -3
Apr 14 02:42:01 localhost smbd[13215]: [2016/04/14 02:42:01.554528,  0] ../source3/lib/dumpcore.c:313(dump_core)
Apr 14 02:42:01 localhost smbd[13215]:  unable to change to /var/log/core
Apr 14 02:42:01 localhost smbd[13215]:  refusing to dump core

[6] less /var/log/audit/audit.log | grep smbd | grep AVC
type=AVC msg=audit(1460616121.553:618): avc:  denied  { search } for  pid=13215 comm="smbd" name="core" dev="dm-0" ino=35063774 scontext=system_u:system_r:smbd_t:s0 tcontext=system_u:object_r:virt_cache_t:s0 tclass=dir

This is the same AVC from bug description. Here in this AVC, tcontext(for /var/log/core) is set as virt_cache_t.

[7] In order to find out who creates and sets context to /var/log/core:
# rpm -qf /var/log/core
vdsm-4.16.30-1.3.el7rhgs.x86_64
# rpm -q --scripts vdsm | less
These scripts contain the following snippet.
# hack until we replace core dump with abrt
if /usr/sbin/selinuxenabled; then
    /usr/sbin/semanage fcontext -a -t virt_cache_t '/var/log/core(/.*)?'
fi
/sbin/restorecon -R /var/log/core >/dev/null 2>&1
# hack until we replace core dump with abrt

[8] Finally who writes '/var/log/core/core.%p.%t.dump' to /proc/sys/kernel/core_pattern. Further investigation led us to the following init script from vdsm package.
# cat /usr/libexec/vdsm/vdsmd_init_common.sh | grep core_pattern
echo "/var/log/core/core.%p.%t.dump" > /proc/sys/kernel/core_pattern

Comment 8 Michael Adam 2018-04-09 10:21:07 UTC
This is just a tracker for the vdsm bug
https://bugzilla.redhat.com/show_bug.cgi?id=1259402.

It has a recent comment that it might be fixed by a rebase of the vdsm component in RHGS.

Comment 10 Amar Tumballi 2018-04-17 11:08:22 UTC
Noticed that this bug got closed by accident when we were acting on bugs with PM_Score < 0. Opening it back.

Comment 11 Guenther Deschner 2018-11-20 11:15:20 UTC
Dependent Bug has been closed as wontfix, we still have to check for properly documenting it as a known issue.


Note You need to log in before you can comment on or make changes to this bug.