Bug 638511 - dirsrv-admin segfaults if selinux is enforced
Summary: dirsrv-admin segfaults if selinux is enforced
Status: CLOSED CURRENTRELEASE
Alias: None
Product: 389
Classification: Retired
Component: Admin
Version: 1.2.6
Hardware: All
OS: Linux
high
high
Target Milestone: ---
Assignee: Nathan Kinder
QA Contact: Viktor Ashirov
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: 434915 389_1.2.8
TreeView+ depends on / blocked
 
Reported: 2010-09-29 08:17 UTC by Peter Bieringer
Modified: 2015-12-07 16:45 UTC (History)
7 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2015-12-07 16:45:37 UTC


Attachments (Terms of Use)
threads stack traces (38.14 KB, text/plain)
2011-01-04 21:53 UTC, Nathan Kinder
no flags Details
AVC messages (2.83 KB, text/plain)
2011-01-04 22:44 UTC, Nathan Kinder
no flags Details
Patch (1.04 KB, patch)
2011-01-04 23:07 UTC, Nathan Kinder
nhosoi: review+
Details | Diff

Description Peter Bieringer 2010-09-29 08:17:00 UTC
Description of problem:

/etc/init.d/dirsrv-admin segfaults if selinux is enforced

Version-Release number of selected component (if applicable):
# rpm -qa |grep 389
389-ds-1.2.1-1.el5
389-admin-1.1.11-1.el5
389-admin-console-doc-1.1.5-1.el5
389-ds-base-1.2.6-1.el5
389-dsgw-1.1.5-1.el5
389-ds-console-1.2.3-1.el5
389-ds-console-doc-1.2.3-1.el5
389-adminutil-1.1.8-4.el5
389-console-1.1.4-1.el5
389-admin-console-1.1.5-1.el5


How reproducible:
Mostly always, on one system on each second start.

Steps to Reproduce:
1. /etc/init.d/dirsrv-admin start
  
Actual results:
/var/log/dirsrv/admin-serv/error

[Wed Sep 29 08:08:54 2010] [notice] SELinux policy enabled; httpd running as context user_u:system_r:httpd_t:s0
[Wed Sep 29 08:08:55 2010] [notice] Access Host filter is: *.prod.venyon.local
[Wed Sep 29 08:08:55 2010] [notice] Access Address filter is: *
[Wed Sep 29 08:08:56 2010] [notice] Access Host filter is: *.prod.venyon.local
[Wed Sep 29 08:08:56 2010] [notice] Access Address filter is: *
[Wed Sep 29 08:08:56 2010] [notice] Apache/2.2 configured -- resuming normal operations
[Wed Sep 29 08:08:57 2010] [notice] child pid 8614 exit signal Segmentation fault (11)
[Wed Sep 29 08:08:58 2010] [notice] Access Host filter is: *.prod.venyon.local
[Wed Sep 29 08:08:58 2010] [notice] Access Address filter is: *
[Wed Sep 29 08:08:59 2010] [notice] child pid 8684 exit signal Segmentation fault (11)
[Wed Sep 29 08:09:00 2010] [notice] Access Host filter is: *.prod.venyon.local
[Wed Sep 29 08:09:00 2010] [notice] Access Address filter is: *
[Wed Sep 29 08:09:01 2010] [notice] child pid 8751 exit signal Segmentation fault (11)
[Wed Sep 29 08:09:02 2010] [notice] Access Host filter is: *.prod.venyon.local
[Wed Sep 29 08:09:02 2010] [notice] Access Address filter is: *
[Wed Sep 29 08:09:03 2010] [notice] child pid 8818 exit signal Segmentation fault (11)
[Wed Sep 29 08:09:04 2010] [notice] Access Host filter is: *.prod.venyon.local
[Wed Sep 29 08:09:04 2010] [notice] Access Address filter is: *
[Wed Sep 29 08:09:04 2010] [notice] caught SIGTERM, shutting down


Expected results:
Not such result


Additional info:

/var/log/audit.log tells:
type=ANOM_ABEND msg=audit(1285747846.609:3349): auid=1000 uid=99 gid=99 ses=403 subj=user_u:system_r:httpd_t:s0 pid=9150 comm="httpd.worker" sig=11


I have already applied workaround from https://bugzilla.redhat.com/show_bug.cgi?id=552419#c21 (port 9830 issue) to be able to start, and also I've tried the semodule command from https://bugzilla.redhat.com/show_bug.cgi?id=624310

But this won't help.

BTW: looks like something is very strange in addition:

# pwd
/root
# /etc/init.d/dirsrv-admin start
Starting dirsrv-admin:
                                                           [  OK  ]
=> segfaulting


# pwd
/var/lib/dirsrv/slapd-myinstance
# /etc/init.d/dirsrv-admin start
Starting dirsrv-admin:
                                                           [FAILED]
Log:

[Wed Sep 29 08:13:45 2010] [notice] SELinux policy enabled; httpd running as context user_u:system_r:httpd_t:s0
[Wed Sep 29 08:13:46 2010] [error] Unable to change directory to /var/lib/dirsrv/slapd-myinstance

Why depends it on the current pwd, whether the initscript will start dirsrv-admin or not.


Perhaps this maillist entry relates on the same issue: http://www.spinics.net/lists/fedora-directory/msg12144.html

Comment 1 Nathan Kinder 2010-09-30 14:53:26 UTC
Do any AVC messages related to dirsrv or dirsrv-admin show up in /var/log/audit/audit when you run 'service dirsrv-admin start'?

Comment 2 Peter Bieringer 2010-09-30 20:04:35 UTC
Unfortunately, no AVC messages appear besides the one shown above. But I know that SELinux is able to have "silent denies", which are not shown by default in log. But I don't remember, how to display them...

Comment 5 Nathan Kinder 2010-10-12 16:31:53 UTC
If you can reproduce the crash, I would appreciate using gdb to get a stack trace.  To make the stack trace useful, the 389-admin debuginfo would need to be installed.

Comment 7 Massimiliano 2010-11-11 15:11:36 UTC
Just the same problem when SELinux is enforced. Putting SELinux in permissive mode solves the problem.

Comment 8 Rich Megginson 2010-11-11 15:25:05 UTC
(In reply to comment #7)
> Just the same problem when SELinux is enforced. Putting SELinux in permissive
> mode solves the problem.

Are you running RHEL5 or CentOS 5?

Comment 9 Massimiliano 2010-11-11 18:43:59 UTC
(In reply to comment #8)
> Are you running RHEL5 or CentOS 5?

CentOS release 5.5

Comment 10 Nathan Kinder 2010-11-11 20:49:17 UTC
(In reply to comment #7)
> Just the same problem when SELinux is enforced. Putting SELinux in permissive
> mode solves the problem.

Do you see any AVCs in the audit log (/var/log/audit/audit)?

Comment 11 Peter Bieringer 2010-11-12 15:59:43 UTC
I got some spare time(In reply to comment #5)
> If you can reproduce the crash, I would appreciate using gdb to get a stack
> trace.  To make the stack trace useful, the 389-admin debuginfo would need to
> be installed.

Unfortunately debuginfo.centos.org is out-of-date:

Could not find debuginfo for main pkg: 389-admin-1.1.11-1.el5.i386
Could not find debuginfo pkg for dependency package 389-adminutil-1.1.8-4.el5.i386
Could not find debuginfo pkg for dependency package glibc-2.5-49.el5_5.7.i386

Therefore I can't debug further on as requested

Comment 12 Rich Megginson 2010-11-12 16:49:46 UTC
(In reply to comment #11)
> I got some spare time(In reply to comment #5)
> > If you can reproduce the crash, I would appreciate using gdb to get a stack
> > trace.  To make the stack trace useful, the 389-admin debuginfo would need to
> > be installed.
> 
> Unfortunately debuginfo.centos.org is out-of-date:
> 
> Could not find debuginfo for main pkg: 389-admin-1.1.11-1.el5.i386
> Could not find debuginfo pkg for dependency package
> 389-adminutil-1.1.8-4.el5.i386
> Could not find debuginfo pkg for dependency package glibc-2.5-49.el5_5.7.i386
> 
> Therefore I can't debug further on as requested

try 

yum install --enablerepo=epel-debuginfo 389-admin-debuginfo

Comment 14 Massimiliano 2010-11-17 17:43:38 UTC
(In reply to comment #10)
> Do you see any AVCs in the audit log (/var/log/audit/audit)?

When I restart the dirsrv-admin service with SELinux enforced:

$/etc/init.d/dirsrv-admin restart

I see the following messages:

$tailf /var/log/audit/audit.log
type=ANOM_ABEND msg=audit(1290015232.285:21): auid=0 uid=501 gid=501 ses=2 subj=root:system_r:httpd_t:s0 pid=2400 comm="httpd.worker" sig=11
type=ANOM_ABEND msg=audit(1290015234.298:22): auid=0 uid=501 gid=501 ses=2 subj=root:system_r:httpd_t:s0 pid=2470 comm="httpd.worker" sig=11
type=ANOM_ABEND msg=audit(1290015236.307:23): auid=0 uid=501 gid=501 ses=2 subj=root:system_r:httpd_t:s0 pid=2537 comm="httpd.worker" sig=11
type=ANOM_ABEND msg=audit(1290015238.342:24): auid=0 uid=501 gid=501 ses=2 subj=root:system_r:httpd_t:s0 pid=2604 comm="httpd.worker" sig=11
...

Comment 15 Peter Bieringer 2010-11-19 15:31:48 UTC
While I was not able to start the ds with gdb, at least I found that it depends on the SELinux context, whether it crashes or not.

-rwxr-xr-x  root root system_u:object_r:initrc_exec_t:s0 /etc/init.d/dirsrv-admin
-rwxr-xr-x  root root system_u:object_r:dirsrvadmin_exec_t:s0 /usr/sbin/start-ds-admin

If scripts are copied and have e.g.

-rwxr-xr-x  root root root:object_r:user_home_t:s0  

Then the problem went away.

Comment 16 Rui Gouveia 2010-12-21 19:08:19 UTC
same problem.

CentOS 5.5 x64

389-ds-1.2.1-1.el5
389-admin-1.1.13-1.el5
389-ds-base-1.2.7.5-1.el5
389-ds-console-1.2.3-1.el5
389-console-1.1.4-1.el5
389-dsgw-1.1.5-1.el5
389-ds-console-doc-1.2.3-1.el5
389-adminutil-1.1.8-4.el5
389-admin-console-1.1.5-1.el5

Comment 17 Rich Megginson 2011-01-04 19:01:44 UTC
you must upgrade 389-adminutil to the latest

Comment 18 Nathan Kinder 2011-01-04 21:21:28 UTC
I am able to reproduce this issue on a fully up to date RHEL5 system using the latest 389 packages from EPEL.  Installing 389-adminutil-1.1.13 from epel-testing does not fix the issue.

Comment 19 Nathan Kinder 2011-01-04 21:52:06 UTC
I was able to get a stack trace from one of the httpd child process when it crashes:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x40d14940 (LWP 15914)]
0x00002b17df807820 in ?? ()
(gdb) bt
#0  0x00002b17df807820 in ?? ()
#1  0x00002b17d96c7ad9 in __nptl_deallocate_tsd () from /lib64/libpthread.so.0
#2  0x00002b17d96c874b in start_thread () from /lib64/libpthread.so.0
#3  0x00002b17d9bb5f6d in clone () from /lib64/libc.so.6

I'll attach a dump of all threads.  I'm not sure if this is a selinux issue or an httpd issue, but it doesn't appear to be an issue with the 389-admin code itself.

Comment 20 Nathan Kinder 2011-01-04 21:53:20 UTC
Created attachment 471755 [details]
threads stack traces

Comment 21 Daniel Walsh 2011-01-04 22:36:31 UTC
You can turn off dontaudit rules

semodule -DB

To see if an SELinux dontaudit rule is causing the problem.

Comment 22 Nathan Kinder 2011-01-04 22:44:01 UTC
Created attachment 471766 [details]
AVC messages

Here are the AVC messages that are reported after turning off dontaudit messages.

Comment 23 Nathan Kinder 2011-01-04 22:56:37 UTC
Creating a custom selinux module with the following rule appears to fix this issue:

allow dirsrvadmin_t httpd_t:process { siginh rlimitinh noatsecure };

Comment 24 Nathan Kinder 2011-01-04 23:07:44 UTC
Created attachment 471771 [details]
Patch

Comment 25 Nathan Kinder 2011-01-04 23:40:21 UTC
Pushed patch to master.  Thanks to Noriko for her review!

Counting objects: 7, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 555 bytes, done.
Total 4 (delta 3), reused 0 (delta 0)
To ssh://git.fedorahosted.org/git/389/admin.git
   f40a2f1..6d86721  master -> master

Comment 26 Amita Sharma 2011-05-18 11:48:20 UTC
1. Tested on Red Hat Enterprise Linux Server release 6.0 (Santiago)
with 
[root@testvm ~]# rpm -qa | grep 389
389-adminutil-1.1.13-1.el6.x86_64
389-console-1.1.4-1.el6.noarch
389-admin-console-1.1.7-1.el6.noarch
389-ds-base-libs-1.2.8.2-1.el6_1.1.x86_64
389-admin-1.1.16-2.el6.x86_64
389-ds-console-1.2.5-1.el6.noarch
389-ds-base-1.2.8.2-1.el6_1.1.x86_64

2. [root@testvm ~]# cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted 

3. /etc/init.d/dirsrv-admin start

4. [root@testvm ~]# tail -f /var/log/audit/audit.log 
type=LOGIN msg=audit(1305718261.449:1359): login pid=8480 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=220
type=USER_START msg=audit(1305718261.449:1360): user pid=8480 uid=0 auid=0 ses=220 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_open acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_DISP msg=audit(1305718261.483:1361): user pid=8480 uid=0 auid=0 ses=220 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=USER_END msg=audit(1305718261.483:1362): user pid=8480 uid=0 auid=0 ses=220 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=USER_ACCT msg=audit(1305718801.491:1363): user pid=8524 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:accounting acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_ACQ msg=audit(1305718801.491:1364): user pid=8524 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=LOGIN msg=audit(1305718801.491:1365): login pid=8524 uid=0 old auid=4294967295 new auid=0 old ses=4294967295 new ses=221
type=USER_START msg=audit(1305718801.491:1366): user pid=8524 uid=0 auid=0 ses=221 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_open acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_DISP msg=audit(1305718801.507:1367): user pid=8524 uid=0 auid=0 ses=221 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=USER_END msg=audit(1305718801.507:1368): user pid=8524 uid=0 auid=0 ses=221 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'


No problem encountered here, marking the bug as VERIFIED.


Note You need to log in before you can comment on or make changes to this bug.