Bug 1007606

Created attachment 797057 [details]
offending client install log

Does /var/run%/slapd-TESTRELM-COM.socket exist? Are there any AVCs?

Can you see if there are any errors in the 389-ds error log, or look for issues at the tail of /var/log/httpd/error_log?

The socket file does exist:
[root@ipaqa64vml ~]#  ls /var/run/slapd-TESTRELM-COM.socket  -alh
srw-rw-rw-. 1 root root 0 Sep 13 14:50 /var/run/slapd-TESTRELM-COM.socket

There is one AVC that may be helpful:

type=AVC msg=audit(1379098222.224:360): avc:  denied  { name_connect } for  pid=29785 comm="httpd" dest=389 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:ldap_port_t:s0 tclass=tcp_socket


The httpd error log doesn't seem to have much useful. 
This is the last 2 starts worth:

[Fri Sep 13 14:48:57.156476 2013] [core:notice] [pid 28975] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Fri Sep 13 14:49:01.580854 2013] [:error] [pid 28977] ipa: INFO: *** PROCESS START ***
[Fri Sep 13 14:49:01.740416 2013] [:error] [pid 28978] ipa: INFO: *** PROCESS START ***
[Fri Sep 13 14:50:15.917708 2013] [mpm_prefork:notice] [pid 28975] AH00170: caught SIGWINCH, shutting down gracefully
[Fri Sep 13 14:50:17.238747 2013] [core:notice] [pid 29743] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0
[Fri Sep 13 14:50:17.240876 2013] [suexec:notice] [pid 29743] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Sep 13 14:50:17.555577 2013] [auth_digest:notice] [pid 29743] AH01757: generating secret for digest authentication ...
[Fri Sep 13 14:50:17.557087 2013] [lbmethod_heartbeat:notice] [pid 29743] AH02282: No slotmem from mod_heartmonitor
[Fri Sep 13 14:50:17.564756 2013] [:warn] [pid 29743] mod_wsgi: Compiled for Python/2.7.3.
[Fri Sep 13 14:50:17.564814 2013] [:warn] [pid 29743] mod_wsgi: Runtime using Python/2.7.5.
[Fri Sep 13 14:50:17.637543 2013] [mpm_prefork:notice] [pid 29743] AH00163: Apache/2.4.6 (Red Hat) mod_auth_kerb/5.4 mod_nss/2.4.4 NSS/3.15 mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations
[Fri Sep 13 14:50:17.637670 2013] [core:notice] [pid 29743] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Fri Sep 13 14:50:21.576975 2013] [:error] [pid 29745] ipa: INFO: *** PROCESS START ***
[Fri Sep 13 14:50:22.042598 2013] [:error] [pid 29746] ipa: INFO: *** PROCESS START ***

Marking this one TestBlocker. 

I'm hitting the same bug.  It's causing problems with my trust tests.

Scott, are you seeing the same AVC?

yes, similar:

type=SOCKADDR msg=audit(1379098968.774:216): saddr=020001850A1004110000000000000000

type=SYSCALL msg=audit(1379098968.774:216): arch=c000003e syscall=42 success=no exit=-13 a0=13 a1=7fd1bc0161b0 a2=10 a3=0 items=0 ppid=17527 pid=17577 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)

type=AVC msg=audit(1379098968.774:216): avc:  denied  { name_connect } for  pid=17577 comm="httpd" dest=389 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:ldap_port_t:s0 tclass=tcp_socket

Scott provided a system demonstrating this. Putting it into permissive mode results in a successful install when re-running the ipa-client-install command.

I'm a little confused by this AVC because it would appear that we're connecting over 389, but the error coming out of the server refers to ldapi.

audit2allow suggests adding this:

allow httpd_t ldap_port_t:tcp_socket name_connect;

Adding a selinux module containing "allow httpd_t ldap_port_t:tcp_socket name_connect;" does allow ipa to install properly.

Also, I did setenforce 0 before a test that was failing and install worked without a problem.

I think the failure is caused by ipa-server-install's inability to set respective SELinux booleans:

WARNING: could not set the following SELinux boolean(s):
  httpd_can_network_connect -> on
  httpd_manage_ipa -> on
The web interface may not function correctly until the booleans
are successfully changed with the command:
/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
Try updating the policycoreutils and selinux-policy packages.

It would be useful to get respective excerpt from ipaserver-install.log so that we can see what error was printed by setsebool.

This is what I see on a quick install check this morning:

2013-09-16T14:09:20Z DEBUG   [12/15]: configuring SELinux for httpd
2013-09-16T14:09:20Z DEBUG Starting external process
2013-09-16T14:09:20Z DEBUG args=/usr/sbin/selinuxenabled
2013-09-16T14:09:20Z DEBUG Process finished, return code=0
2013-09-16T14:09:20Z DEBUG stdout=
2013-09-16T14:09:20Z DEBUG stderr=
2013-09-16T14:09:20Z DEBUG Starting external process
2013-09-16T14:09:20Z DEBUG args=/usr/sbin/getsebool httpd_can_network_connect
2013-09-16T14:09:20Z DEBUG Process finished, return code=0
2013-09-16T14:09:20Z DEBUG stdout=httpd_can_network_connect --> off

2013-09-16T14:09:20Z DEBUG stderr=
2013-09-16T14:09:20Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-16T14:09:20Z DEBUG Starting external process
2013-09-16T14:09:20Z DEBUG args=/usr/sbin/getsebool httpd_manage_ipa
2013-09-16T14:09:20Z DEBUG Process finished, return code=0
2013-09-16T14:09:20Z DEBUG stdout=httpd_manage_ipa --> off

2013-09-16T14:09:20Z DEBUG stderr=
2013-09-16T14:09:20Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-16T14:09:20Z DEBUG Starting external process
2013-09-16T14:09:20Z DEBUG args=/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
2013-09-16T14:09:21Z DEBUG Process finished, return code=255
2013-09-16T14:09:21Z DEBUG stdout=
2013-09-16T14:09:21Z DEBUG stderr=Boolean httpd_can_network_connect is not defined

2013-09-16T14:09:21Z DEBUG WARNING: could not set the following SELinux boolean(s):
  httpd_can_network_connect -> on
  httpd_manage_ipa -> on
The web interface may not function correctly until the booleans
are successfully changed with the command:
/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
Try updating the policycoreutils and selinux-policy packages.
2013-09-16T14:09:21Z DEBUG   duration: 0 seconds

Need me to also post entire ipaserver-install.log?

No, this should be enough. I am reassigning the Bug to policycoreutils, they have more experience with finding what gone wrong with setsebool.

Dan
I sent you a patch for this issue.

FYI: 

Seems like this may be a workaround to run before ipa-server-install or ipa-replica-install:

/usr/sbin/semanage boolean --modify --on httpd_can_network_connect
/usr/sbin/semanage boolean --modify --on httpd_manage_ipa

So can this be fixed the same way bug 998974 was?  Is there a build of policycoreutils I can test with yet?

Thanks,
Scott

(In reply to Scott Poore from comment #18)
> FYI: 
> 
> Seems like this may be a workaround to run before ipa-server-install or
> ipa-replica-install:
> 
> /usr/sbin/semanage boolean --modify --on httpd_can_network_connect
> /usr/sbin/semanage boolean --modify --on httpd_manage_ipa

Yes, it can be workaround for now.
> 
> So can this be fixed the same way bug 998974 was?

Yes, it should be fixed in the same way.

>  Is there a build of
> policycoreutils I can test with yet?

I sent a patch. So I believe Dan will build a new policycoreutils soon.
> 
> Thanks,
> Scott

Hi Dan,

Is this going to be fixed soon for RHEL7?

Thanks,
Scott

I need flags in order to build.

Fixed in /policycoreutils-2.1.14-81.el7

We have the flags now, can we please build now to unblock IPA QE?

*** Bug 1011164 has been marked as a duplicate of this bug. ***

This bug still seems to be around. 

After switching to:
policycoreutils-python-2.1.14-81.el7.x86_64
ipa-server-3.3.1-5.el7.x86_64

I no longer get the error 907 during client install, but the install still fails with very similar looking AVC's

type=AVC msg=audit(1380584149.400:221): avc:  denied  { write } for  pid=5955 comm="httpd" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=key
type=SYSCALL msg=audit(1380584149.400:221): arch=c000003e syscall=248 success=no exit=-13 a0=7ffb134e9b2e a1=7ffb22d3b240 a2=0 a3=0 items=0 ppid=5852 pid=5955 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1380584149.401:222): avc:  denied  { write } for  pid=5955 comm="httpd" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=key
type=SYSCALL msg=audit(1380584149.401:222): arch=c000003e syscall=248 success=no exit=-13 a0=7ffb134e9b2e a1=7ffb23325570 a2=0 a3=0 items=0 ppid=5852 pid=5955 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)


I'll submit my install and audit logs for review.

Created attachment 805630 [details]
audit log from install after eratta

Created attachment 805631 [details]
client install log from after eratta

Looks like a Bug 1012051 to me.

I'm seeing this fail again as well today:

[root@qe-blade-05 ~]# rpm -q policycoreutils
policycoreutils-2.1.14-81.el7.x86_64
[root@qe-blade-05 ~]# rpm -q selinux-policy
selinux-policy-3.12.1-84.el7.noarch
[root@qe-blade-05 ~]# getenforce 
Permissive
[root@qe-blade-05 ~]# getsebool -a|grep -i httpd_can_network_connect
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
[root@qe-blade-05 ~]# getsebool -a|grep -i httpd_manage_ipa
httpd_manage_ipa --> off
[root@qe-blade-05 ~]# /usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
Boolean httpd_can_network_connect is not defined

The above was run after ipa-server-install was run.  From ipaserver-install.log:

2013-10-01T17:30:37Z DEBUG args=/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
2013-10-01T17:30:37Z DEBUG Process finished, return code=255
2013-10-01T17:30:37Z DEBUG stdout=
2013-10-01T17:30:37Z DEBUG stderr=Boolean httpd_can_network_connect is not defined

2013-10-01T17:30:37Z DEBUG WARNING: could not set the following SELinux boolean(s):
  httpd_can_network_connect -> on
  httpd_manage_ipa -> on
The web interface may not function correctly until the booleans
are successfully changed with the command:
/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
Try updating the policycoreutils and selinux-policy packages.
2013-10-01T17:30:37Z DEBUG   duration: 0 seconds

Ok, I'm able to manually reproduce this one manually with the following:

[root@rhel7-1 ~]#  /usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on

[root@rhel7-1 ~]# echo $?
255

[root@rhel7-1 ~]# rpm -q policycoreutils selinux-policy
policycoreutils-2.1.14-81.el7.x86_64
selinux-policy-3.12.1-84.el7.noarch

[root@rhel7-1 ~]# getsebool -a |egrep "httpd_can_network_connect|httpd_manage_ipa"
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_manage_ipa --> off

So, is this a different issue than previously believed?

What other information is needed to troubleshoot and resolve this?

Thanks,
Scott

Dan,

Can we get an estimate for when this will be fixed?

Our workaround seems to have failed us in an automated test I ran earlier today:

:: [ 16:26:59 ] ::  force setting selinux booleans for ipa install failure
libsemanage.semanage_read_policydb: Could not open kernel policy /etc/selinux/targeted/modules/active/policy.kern for reading. (No such file or directory).
ValueError: Could not test MLS enabled status

:: [   FAIL   ] :: Running '/usr/sbin/semanage boolean --modify --on httpd_can_network_connect' (Expected 0, got 1)
libsemanage.semanage_read_policydb: Could not open kernel policy /etc/selinux/targeted/modules/active/policy.kern for reading. (No such file or directory).
ValueError: Could not test MLS enabled status

:: [   FAIL   ] :: Running '/usr/sbin/semanage boolean --modify --on httpd_manage_ipa' (Expected 0, got 1)

Version:

policycoreutils-2.1.14-81.el7.x86_64
selinux-policy-3.12.1-84.el7.noarch

I'm re-running that test now to see if it occurs again.  Does that error need a separate bug or is it related here?

Thanks,
Scott

Any chance to re-test it with the latest builds.

selinux-policy,policycoreutils,libsemanage.

I believe it should work correctly.

This is from a test today.

[root@qe-blade-12 ~]# rpm -q selinux-policy policycoreutils libsemanage
selinux-policy-3.12.1-85.el7.noarch
policycoreutils-2.1.14-81.el7.x86_64
libsemanage-2.1.10-12.el7.x86_64

[root@qe-blade-12 ~]# less /var/log/ipaserver-install.log
...
2013-10-03T15:52:06Z DEBUG args=/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
2013-10-03T15:52:06Z DEBUG Process finished, return code=255
2013-10-03T15:52:06Z DEBUG stdout=
2013-10-03T15:52:06Z DEBUG stderr=Boolean httpd_can_network_connect is not defined

2013-10-03T15:52:06Z DEBUG WARNING: could not set the following SELinux boolean(s):
  httpd_can_network_connect -> on
  httpd_manage_ipa -> on
The web interface may not function correctly until the booleans
are successfully changed with the command:
/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
Try updating the policycoreutils and selinux-policy packages.
...

[root@qe-blade-12 ~]# getenforce
Permissive

[root@qe-blade-12 ~]# /usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
Boolean httpd_can_network_connect is not defined

[root@qe-blade-12 ~]# getsebool -a |egrep "httpd_can_network_connect|httpd_manage_ipa"
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_manage_ipa --> off

So, what else can we check?

policycoreutils-2.1.14-84.el7 is now in rhel7

This still does not appear to be fixed:

[root@ipaqa64vmd ~]# setenforce 1

[root@ipaqa64vmd ~]# /usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
Boolean httpd_can_network_connect is not defined

[root@ipaqa64vmd ~]# setenforce 0

[root@ipaqa64vmd ~]# /usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
Boolean httpd_can_network_connect is not defined

[root@ipaqa64vmd ~]# getsebool -a |egrep "httpd_can_network_connect|httpd_manage_ipa"
httpd_can_network_connect --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_manage_ipa --> off

[root@ipaqa64vmd ~]# /usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
Boolean httpd_can_network_connect is not defined

[root@ipaqa64vmd ~]# rpm -q policycoreutils selinux-policy libsemanage
policycoreutils-2.1.14-84.el7.x86_64
selinux-policy-3.12.1-86.el7.noarch
libsemanage-2.1.10-12.el7.x86_64

I'll also attach an strace in case that helps here.

Created attachment 807991 [details]
strace from setsebool command

I see

# rpm -q policycoreutils
policycoreutils-2.1.14-84.el7.x86_64

# getsebool httpd_can_network_connect
httpd_can_network_connect --> on

# /usr/sbin/setsebool -P httpd_can_network_connect=off
# echo $?
0

# getsebool httpd_can_network_connect
httpd_can_network_connect --> off

So it works for me.

Michal,
could you also check it?

Looks like it's working again with a build from the latest repos from this morning:

[root@dell-pem805-01 ~]# rpm -q policycoreutils selinux-policy libsemanage
policycoreutils-2.1.14-84.el7.x86_64
selinux-policy-3.12.1-86.el7.noarch
libsemanage-2.1.10-13.el7.x86_64

[root@dell-pem805-01 ~]# setsebool -P httpd_can_network_connect=on 

[root@dell-pem805-01 ~]# echo $?
0

[root@dell-pem805-01 ~]# getsebool -a |grep httpd_can_network_connect
httpd_can_network_connect --> on
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off

So, not sure if the updates to libsemanage affected this or if it was a result of the build that's now resolved in the latest.

Going back to doublecheck Friday's failure, it looks like I had this version of libselinux:

libselinux-2.1.13-16.el7.x86_64

While now, I have:

libselinux-2.1.13-21.el7.x86_64

Was something updated there that could have caused the failure I saw in comment #37?

Thanks,
Scott

Well libselinux/policycoreutils/libsemanage could all be involved with this.

Well, looks better now.  I just ran a full IPA server install without the semanage workarounds and it looks good.

I'm guessing my recent previous failures may have had something to do with the repos I was pointing to missing some updates.

Thanks,
Scott

Miroslav, Michal, 

What's the status here?  When I test this from a VM just built from a rhel7 repo, I am still seeing this error.

Also, in a training class we have today, we're seeing this in VMs that were built for the class.

Thanks,
Scott

This will be a problem with old versions, I guess.

It does finally look like the repos built on 10/24 seem to be fixed:

[root@vm1 ~]# getsebool httpd_can_network_connect
httpd_can_network_connect --> off

[root@vm1 ~]# rpm -qa|egrep "policy|selinux|semanage"|sort
libselinux-2.1.13-21.el7.x86_64
libselinux-python-2.1.13-21.el7.x86_64
libselinux-utils-2.1.13-21.el7.x86_64
libsemanage-2.1.10-14.el7.x86_64
policycoreutils-2.1.14-88.el7.x86_64
selinux-policy-3.12.1-94.el7.noarch
selinux-policy-targeted-3.12.1-94.el7.noarch

[root@vm1 ~]# setsebool -P httpd_can_network_connect=on

[root@vm1 ~]# 

Thanks

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.