Bug 1489863
| Summary: | [OSP12] Undercloud deployment failed due to httpd service didn't started | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Artem Hrechanychenko <ahrechan> | ||||
| Component: | openstack-selinux | Assignee: | Lon Hohberger <lhh> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Udi Shkalim <ushkalim> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 12.0 (Pike) | CC: | ahrechan, mgrepl, rcritten, rhallise, sasha, srevivo | ||||
| Target Milestone: | beta | Keywords: | Triaged | ||||
| Target Release: | 12.0 (Pike) | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openstack-selinux-0.8.10-0.20170914195211.e16a8f8.2.el7ost | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1671514 (view as bug list) | Environment: | |||||
| Last Closed: | 2017-12-13 22:08:14 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1671514 | ||||||
| Attachments: |
|
||||||
|
Description
Artem Hrechanychenko
2017-09-08 14:01:42 UTC
[stack@undercloud-0 ~]$ sudo ausearch -m AVC
----
time->Fri Sep 8 09:33:35 2017
type=PROCTITLE msg=audit(1504877615.040:940): proctitle=67726570002D71497345005E696E7374616C6C5B5B3A73706163653A5D5D2B697076365B5B3A73706163653A5D5D2B2F62696E2F28747275657C66616C736529002F6574632F6D6F6470726F62652E636F6E66002F6574632F6D6F6470726F62652E642F6C6F636B642E636F6E66002F6574632F6D6F6470726F62652E642F74
type=SYSCALL msg=audit(1504877615.040:940): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=7ffe96387f49 a2=0 a3=0 items=0 ppid=26119 pid=26129 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="grep" exe="/usr/bin/grep" subj=system_u:system_r:iptables_t:s0 key=(null)
type=AVC msg=audit(1504877615.040:940): avc: denied { read } for pid=26129 comm="grep" name="lockd.conf" dev="vda1" ino=8596051 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file
----
time->Fri Sep 8 09:33:35 2017
type=PROCTITLE msg=audit(1504877615.040:941): proctitle=67726570002D71497345005E696E7374616C6C5B5B3A73706163653A5D5D2B697076365B5B3A73706163653A5D5D2B2F62696E2F28747275657C66616C736529002F6574632F6D6F6470726F62652E636F6E66002F6574632F6D6F6470726F62652E642F6C6F636B642E636F6E66002F6574632F6D6F6470726F62652E642F74
type=SYSCALL msg=audit(1504877615.040:941): arch=c000003e syscall=257 success=no exit=-13 a0=ffffffffffffff9c a1=7ffe96387f64 a2=0 a3=0 items=0 ppid=26119 pid=26129 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="grep" exe="/usr/bin/grep" subj=system_u:system_r:iptables_t:s0 key=(null)
type=AVC msg=audit(1504877615.040:941): avc: denied { read } for pid=26129 comm="grep" name="tuned.conf" dev="vda1" ino=8595095 scontext=system_u:system_r:iptables_t:s0 tcontext=system_u:object_r:modules_conf_t:s0 tclass=file
----
time->Fri Sep 8 09:48:09 2017
type=PROCTITLE msg=audit(1504878489.926:1288): proctitle=7375646F006E657574726F6E2D726F6F74777261702D6461656D6F6E002F6574632F6E657574726F6E2F726F6F74777261702E636F6E66
type=SYSCALL msg=audit(1504878489.926:1288): arch=c000003e syscall=2 success=no exit=-13 a0=7fae5cc076ed a1=0 a2=1b6 a3=24 items=0 ppid=4219 pid=4288 auid=4294967295 uid=0 gid=990 euid=0 suid=0 fsuid=0 egid=0 sgid=990 fsgid=0 tty=(none) ses=4294967295 comm="sudo" exe="/usr/bin/sudo" subj=system_u:system_r:neutron_t:s0 key=(null)
type=AVC msg=audit(1504878489.926:1288): avc: denied { search } for pid=4288 comm="sudo" name="sssd" dev="vda1" ino=192938588 scontext=system_u:system_r:neutron_t:s0 tcontext=system_u:object_r:sssd_conf_t:s0 tclass=dir
----
time->Fri Sep 8 09:48:39 2017
type=PROCTITLE msg=audit(1504878519.137:1300): proctitle=2F7573722F7362696E2F6874747064002D44464F524547524F554E44
type=SYSCALL msg=audit(1504878519.137:1300): arch=c000003e syscall=49 success=no exit=-13 a0=6 a1=555b47dbe5e8 a2=10 a3=7ffd834f7f4c items=0 ppid=1 pid=5079 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1504878519.137:1300): avc: denied { name_bind } for pid=5079 comm="httpd" src=6385 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
Can you please attach a full audit.log from a permissive run? The call trace is important here, since I kind of need to understand why iptables is executing grep in the first place. Created attachment 1324999 [details]
audit.log
Thanks - this is peculiar.
type=AVC msg=audit(1505242381.916:1184): avc: denied { name_bind } for pid=14940 comm="httpd" src=6385 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket
Was caused by:
The boolean nis_enabled was set incorrectly.
Description:
Allow nis to enabled
Allow access by executing:
# setsebool -P nis_enabled 1
... this is turned on by having openstack-selinux installed. Is it not presently installed or something?
I'm seeing the same issue from an oooq install. I have openstack-selinux-0.8.9-0.1.el7ost.noarch installed. # semanage boolean --list | grep nis_enabled nis_enabled (off , off) Allow nis to enabled OK, so it's installed (and the boolean is set) in the overcloud-full image. This seems to be specific to the undercloud - it's like the boolean is getting turned off somehow. Rob, could you also check: os_nova_use_execmem neutron_can_network httpd_use_openstack ? Rob said they're all 'on'. This is unfortunate, since it means something is explicitly disabling the nis_enabled boolean between the time openstack-selinux is installed and the time httpd is started. This may very well not be an openstack-selinux bug. I was unable to find what might be disabling the nis_enabled boolean, so I removed the need to use it. https://github.com/redhat-openstack/openstack-selinux/commit/e16a8f8ef52cc5147b73dda508f7da41368b7ea8 The patch passes all previously-known AVCs, but may suffer from others that were masked while nis_enabled was used. *** Bug 1493561 has been marked as a duplicate of this bug. *** The reason the issue wasn't fixed during deployments is because of the fact that images are built in effectively a chroot, rendering the kernel's selinux policy unavailable. Thus, the block in local_settings.sh which asks the kernel policy if it is enabled, followed by setting a bunch of booleans wasn't executing, rendering the original fix useless on images. One of the booleans it was setting was os_httpd_wsgi, which gives httpd access to bind to any port (a la nis_enabled). This is why it worked correctly after normal installation in baremetal or virtualized environments, but was failing in all of our CI tests. Preliminary checks on overcloud-full-12.0-20170922.2: * Image contains openstack-selinux-0.8.10-0.20170914195211.e16a8f8.2.el7ost.noarch (GOOD) * All booleans are correctly set (GOOD) * nis_enabled is no longer a part of openstack-selinux (GOOD) * All os-* modules are installed (GOOD) Will help review CI results when they are available. VERIFIED (undercloud) [stack@undercloud-0 ~]$ sudo rpm -qa "*stack*" openstack-mistral-engine-5.1.1-0.20171027222844.fd979d9.el7ost.noarch openstack-puppet-modules-11.0.0-0.20170828113154.el7ost.noarch openstack-neutron-openvswitch-11.0.1-3.el7ost.noarch openstack-heat-engine-9.0.1-0.20171023060845.be1e2e9.el7ost.noarch instack-7.0.1-1.el7ost.noarch openstack-tripleo-puppet-elements-7.0.1-0.20171020122223.82d7e6c.el7ost.noarch openstack-nova-common-16.0.3-0.20171028031400.60d6e87.el7ost.noarch openstack-swift-account-2.15.2-0.20170927035729.0344d6e.el7ost.noarch openstack-heat-common-9.0.1-0.20171023060845.be1e2e9.el7ost.noarch openstack-tripleo-common-7.6.3-0.20171028055750.el7ost.noarch openstack-mistral-common-5.1.1-0.20171027222844.fd979d9.el7ost.noarch openstack-tripleo-validations-7.4.1-2.el7ost.noarch openstack-nova-api-16.0.3-0.20171028031400.60d6e87.el7ost.noarch openstack-nova-conductor-16.0.3-0.20171028031400.60d6e87.el7ost.noarch openstack-keystone-12.0.1-0.20171012013909.5c9ccce.el7ost.noarch puppet-openstack_extras-11.3.1-0.20170906070209.b99c3a4.el7ost.noarch python-openstackclient-lang-3.12.0-1.el7ost.noarch openstack-ironic-common-9.1.2-0.20171025074857.cf3665f.el7ost.noarch python-openstacksdk-0.9.17-1.el7ost.noarch openstack-tripleo-image-elements-7.0.1-0.20171020101256.2e61e31.el7ost.noarch instack-undercloud-7.4.2-2.el7ost.noarch openstack-mistral-executor-5.1.1-0.20171027222844.fd979d9.el7ost.noarch openstack-selinux-0.8.11-0.20171013192233.ce13ba7.el7ost.noarch openstack-nova-placement-api-16.0.3-0.20171028031400.60d6e87.el7ost.noarch openstack-glance-15.0.1-0.20171017090105.06af2eb.el7ost.noarch openstack-swift-object-2.15.2-0.20170927035729.0344d6e.el7ost.noarch openstack-neutron-common-11.0.1-3.el7ost.noarch openstack-neutron-ml2-11.0.1-3.el7ost.noarch openstack-swift-proxy-2.15.2-0.20170927035729.0344d6e.el7ost.noarch openstack-heat-api-cfn-9.0.1-0.20171023060845.be1e2e9.el7ost.noarch openstack-ironic-api-9.1.2-0.20171025074857.cf3665f.el7ost.noarch openstack-zaqar-5.0.1-0.20171027110724.4f07aed.el7ost.noarch puppet-openstacklib-11.3.1-0.20170921022915.6e2b844.el7ost.noarch openstack-swift-container-2.15.2-0.20170927035729.0344d6e.el7ost.noarch openstack-neutron-11.0.1-3.el7ost.noarch openstack-ironic-conductor-9.1.2-0.20171025074857.cf3665f.el7ost.noarch openstack-tempest-17.1.0-1.el7ost.noarch openstack-tripleo-heat-templates-7.0.3-0.20171024200823.el7ost.noarch openstack-tripleo-common-containers-7.6.3-0.20171028055750.el7ost.noarch openstack-mistral-api-5.1.1-0.20171027222844.fd979d9.el7ost.noarch openstack-nova-scheduler-16.0.3-0.20171028031400.60d6e87.el7ost.noarch openstack-ironic-inspector-6.0.1-0.20170920142417.77e2b1a.el7ost.noarch openstack-tripleo-ui-7.4.2-2.el7ost.noarch openstack-nova-compute-16.0.3-0.20171028031400.60d6e87.el7ost.noarch openstack-heat-api-9.0.1-0.20171023060845.be1e2e9.el7ost.noarch python-openstackclient-3.12.0-1.el7ost.noarch (undercloud) [stack@undercloud-0 ~]$ sudo sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: enforcing Mode from config file: enforcing Policy MLS status: enabled Policy deny_unknown status: allowed Max kernel policy version: 28 (undercloud) [stack@undercloud-0 ~]$ sudo rpm -qa "*selinux*" libselinux-utils-2.5-11.el7.x86_64 selinux-policy-targeted-3.13.1-166.el7_4.5.noarch libselinux-2.5-11.el7.x86_64 selinux-policy-3.13.1-166.el7_4.5.noarch libselinux-python-2.5-11.el7.x86_64 openstack-selinux-0.8.11-0.20171013192233.ce13ba7.el7ost.noarch container-selinux-2.28-1.git85ce147.el7.noarch libselinux-ruby-2.5-11.el7.x86_64 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:3462 |