Created attachment 983415 [details] audit.log Description of problem: Our automation noticed that /var/log/messages contained several errors related to celery not being able to connect to qpid. The symptoms were shown by basically pegging the system and preventing us from connecting to the web UI after a few preliminary tests were run. Justin has spent some time looking into it and it seems that python processes are not able to read the CA file. Since our automation uses SELinux in enforcing mode, the moment we turned off SELinux the issues disappeared. Attached you will find the audit log. Version-Release number of selected component (if applicable): * apr-util-ldap-1.3.9-3.el6_0.1.x86_64 * candlepin-0.9.38-1.el6.noarch * candlepin-common-1.0.18-1.el6.noarch * candlepin-scl-1-5.el6_4.noarch * candlepin-scl-quartz-2.1.5-5.el6_4.noarch * candlepin-scl-rhino-1.7R3-1.el6_4.noarch * candlepin-scl-runtime-1-5.el6_4.noarch * candlepin-selinux-0.9.38-1.el6.noarch * candlepin-tomcat6-0.9.38-1.el6.noarch * elasticsearch-0.90.10-6.el6sat.noarch * foreman-1.7.1.2-1.el6_6sat.noarch * foreman-compute-1.7.1.2-1.el6_6sat.noarch * foreman-gce-1.7.1.2-1.el6_6sat.noarch * foreman-libvirt-1.7.1.2-1.el6_6sat.noarch * foreman-ovirt-1.7.1.2-1.el6_6sat.noarch * foreman-postgresql-1.7.1.2-1.el6_6sat.noarch * foreman-proxy-1.7.1.1-1.el6_6sat.noarch * foreman-selinux-1.7.1.1-1.el6_6sat.noarch * foreman-vmware-1.7.1.2-1.el6_6sat.noarch * katello-2.2.0.1-1.el6_6sat.noarch * katello-certs-tools-2.2.1-1.el6_6sat.noarch * katello-default-ca-1.0-1.noarch * katello-installer-2.2.0.2-1.el6_6sat.noarch * katello-installer-base-2.2.0.2-1.el6_6sat.noarch * katello-server-ca-1.0-1.noarch * openldap-2.4.39-8.el6.x86_64 * pulp-docker-plugins-0.2.1-0.2.beta.el6_6sat.noarch * pulp-katello-0.3-4.el6sat.noarch * pulp-nodes-common-2.5.0-0.7.beta.el6_6sat.noarch * pulp-nodes-parent-2.5.0-0.7.beta.el6_6sat.noarch * pulp-puppet-plugins-2.5.0-0.7.beta.el6sat.noarch * pulp-puppet-tools-2.5.0-0.7.beta.el6sat.noarch * pulp-rpm-plugins-2.5.0-0.7.beta.el6_6sat.noarch * pulp-selinux-2.5.0-0.7.beta.el6_6sat.noarch * pulp-server-2.5.0-0.7.beta.el6_6sat.noarch * python-ldap-2.3.10-1.el6.x86_64 * ruby193-rubygem-ldap_fluff-0.3.2-1.el6_6sat.noarch * ruby193-rubygem-net-ldap-0.3.1-3.el6sat.noarch * ruby193-rubygem-runcible-1.3.0-1.el6_6sat.noarch * rubygem-hammer_cli-0.1.4.2-1.el6_6sat.noarch * rubygem-hammer_cli_foreman-0.1.4.2-1.el6_6sat.noarch * rubygem-hammer_cli_foreman_bootdisk-0.1.2.4-1.el6_6sat.noarch * rubygem-hammer_cli_foreman_tasks-0.0.3.1-1.el6_6sat.noarch * rubygem-hammer_cli_import-0.10.6-1.el6sat.noarch * rubygem-hammer_cli_katello-0.0.7.1-1.el6_6sat.noarch How reproducible: Steps to Reproduce: 1. Install latest compose Satellite-6.1.0-RHEL-6-20150121.0 2. Try to create a custom repo and sync it 3. Actual results: Expected results: Additional info:
Upstream/nighly also has the same issue but the katello deploy script may be masking it as it currently disables SELinux: * https://github.com/Katello/katello-deploy/blob/master/setup.rb#L80
To also clarify, the pulp celery processes used to be unconstrained, but they no longer are.
There are several problems. A) Satellite6 tries to connect to AMQP/qpidd port. We never had this rule in. Filed a different BZ to handle this: https://bugzilla.redhat.com/show_bug.cgi?id=1185801 B) There are problems with Celery reading katello-default-ca.crt which is labeled cert_t. This should be added to Pulp SELinux policy as this is pretty standard configuration case (configure celery with SSL -> it needs to read the files labeled with cert_t which is policy standard type): miscfiles_manage_cert_files(celery_t) C) There are some issues with qpidd startup, which might be RHEL6 bug or misconfiguration on our side. I will investiagete further: time->Wed Jan 21 15:38:32 2015 type=SYSCALL msg=audit(1421872712.984:567): arch=c000003e syscall=59 success=yes exit=0 a0=16de870 a1=16dddb0 a2=16dcc00 a3=7fffb865c380 items=0 ppid=17285 pid=17286 auid=0 uid=498 gid=498 euid=498 suid=498 fsuid=498 egid=498 sgid=498 fsgid=498 tty=(none) ses=1 comm="qpidd" exe="/usr/sbin/qpidd" subj=unconfined_u:system_r:qpidd_t:s0 key=(null) type=AVC msg=audit(1421872712.984:567): avc: denied { read } for pid=17286 comm="qpidd" path="/etc/rc.d/init.d/qpidd" dev=dm-1 ino=6030256 scontext=unconfined_u:system_r:qpidd_t:s0 tcontext=system_u:object_r:qpidd_initrc_exec_t:s0 tclass=file So Satellite 6 team is solving A and C, but case B is on Pulp team. I need input from the Pulp team on this.
The base pulp policy SELinux policy expects a type called pulp_cert_t to be applied to certs that Celery pulp should manage. By default those files usually exist in /etc/pki/pulp/* per [0]. Pulp celery workers should be able to read and write files with the pulp_cert_t type with this rule [1]. That should resolve your issue, but let me know if it does not. [0]: https://github.com/pulp/pulp/blob/master/server/selinux/server/pulp-server.fc#L1
Forgot second link from comment 6 [1]: https://github.com/pulp/pulp/blob/master/server/selinux/server/pulp-celery.te#L51
Thanks for explanation, I have a request: https://github.com/pulp/pulp/pull/1575 Since cert_t is generic label for most certificates, I've added new rule to allow reading of these (not writing as I don't expect Pulp to write the CA file). If you accept this patch, please backport it into Satellite 6 version of Pulp. Because our CA file is already used in other services which expect the generic cert_t, we'd need to create a symlink with pulp_cert_t otherwise. I don't like this solution and it's less secure as I don't think Pulp should have write access to the CA file and the key. You could think this is a "hack" in Pulp for Satellite 6, but there can be more users providing their own CA certificates. For example if you generate a CA certificate using the standard OpenSSL tools in a different /etc/pki directory, it will end up with cert_t anyway so Pulp will not work until the user changes this to pulp_cert_t. This is unnecessary I think, the macro is distributed in the RHEL6/7 and Fedora core policies and it's a standard practice to allow reading of that. I'd recommend you to drop the write access to CA cert/key for future versions to improve security. But this is a different story. Thanks.
*** Bug 1186760 has been marked as a duplicate of this bug. ***
I could no longer see the same issue with latest build. Verified on Satellite-6.1.0-RHEL-6-20150303.0
This bug is slated to be released with Satellite 6.1.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2015:1592