Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1185401 - celery.worker.consumer:ERROR: consumer: Cannot connect to qpid
celery.worker.consumer:ERROR: consumer: Cannot connect to qpid
Status: CLOSED ERRATA
Product: Red Hat Satellite 6
Classification: Red Hat
Component: SELinux (Show other bugs)
Nightly
Unspecified Unspecified
unspecified Severity high (vote)
: Unspecified
: Unused
Assigned To: Lukas Zapletal
Og Maciel
: Triaged
: 1186760 (view as bug list)
Depends On: 1186420
Blocks:
  Show dependency treegraph
 
Reported: 2015-01-23 11:14 EST by Og Maciel
Modified: 2017-07-26 15:43 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1186420 (view as bug list)
Environment:
Last Closed: 2015-08-12 01:22:13 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
audit.log (5.42 MB, text/plain)
2015-01-23 11:14 EST, Og Maciel
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:1592 normal SHIPPED_LIVE Important: Red Hat Satellite 6.1.1 on RHEL 6 2015-08-12 05:04:35 EDT

  None (edit)
Description Og Maciel 2015-01-23 11:14:45 EST
Created attachment 983415 [details]
audit.log

Description of problem:

Our automation noticed that /var/log/messages contained several errors related to celery not being able to connect to qpid. The symptoms were shown by basically pegging the system and preventing us from connecting to the web UI after a few preliminary tests were run.

Justin has spent some time looking into it and it seems that python processes are not able to read the CA file. Since our automation uses SELinux in enforcing mode, the moment we turned off SELinux the issues disappeared.

Attached you will find the audit log.


Version-Release number of selected component (if applicable):

* apr-util-ldap-1.3.9-3.el6_0.1.x86_64
* candlepin-0.9.38-1.el6.noarch
* candlepin-common-1.0.18-1.el6.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.9.38-1.el6.noarch
* candlepin-tomcat6-0.9.38-1.el6.noarch
* elasticsearch-0.90.10-6.el6sat.noarch
* foreman-1.7.1.2-1.el6_6sat.noarch
* foreman-compute-1.7.1.2-1.el6_6sat.noarch
* foreman-gce-1.7.1.2-1.el6_6sat.noarch
* foreman-libvirt-1.7.1.2-1.el6_6sat.noarch
* foreman-ovirt-1.7.1.2-1.el6_6sat.noarch
* foreman-postgresql-1.7.1.2-1.el6_6sat.noarch
* foreman-proxy-1.7.1.1-1.el6_6sat.noarch
* foreman-selinux-1.7.1.1-1.el6_6sat.noarch
* foreman-vmware-1.7.1.2-1.el6_6sat.noarch
* katello-2.2.0.1-1.el6_6sat.noarch
* katello-certs-tools-2.2.1-1.el6_6sat.noarch
* katello-default-ca-1.0-1.noarch
* katello-installer-2.2.0.2-1.el6_6sat.noarch
* katello-installer-base-2.2.0.2-1.el6_6sat.noarch
* katello-server-ca-1.0-1.noarch
* openldap-2.4.39-8.el6.x86_64
* pulp-docker-plugins-0.2.1-0.2.beta.el6_6sat.noarch
* pulp-katello-0.3-4.el6sat.noarch
* pulp-nodes-common-2.5.0-0.7.beta.el6_6sat.noarch
* pulp-nodes-parent-2.5.0-0.7.beta.el6_6sat.noarch
* pulp-puppet-plugins-2.5.0-0.7.beta.el6sat.noarch
* pulp-puppet-tools-2.5.0-0.7.beta.el6sat.noarch
* pulp-rpm-plugins-2.5.0-0.7.beta.el6_6sat.noarch
* pulp-selinux-2.5.0-0.7.beta.el6_6sat.noarch
* pulp-server-2.5.0-0.7.beta.el6_6sat.noarch
* python-ldap-2.3.10-1.el6.x86_64
* ruby193-rubygem-ldap_fluff-0.3.2-1.el6_6sat.noarch
* ruby193-rubygem-net-ldap-0.3.1-3.el6sat.noarch
* ruby193-rubygem-runcible-1.3.0-1.el6_6sat.noarch
* rubygem-hammer_cli-0.1.4.2-1.el6_6sat.noarch
* rubygem-hammer_cli_foreman-0.1.4.2-1.el6_6sat.noarch
* rubygem-hammer_cli_foreman_bootdisk-0.1.2.4-1.el6_6sat.noarch
* rubygem-hammer_cli_foreman_tasks-0.0.3.1-1.el6_6sat.noarch
* rubygem-hammer_cli_import-0.10.6-1.el6sat.noarch
* rubygem-hammer_cli_katello-0.0.7.1-1.el6_6sat.noarch

How reproducible:


Steps to Reproduce:
1. Install latest compose Satellite-6.1.0-RHEL-6-20150121.0
2. Try to create a custom repo and sync it
3.

Actual results:



Expected results:


Additional info:
Comment 2 Og Maciel 2015-01-23 11:18:14 EST
Upstream/nighly also has the same issue but the katello deploy script may be masking it as it currently disables SELinux:

* https://github.com/Katello/katello-deploy/blob/master/setup.rb#L80
Comment 3 Justin Sherrill 2015-01-23 11:22:55 EST
To also clarify, the pulp celery processes used to be unconstrained, but they no longer are.
Comment 5 Lukas Zapletal 2015-01-26 05:50:35 EST
There are several problems.

A) Satellite6 tries to connect to AMQP/qpidd port. We never had this rule in. Filed a different BZ to handle this: https://bugzilla.redhat.com/show_bug.cgi?id=1185801

B) There are problems with Celery reading katello-default-ca.crt which is labeled cert_t. This should be added to Pulp SELinux policy as this is pretty standard configuration case (configure celery with SSL -> it needs to read the files labeled with cert_t which is policy standard type):

miscfiles_manage_cert_files(celery_t)

C) There are some issues with qpidd startup, which might be RHEL6 bug or misconfiguration on our side. I will investiagete further:


time->Wed Jan 21 15:38:32 2015
type=SYSCALL msg=audit(1421872712.984:567): arch=c000003e syscall=59 success=yes exit=0 a0=16de870 a1=16dddb0 a2=16dcc00 a3=7fffb865c380 items=0 ppid=17285 pid=17286 auid=0 uid=498 gid=498 euid=498 suid=498 fsuid=498 egid=498 sgid=498 fsgid=498 tty=(none) ses=1 comm="qpidd" exe="/usr/sbin/qpidd" subj=unconfined_u:system_r:qpidd_t:s0 key=(null)
type=AVC msg=audit(1421872712.984:567): avc:  denied  { read } for  pid=17286 comm="qpidd" path="/etc/rc.d/init.d/qpidd" dev=dm-1 ino=6030256 scontext=unconfined_u:system_r:qpidd_t:s0 tcontext=system_u:object_r:qpidd_initrc_exec_t:s0 tclass=file

So Satellite 6 team is solving A and C, but case B is on Pulp team. I need input from the Pulp team on this.
Comment 6 Brian Bouterse 2015-01-26 11:47:57 EST
The base pulp policy SELinux policy expects a type called pulp_cert_t to be applied to certs that Celery pulp should manage. By default those files usually exist in /etc/pki/pulp/* per [0]. Pulp celery workers should be able to read and write files with the pulp_cert_t type with this rule [1]. That should resolve your issue, but let me know if it does not.

[0]: https://github.com/pulp/pulp/blob/master/server/selinux/server/pulp-server.fc#L1
Comment 7 Brian Bouterse 2015-01-26 11:50:08 EST
Forgot second link from comment 6

[1]: https://github.com/pulp/pulp/blob/master/server/selinux/server/pulp-celery.te#L51
Comment 8 Lukas Zapletal 2015-01-27 04:02:01 EST
Thanks for explanation, I have a request:

https://github.com/pulp/pulp/pull/1575

Since cert_t is generic label for most certificates, I've added new rule to allow reading of these (not writing as I don't expect Pulp to write the CA file). If you accept this patch, please backport it into Satellite 6 version of Pulp.

Because our CA file is already used in other services which expect the generic cert_t, we'd need to create a symlink with pulp_cert_t otherwise. I don't like this solution and it's less secure as I don't think Pulp should have write access to the CA file and the key.

You could think this is a "hack" in Pulp for Satellite 6, but there can be more users providing their own CA certificates. For example if you generate a CA certificate using the standard OpenSSL tools in a different /etc/pki directory, it will end up with cert_t anyway so Pulp will not work until the user changes this to pulp_cert_t. This is unnecessary I think, the macro is distributed in the RHEL6/7 and Fedora core policies and it's a standard practice to allow reading of that.

I'd recommend you to drop the write access to CA cert/key for future versions to improve security. But this is a different story.

Thanks.
Comment 12 Brad Buckingham 2015-01-29 13:53:59 EST
*** Bug 1186760 has been marked as a duplicate of this bug. ***
Comment 16 Og Maciel 2015-03-06 13:46:09 EST
I could no longer see the same issue with latest build.

Verified on Satellite-6.1.0-RHEL-6-20150303.0
Comment 17 Bryan Kearney 2015-08-11 09:35:09 EDT
This bug is slated to be released with Satellite 6.1.
Comment 18 errata-xmlrpc 2015-08-12 01:22:13 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:1592

Note You need to log in before you can comment on or make changes to this bug.