Bug 1185401 - celery.worker.consumer:ERROR: consumer: Cannot connect to qpid
Summary: celery.worker.consumer:ERROR: consumer: Cannot connect to qpid
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: SELinux
Version: Nightly
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Lukas Zapletal
QA Contact: Og Maciel
URL:
Whiteboard:
: 1186760 (view as bug list)
Depends On: 1186420
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-01-23 16:14 UTC by Og Maciel
Modified: 2017-07-26 19:43 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1186420 (view as bug list)
Environment:
Last Closed: 2015-08-12 05:22:13 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
audit.log (5.42 MB, text/plain)
2015-01-23 16:14 UTC, Og Maciel
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:1592 0 normal SHIPPED_LIVE Important: Red Hat Satellite 6.1.1 on RHEL 6 2015-08-12 09:04:35 UTC

Description Og Maciel 2015-01-23 16:14:45 UTC
Created attachment 983415 [details]
audit.log

Description of problem:

Our automation noticed that /var/log/messages contained several errors related to celery not being able to connect to qpid. The symptoms were shown by basically pegging the system and preventing us from connecting to the web UI after a few preliminary tests were run.

Justin has spent some time looking into it and it seems that python processes are not able to read the CA file. Since our automation uses SELinux in enforcing mode, the moment we turned off SELinux the issues disappeared.

Attached you will find the audit log.


Version-Release number of selected component (if applicable):

* apr-util-ldap-1.3.9-3.el6_0.1.x86_64
* candlepin-0.9.38-1.el6.noarch
* candlepin-common-1.0.18-1.el6.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.9.38-1.el6.noarch
* candlepin-tomcat6-0.9.38-1.el6.noarch
* elasticsearch-0.90.10-6.el6sat.noarch
* foreman-1.7.1.2-1.el6_6sat.noarch
* foreman-compute-1.7.1.2-1.el6_6sat.noarch
* foreman-gce-1.7.1.2-1.el6_6sat.noarch
* foreman-libvirt-1.7.1.2-1.el6_6sat.noarch
* foreman-ovirt-1.7.1.2-1.el6_6sat.noarch
* foreman-postgresql-1.7.1.2-1.el6_6sat.noarch
* foreman-proxy-1.7.1.1-1.el6_6sat.noarch
* foreman-selinux-1.7.1.1-1.el6_6sat.noarch
* foreman-vmware-1.7.1.2-1.el6_6sat.noarch
* katello-2.2.0.1-1.el6_6sat.noarch
* katello-certs-tools-2.2.1-1.el6_6sat.noarch
* katello-default-ca-1.0-1.noarch
* katello-installer-2.2.0.2-1.el6_6sat.noarch
* katello-installer-base-2.2.0.2-1.el6_6sat.noarch
* katello-server-ca-1.0-1.noarch
* openldap-2.4.39-8.el6.x86_64
* pulp-docker-plugins-0.2.1-0.2.beta.el6_6sat.noarch
* pulp-katello-0.3-4.el6sat.noarch
* pulp-nodes-common-2.5.0-0.7.beta.el6_6sat.noarch
* pulp-nodes-parent-2.5.0-0.7.beta.el6_6sat.noarch
* pulp-puppet-plugins-2.5.0-0.7.beta.el6sat.noarch
* pulp-puppet-tools-2.5.0-0.7.beta.el6sat.noarch
* pulp-rpm-plugins-2.5.0-0.7.beta.el6_6sat.noarch
* pulp-selinux-2.5.0-0.7.beta.el6_6sat.noarch
* pulp-server-2.5.0-0.7.beta.el6_6sat.noarch
* python-ldap-2.3.10-1.el6.x86_64
* ruby193-rubygem-ldap_fluff-0.3.2-1.el6_6sat.noarch
* ruby193-rubygem-net-ldap-0.3.1-3.el6sat.noarch
* ruby193-rubygem-runcible-1.3.0-1.el6_6sat.noarch
* rubygem-hammer_cli-0.1.4.2-1.el6_6sat.noarch
* rubygem-hammer_cli_foreman-0.1.4.2-1.el6_6sat.noarch
* rubygem-hammer_cli_foreman_bootdisk-0.1.2.4-1.el6_6sat.noarch
* rubygem-hammer_cli_foreman_tasks-0.0.3.1-1.el6_6sat.noarch
* rubygem-hammer_cli_import-0.10.6-1.el6sat.noarch
* rubygem-hammer_cli_katello-0.0.7.1-1.el6_6sat.noarch

How reproducible:


Steps to Reproduce:
1. Install latest compose Satellite-6.1.0-RHEL-6-20150121.0
2. Try to create a custom repo and sync it
3.

Actual results:



Expected results:


Additional info:

Comment 2 Og Maciel 2015-01-23 16:18:14 UTC
Upstream/nighly also has the same issue but the katello deploy script may be masking it as it currently disables SELinux:

* https://github.com/Katello/katello-deploy/blob/master/setup.rb#L80

Comment 3 Justin Sherrill 2015-01-23 16:22:55 UTC
To also clarify, the pulp celery processes used to be unconstrained, but they no longer are.

Comment 5 Lukas Zapletal 2015-01-26 10:50:35 UTC
There are several problems.

A) Satellite6 tries to connect to AMQP/qpidd port. We never had this rule in. Filed a different BZ to handle this: https://bugzilla.redhat.com/show_bug.cgi?id=1185801

B) There are problems with Celery reading katello-default-ca.crt which is labeled cert_t. This should be added to Pulp SELinux policy as this is pretty standard configuration case (configure celery with SSL -> it needs to read the files labeled with cert_t which is policy standard type):

miscfiles_manage_cert_files(celery_t)

C) There are some issues with qpidd startup, which might be RHEL6 bug or misconfiguration on our side. I will investiagete further:


time->Wed Jan 21 15:38:32 2015
type=SYSCALL msg=audit(1421872712.984:567): arch=c000003e syscall=59 success=yes exit=0 a0=16de870 a1=16dddb0 a2=16dcc00 a3=7fffb865c380 items=0 ppid=17285 pid=17286 auid=0 uid=498 gid=498 euid=498 suid=498 fsuid=498 egid=498 sgid=498 fsgid=498 tty=(none) ses=1 comm="qpidd" exe="/usr/sbin/qpidd" subj=unconfined_u:system_r:qpidd_t:s0 key=(null)
type=AVC msg=audit(1421872712.984:567): avc:  denied  { read } for  pid=17286 comm="qpidd" path="/etc/rc.d/init.d/qpidd" dev=dm-1 ino=6030256 scontext=unconfined_u:system_r:qpidd_t:s0 tcontext=system_u:object_r:qpidd_initrc_exec_t:s0 tclass=file

So Satellite 6 team is solving A and C, but case B is on Pulp team. I need input from the Pulp team on this.

Comment 6 Brian Bouterse 2015-01-26 16:47:57 UTC
The base pulp policy SELinux policy expects a type called pulp_cert_t to be applied to certs that Celery pulp should manage. By default those files usually exist in /etc/pki/pulp/* per [0]. Pulp celery workers should be able to read and write files with the pulp_cert_t type with this rule [1]. That should resolve your issue, but let me know if it does not.

[0]: https://github.com/pulp/pulp/blob/master/server/selinux/server/pulp-server.fc#L1

Comment 7 Brian Bouterse 2015-01-26 16:50:08 UTC
Forgot second link from comment 6

[1]: https://github.com/pulp/pulp/blob/master/server/selinux/server/pulp-celery.te#L51

Comment 8 Lukas Zapletal 2015-01-27 09:02:01 UTC
Thanks for explanation, I have a request:

https://github.com/pulp/pulp/pull/1575

Since cert_t is generic label for most certificates, I've added new rule to allow reading of these (not writing as I don't expect Pulp to write the CA file). If you accept this patch, please backport it into Satellite 6 version of Pulp.

Because our CA file is already used in other services which expect the generic cert_t, we'd need to create a symlink with pulp_cert_t otherwise. I don't like this solution and it's less secure as I don't think Pulp should have write access to the CA file and the key.

You could think this is a "hack" in Pulp for Satellite 6, but there can be more users providing their own CA certificates. For example if you generate a CA certificate using the standard OpenSSL tools in a different /etc/pki directory, it will end up with cert_t anyway so Pulp will not work until the user changes this to pulp_cert_t. This is unnecessary I think, the macro is distributed in the RHEL6/7 and Fedora core policies and it's a standard practice to allow reading of that.

I'd recommend you to drop the write access to CA cert/key for future versions to improve security. But this is a different story.

Thanks.

Comment 12 Brad Buckingham 2015-01-29 18:53:59 UTC
*** Bug 1186760 has been marked as a duplicate of this bug. ***

Comment 16 Og Maciel 2015-03-06 18:46:09 UTC
I could no longer see the same issue with latest build.

Verified on Satellite-6.1.0-RHEL-6-20150303.0

Comment 17 Bryan Kearney 2015-08-11 13:35:09 UTC
This bug is slated to be released with Satellite 6.1.

Comment 18 errata-xmlrpc 2015-08-12 05:22:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:1592


Note You need to log in before you can comment on or make changes to this bug.