Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 845224 - Pulp can't connect to qpid on RHEL 6.2
Summary: Pulp can't connect to qpid on RHEL 6.2
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Infrastructure
Version: 6.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: Unspecified
Assignee: Lukas Zapletal
QA Contact: Og Maciel
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-08-02 10:55 UTC by Ivan Necas
Modified: 2019-09-26 15:52 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
The QPID messaging system would fail to start correctly due to incorrect ordering of commands in the deployment configuration, which led to a misnamed certificate. This prevented the QPID daemon from starting with SSL enabled, breaking communication between pulp and QPID. This fix add the broker cert before the broker cert private key. This ensures the name of the cert is correctly listed as 'broker'.
Clone Of:
Environment:
Last Closed: 2012-12-04 19:51:15 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
katello-debug archive (93.76 KB, application/x-compressed-tar)
2012-08-02 10:55 UTC, Ivan Necas 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2012:1543 0 normal SHIPPED_LIVE Important: CloudForms System Engine 1.1 update 2012-12-05 00:39:57 UTC

Description Ivan Necas 2012-08-02 10:55:07 UTC 
Created attachment 601921 [details]
katello-debug archive

Description of problem:
After installation of katello using katello-configure, there is this message in /var/log/pulp/pulp.log

qpid.messaging:WARNING: driver:444 recoverable error[attempt 8]: [Errno 111] Con
nection refused

Version-Release number of selected component (if applicable):

katello-1.0.2-1.el6.noarch
pulp-1.1.11-1.el6
rhel-6.2 without updates

How reproducible:
Always on rhel-6.2

Steps to Reproduce:
1. install katello with katello-configure
2. see the /var/log/pulp/pulp.log
3. you can also try registering and unregistering the machine


Actual results:
error messages in the log, unregistering fails significantly

Expected results:
no error messages in the log, unregistering works fine

Additional info:
seems to work on rhel-6.3
Comment 1 Ivan Necas 2012-08-02 11:07:26 UTC 
It seems qpid fails to start with ssl: this is shown in /var/log/messages:

Aug  2 12:36:47 rhel62-kat3 qpidd[28915]: 2012-08-02 12:36:47 error Failed to in
itialise SSL plugin: Failed to load certificate 'broker' (qpid/sys/ssl/SslSocket
.cpp:184)
Aug  2 12:38:22 rhel62-kat3 qpidd[29042]: 2012-08-02 12:38:22 error Failed to in
itialise SSL plugin: Failed to load certificate 'broker' (qpid/sys/ssl/SslSocket
.cpp:184)
Aug  2 12:39:53 rhel62-kat3 yum[29123]: Updated: openssl-1.0.0-20.el6_2.5.x86_64

Aug  2 12:43:28 rhel62-kat3 qpidd[29238]: 2012-08-02 12:43:28 error Failed to in
itialise SSL plugin: Failed to load certificate 'broker' (qpid/sys/ssl/SslSocket
.cpp:184)
Comment 2 Ivan Necas 2012-08-02 18:29:29 UTC 
Adding private key for the broker cert to nssdb before the cert itself caused
the cert being saved under different name than 'broker' and causes communication
between Pulp and Qpid not working.

Opened a pull request https://github.com/Katello/katello/pull/423 fixing this issue.

Note:

It was not caused by different version of RHEL, but another ordering when running puppet on different machines.
Comment 3 Ivan Necas 2012-08-03 12:51:21 UTC 
Fixed in commit dc04b15
Comment 4 Lukas Zapletal 2012-08-03 15:06:58 UTC 
Merged.
Comment 6 Og Maciel 2012-09-16 13:07:02 UTC 
Ivan:

Found the following message in /var/log/pulp/pulp.log:

  2012-09-14 16:19:17,824 17479:140036293129984: gofer.messaging.broker:INFO:  broker:100 connecting:
  {localhost:5671}:
  transport=SSL
  host=localhost
  port=5671
  cacert=/usr/share/katello/candlepin-cert.crt
  clientcert=/etc/pki/pulp/qpid_client_striped.crt
  2012-09-14 16:19:17,839 17479:140036293129984: qpid.messaging:WARNING: driver:444 recoverable error[attempt 1]: [Errno 111] Connection refused
  2012-09-14 16:19:17,839 17479:140036293129984: qpid.messaging:WARNING: driver:446 sleeping 1 seconds

Further down:

  2012-09-14 16:19:17,856 17479:140036684912608: pulp.server.content.loader:WARNING: loader:393 Profilers load called, but not implemented
  2012-09-14 16:19:18,840 17479:140036066506496: qpid.messaging:WARNING: driver:523 trying: localhost:5671
  2012-09-14 16:19:18,841 17479:140036066506496: qpid.messaging:WARNING: driver:444 recoverable error[attempt 2]: [Errno 111] Connection refused
  2012-09-14 16:19:18,842 17479:140036066506496: qpid.messaging:WARNING: driver:446 sleeping 2 seconds

this repeats for a bit and then

  2012-09-14 16:20:20,889 17479:140036293129984: gofer.messaging.broker:INFO: broker:103 {localhost:5671} connected to AMQP
  2012-09-14 16:21:42,062 17934:139920444143584: pulp.server.db.connection:INFO: connection:46 Attempting Database connection with seeds = localhost
  2012-09-14 16:21:42,067 17934:139920444143584: pulp.server.db.connection:INFO: connection:51 Database connection established with: seeds = localhost, name = pulp_database
  2012-09-14 16:21:42,704 17934:139920444143584: pulp.server.async:INFO: async:404 Task reply handler, started.
  2012-09-14 16:21:42,705 17934:139920052360960: gofer.messaging.broker:INFO: broker:100 connecting:
  {localhost:5671}:
  transport=SSL
  host=localhost
  port=5671
  cacert=/usr/share/katello/candlepin-cert.crt
  clientcert=/etc/pki/pulp/qpid_client_striped.crt
  2012-09-14 16:21:42,754 17934:139920444143584: pulp.server.content.types.parser:INFO: parser:141 Loading type descriptors []
  2012-09-14 16:21:42,754 17934:139920444143584: pulp.server.content.types.parser:INFO: parser:143 Parsing type descriptors
  2012-09-14 16:21:42,755 17934:139920444143584: pulp.server.content.types.parser:INFO: parser:146 Validating type descriptor syntactic integrity
  2012-09-14 16:21:42,755 17934:139920444143584: pulp.server.content.types.parser:INFO: parser:149 Validating type descriptor semantic integrity
  2012-09-14 16:21:42,756 17934:139920444143584: pulp.server.content.types.database:INFO: database:83 Updating the database with types []
  2012-09-14 16:21:42,759 17934:139920444143584: pulp.server.content.loader:WARNING: loader:393 Profilers load called, but not implemented
  2012-09-14 16:21:42,770 17934:139920052360960: gofer.messaging.broker:INFO: broker:103 {localhost:5671} connected to AMQP

Finally, /var/log/messages:

  Jul 13 16:05:30 SERVER qpidd[4800]: 2012-07-13 16:05:30 notice Listening on TCP port 5672
  Jul 13 16:05:30 SERVER qpidd[4800]: 2012-07-13 16:05:30 notice SSL plugin not enabled, you must set --ssl-cert-db to enable it.
  Jul 13 16:05:30 SERVER qpidd[4800]: 2012-07-13 16:05:30 notice Broker running
Comment 7 Ivan Necas 2012-09-17 07:52:31 UTC 
It's possible this might happen during the installation time (e.g. when restarting qpid service while pulp is running or reconfiguring the ports). Another symptom of this was the `subscription-manager unregister` was not working with this error. So if it's working, it means Pulp <-> QPID communication works fine.
Comment 8 Og Maciel 2012-09-17 12:27:38 UTC 
Verified using:

* candlepin-0.7.8-1.el6cf.noarch
* candlepin-selinux-0.7.8-1.el6cf.noarch
* candlepin-tomcat6-0.7.8-1.el6cf.noarch
* katello-1.1.12-7.el6cf.noarch
* katello-all-1.1.12-7.el6cf.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.1.8-1.el6cf.noarch
* katello-cli-1.1.8-4.el6cf.noarch
* katello-cli-common-1.1.8-4.el6cf.noarch
* katello-common-1.1.12-7.el6cf.noarch
* katello-configure-1.1.9-3.el6cf.noarch
* katello-glue-candlepin-1.1.12-7.el6cf.noarch
* katello-glue-pulp-1.1.12-7.el6cf.noarch
* katello-qpid-broker-key-pair-1.0-1.noarch
* katello-qpid-client-key-pair-1.0-1.noarch
* katello-selinux-1.1.1-1.el6cf.noarch
* pulp-1.1.12-1.el6cf.noarch
* pulp-common-1.1.12-1.el6cf.noarch
* pulp-selinux-server-1.1.12-1.el6cf.noarch
Comment 10 errata-xmlrpc 2012-12-04 19:51:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-1543.html
Note You need to log in before you can comment on or make changes to this bug.