Red Hat Bugzilla – Bug 845224
Pulp can't connect to qpid on RHEL 6.2
Last modified: 2013-03-27 16:07:42 EDT
Created attachment 601921 [details] katello-debug archive Description of problem: After installation of katello using katello-configure, there is this message in /var/log/pulp/pulp.log qpid.messaging:WARNING: driver:444 recoverable error[attempt 8]: [Errno 111] Con nection refused Version-Release number of selected component (if applicable): katello-1.0.2-1.el6.noarch pulp-1.1.11-1.el6 rhel-6.2 without updates How reproducible: Always on rhel-6.2 Steps to Reproduce: 1. install katello with katello-configure 2. see the /var/log/pulp/pulp.log 3. you can also try registering and unregistering the machine Actual results: error messages in the log, unregistering fails significantly Expected results: no error messages in the log, unregistering works fine Additional info: seems to work on rhel-6.3
It seems qpid fails to start with ssl: this is shown in /var/log/messages: Aug 2 12:36:47 rhel62-kat3 qpidd[28915]: 2012-08-02 12:36:47 error Failed to in itialise SSL plugin: Failed to load certificate 'broker' (qpid/sys/ssl/SslSocket .cpp:184) Aug 2 12:38:22 rhel62-kat3 qpidd[29042]: 2012-08-02 12:38:22 error Failed to in itialise SSL plugin: Failed to load certificate 'broker' (qpid/sys/ssl/SslSocket .cpp:184) Aug 2 12:39:53 rhel62-kat3 yum[29123]: Updated: openssl-1.0.0-20.el6_2.5.x86_64 Aug 2 12:43:28 rhel62-kat3 qpidd[29238]: 2012-08-02 12:43:28 error Failed to in itialise SSL plugin: Failed to load certificate 'broker' (qpid/sys/ssl/SslSocket .cpp:184)
Adding private key for the broker cert to nssdb before the cert itself caused the cert being saved under different name than 'broker' and causes communication between Pulp and Qpid not working. Opened a pull request https://github.com/Katello/katello/pull/423 fixing this issue. Note: It was not caused by different version of RHEL, but another ordering when running puppet on different machines.
Fixed in commit dc04b15
Merged.
Ivan: Found the following message in /var/log/pulp/pulp.log: 2012-09-14 16:19:17,824 17479:140036293129984: gofer.messaging.broker:INFO: broker:100 connecting: {localhost:5671}: transport=SSL host=localhost port=5671 cacert=/usr/share/katello/candlepin-cert.crt clientcert=/etc/pki/pulp/qpid_client_striped.crt 2012-09-14 16:19:17,839 17479:140036293129984: qpid.messaging:WARNING: driver:444 recoverable error[attempt 1]: [Errno 111] Connection refused 2012-09-14 16:19:17,839 17479:140036293129984: qpid.messaging:WARNING: driver:446 sleeping 1 seconds Further down: 2012-09-14 16:19:17,856 17479:140036684912608: pulp.server.content.loader:WARNING: loader:393 Profilers load called, but not implemented 2012-09-14 16:19:18,840 17479:140036066506496: qpid.messaging:WARNING: driver:523 trying: localhost:5671 2012-09-14 16:19:18,841 17479:140036066506496: qpid.messaging:WARNING: driver:444 recoverable error[attempt 2]: [Errno 111] Connection refused 2012-09-14 16:19:18,842 17479:140036066506496: qpid.messaging:WARNING: driver:446 sleeping 2 seconds this repeats for a bit and then 2012-09-14 16:20:20,889 17479:140036293129984: gofer.messaging.broker:INFO: broker:103 {localhost:5671} connected to AMQP 2012-09-14 16:21:42,062 17934:139920444143584: pulp.server.db.connection:INFO: connection:46 Attempting Database connection with seeds = localhost 2012-09-14 16:21:42,067 17934:139920444143584: pulp.server.db.connection:INFO: connection:51 Database connection established with: seeds = localhost, name = pulp_database 2012-09-14 16:21:42,704 17934:139920444143584: pulp.server.async:INFO: async:404 Task reply handler, started. 2012-09-14 16:21:42,705 17934:139920052360960: gofer.messaging.broker:INFO: broker:100 connecting: {localhost:5671}: transport=SSL host=localhost port=5671 cacert=/usr/share/katello/candlepin-cert.crt clientcert=/etc/pki/pulp/qpid_client_striped.crt 2012-09-14 16:21:42,754 17934:139920444143584: pulp.server.content.types.parser:INFO: parser:141 Loading type descriptors [] 2012-09-14 16:21:42,754 17934:139920444143584: pulp.server.content.types.parser:INFO: parser:143 Parsing type descriptors 2012-09-14 16:21:42,755 17934:139920444143584: pulp.server.content.types.parser:INFO: parser:146 Validating type descriptor syntactic integrity 2012-09-14 16:21:42,755 17934:139920444143584: pulp.server.content.types.parser:INFO: parser:149 Validating type descriptor semantic integrity 2012-09-14 16:21:42,756 17934:139920444143584: pulp.server.content.types.database:INFO: database:83 Updating the database with types [] 2012-09-14 16:21:42,759 17934:139920444143584: pulp.server.content.loader:WARNING: loader:393 Profilers load called, but not implemented 2012-09-14 16:21:42,770 17934:139920052360960: gofer.messaging.broker:INFO: broker:103 {localhost:5671} connected to AMQP Finally, /var/log/messages: Jul 13 16:05:30 SERVER qpidd[4800]: 2012-07-13 16:05:30 notice Listening on TCP port 5672 Jul 13 16:05:30 SERVER qpidd[4800]: 2012-07-13 16:05:30 notice SSL plugin not enabled, you must set --ssl-cert-db to enable it. Jul 13 16:05:30 SERVER qpidd[4800]: 2012-07-13 16:05:30 notice Broker running
It's possible this might happen during the installation time (e.g. when restarting qpid service while pulp is running or reconfiguring the ports). Another symptom of this was the `subscription-manager unregister` was not working with this error. So if it's working, it means Pulp <-> QPID communication works fine.
Verified using: * candlepin-0.7.8-1.el6cf.noarch * candlepin-selinux-0.7.8-1.el6cf.noarch * candlepin-tomcat6-0.7.8-1.el6cf.noarch * katello-1.1.12-7.el6cf.noarch * katello-all-1.1.12-7.el6cf.noarch * katello-candlepin-cert-key-pair-1.0-1.noarch * katello-certs-tools-1.1.8-1.el6cf.noarch * katello-cli-1.1.8-4.el6cf.noarch * katello-cli-common-1.1.8-4.el6cf.noarch * katello-common-1.1.12-7.el6cf.noarch * katello-configure-1.1.9-3.el6cf.noarch * katello-glue-candlepin-1.1.12-7.el6cf.noarch * katello-glue-pulp-1.1.12-7.el6cf.noarch * katello-qpid-broker-key-pair-1.0-1.noarch * katello-qpid-client-key-pair-1.0-1.noarch * katello-selinux-1.1.1-1.el6cf.noarch * pulp-1.1.12-1.el6cf.noarch * pulp-common-1.1.12-1.el6cf.noarch * pulp-selinux-server-1.1.12-1.el6cf.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-1543.html