Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1193584 - Race condition causes qpid-configure command to fail
Summary: Race condition causes qpid-configure command to fail
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Installation
Version: Nightly
Hardware: Unspecified
OS: Linux
unspecified
urgent
Target Milestone: Unspecified
Assignee: Eric Helms
QA Contact: Jan Hutař
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-02-17 16:04 UTC by Og Maciel
Modified: 2017-02-23 20:29 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-12 05:26:16 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
foreman-debug (264.18 KB, application/x-xz)
2015-02-17 16:04 UTC, Og Maciel 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 9364 0 Normal Closed Install katello-deploy on Centos 7 2020-08-06 20:26:47 UTC
Red Hat Product Errata RHSA-2015:1592 0 normal SHIPPED_LIVE Important: Red Hat Satellite 6.1.1 on RHEL 6 2015-08-12 09:04:35 UTC

Description Og Maciel 2015-02-17 16:04:07 UTC 
Created attachment 992766 [details]
foreman-debug

Description of problem:

While configuring Katello nightly on my RHEL 7.0 system I kept getting stuck with the following error:

[DEBUG 2015-02-17 10:10:10 main]  Executing 'qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://<SERVER>:5671' exchanges event'
[DEBUG 2015-02-17 10:10:10 main]  /Stage[main]/Certs::Candlepin/Exec[create candlepin qpid exchange]/unless: Failed: ConnectError: [Errno 111] Connection refused
[DEBUG 2015-02-17 10:10:10 main]  Exec[create candlepin qpid exchange](provider=posix): Executing 'qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://<SERVER>:5671' add exchange topic event --durable'
[DEBUG 2015-02-17 10:10:10 main]  Executing 'qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://<SERVER>:5671' add exchange topic event --durable'
[ WARN 2015-02-17 10:10:10 main]  /Stage[main]/Certs::Candlepin/Exec[create candlepin qpid exchange]/returns: Failed: ConnectError: [Errno 111] Connection refused
[ERROR 2015-02-17 10:10:10 main]  qpid-config --ssl-certificate /etc/pki/katello/certs/java-client.crt --ssl-key /etc/pki/katello/private/java-client.key -b 'amqps://<SERVER>:5671' add exchange topic event --durable returned 1 instead of one of [0]

After talking to Al he told me that perhaps the issue is due to systemd may be reporting that qpid has been restarted even though it may still be in the process of doing so, thus causing the command qpid-configure to fail over and over.

Version-Release number of selected component (if applicable):

* candlepin-0.9.41-1.el7.noarch
* candlepin-common-1.0.20-1.el7.noarch
* candlepin-selinux-0.9.41-1.el7.noarch
* candlepin-tomcat-0.9.41-1.el7.noarch
* elasticsearch-0.90.10-7.el7.noarch
* foreman-1.8.0-0.develop.201502121510git9edf91b.el7.noarch
* foreman-compute-1.8.0-0.develop.201502121510git9edf91b.el7.noarch
* foreman-debug-1.8.0-0.develop.201502121510git9edf91b.el7.noarch
* foreman-gce-1.8.0-0.develop.201502121510git9edf91b.el7.noarch
* foreman-libvirt-1.8.0-0.develop.201502121510git9edf91b.el7.noarch
* foreman-ovirt-1.8.0-0.develop.201502121510git9edf91b.el7.noarch
* foreman-postgresql-1.8.0-0.develop.201502121510git9edf91b.el7.noarch
* foreman-proxy-1.8.0-0.develop.201502121459git0207401.el7.noarch
* foreman-release-1.8.0-0.develop.201502121510git9edf91b.el7.noarch
* foreman-selinux-1.8.0-0.develop.201412151103gite2863e4.el7.noarch
* foreman-vmware-1.8.0-0.develop.201502121510git9edf91b.el7.noarch
* katello-2.2.0-1.201502161312git49289e5.el7.noarch
* katello-certs-tools-2.0.1-1.el7.noarch
* katello-common-2.2.0-1.201502161312git49289e5.el7.noarch
* katello-default-ca-1.0-1.noarch
* katello-installer-2.2.0-1.201502160627gite8ff373.el7.noarch
* katello-installer-base-2.2.0-1.201502160627gite8ff373.el7.noarch
* katello-repos-2.1.1-1.el7.noarch
* katello-server-ca-1.0-1.noarch
* openldap-2.4.39-3.el7.x86_64
* pulp-docker-plugins-0.2.1-0.2.beta.el7.noarch
* pulp-katello-0.3-3.el7.noarch
* pulp-nodes-common-2.5.1-1.el7.noarch
* pulp-nodes-parent-2.5.1-1.el7.noarch
* pulp-puppet-plugins-2.5.1-2.katello.el7.noarch
* pulp-puppet-tools-2.5.1-2.katello.el7.noarch
* pulp-rpm-plugins-2.5.1-1.el7.noarch
* pulp-selinux-2.5.1-1.el7.noarch
* pulp-server-2.5.1-1.el7.noarch
* python-ldap-2.4.6-6.el7.x86_64
* ruby193-rubygem-ldap_fluff-0.3.3-1.el7.noarch
* ruby193-rubygem-net-ldap-0.10.0-1.el7.noarch
* ruby193-rubygem-runcible-1.3.1-1.el7.noarch
* rubygem-hammer_cli-0.1.4-1.201502121207git0ab2866.el7.noarch
* rubygem-hammer_cli_foreman-0.1.4-1.201501221305git706b057.el7.noarch
* rubygem-hammer_cli_foreman_bootdisk-0.1.2-1.el7.noarch
* rubygem-hammer_cli_foreman_tasks-0.0.3-2.201409091410git163c264.git.0.988ca80.el7.noarch
* rubygem-hammer_cli_gutterball-0.0.1-1.201501072024git01fe139.git.0.06e884f.el7.noarch
* rubygem-hammer_cli_import-0.10.4-1.el7.noarch
* rubygem-hammer_cli_katello-0.0.7-1.201502061831git68a34d6.git.0.dd5c904.el7.noarch

How reproducible:


Steps to Reproduce:
1. Install katello nightly using the katello-deploy script on RHEL 7.0 system
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Og Maciel 2015-02-17 16:11:41 UTC 
10:52:17 al: omaciel, okay, I think I have an idea
10:52:31 al: here's what happens when I try to connect with a totally bogus certificate:
10:52:38 al: Failed: ConnectError: [Errno 1] _ssl.c:504: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca
10:52:54 al: but in your paste I see [Errno 111] Connection refused
10:53:22 al: is it possible that the qpid daemon is still in the process of restarting when the qpid-configure command is issued?
10:54:08 al: yeah, if I try to connect to a bogus port, I get the same error.
10:54:24 al: You may need a "sleep 10" in the puppet configs somewhere
10:55:27 al: this is on RHEL 7 too and I have noticed that systemd is a lot more lenient that sysVinit about believing that services have started.
10:55:32 omaciel: al: hmmm Let me file a BZ and maybe you can chime in?
10:55:55 al: e.g. when I "systemctl restart tomcat" it returns instantly even though Tomcat will take 20 seconds to start
10:58:42 al: looks like someone else has the same issue: https://github.com/Stebalien/systemd-wait/blob/master/systemd-wait
Comment 2 Brad Buckingham 2015-02-17 16:13:19 UTC 
Adding link to upstream redmine issue which looks very similar.
Comment 3 Alex Wood 2015-02-17 16:19:25 UTC 
After some investigation, I think the problem is that Puppet is trying to connect to the qpid daemon before it has started up completely.

If I try to connect to a port that I know qpid is NOT running on, I get the same error message (note that I'm just running "exchanges" to list the existing exchanges instead of trying to create a new one):

# qpid-config --ssl-certificate /etc/pki/katello/certs/katello-apache.crt --ssl-key /etc/pki/katello/private/katello-apache.key -b 'amqps://cloud-qe-8.idmqe.lab.eng.bos.redhat.com:9000' exchanges
Failed: ConnectError: [Errno 111] Connection refused

If I try to connect with a certificate that I know is bogus, I get a different type of error:

# openssl genrsa -out foo.key 2048
# openssl req -new -key foo.key -out foo.csr -subj "/C=US/CN=localhost"
# openssl x509 -req -in foo.csr -days 365 -signkey foo.key -out foo.crt
# qpid-config --ssl-certificate foo.crt --ssl-key foo.key -b 'amqps://cloud-qe-8.idmqe.lab.eng.bos.redhat.com:5671' exchanges
Failed: ConnectError: [Errno 1] _ssl.c:504: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown ca

And if I run the command after Puppet has already quit, I am able to connect.

This error appears on a RHEL 7 box and systemd is much more aggressive than SysVInit at telling you services have started.  For example, if I restart Tomcat on my system, systemctl returns almost immediately even though Tomcat actually takes about 20 seconds to start.  Other people seem to have this issue as well and have written various solutions:

  https://gist.github.com/ghedo/4b005ce111591781e3fd
  https://github.com/Stebalien/systemd-wait

The puppet configuration scripts will probably either need something like this or just a "sleep 10" if a simpler (albeit less robust) approach is desired.
Comment 4 Bryan Kearney 2015-02-18 21:33:08 UTC 
Connecting redmine issue http://projects.theforeman.org/issues/9364 from this bug
Comment 5 Bryan Kearney 2015-02-18 23:07:14 UTC 
Moving to POST since upstream bug http://projects.theforeman.org/issues/9364 has been closed
-------------
Eric Helms
Testing this patch for this issue -- https://github.com/Katello/puppet-service_wait/pull/8
-------------
Eric Helms
Applied in changeset commit:katello-installer|7097861d5de1c233170050417afaf2e721871ff7.
Comment 9 Bryan Kearney 2015-08-11 13:24:46 UTC 
This bug is slated to be released with Satellite 6.1.
Comment 10 errata-xmlrpc 2015-08-12 05:26:16 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:1592
Note You need to log in before you can comment on or make changes to this bug.