Bug 1665466
| Summary: | satellite-installer --upgrade qpid-config returned 1 instead of one of [0] | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Ales Dujicek <adujicek> | ||||
| Component: | Installation | Assignee: | Evgeni Golov <egolov> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Perry Gagne <pgagne> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 6.5.0 | CC: | dhjoshi, egolov, ehelms, janarula, jbhatia, mlele, nkathole, pmoravec, rajukuma, rbertolj, smane, zhunting | ||||
| Target Milestone: | 6.5.0 | Keywords: | Triaged, Upgrades | ||||
| Target Release: | Unused | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | katello-installer-base-3.10.0.1-1 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-05-14 12:39:45 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1653153 | ||||||
| Attachments: |
|
||||||
|
Description
Ales Dujicek
2019-01-11 13:53:08 UTC
Ales, Nikhil, thanks for raising this and the initial analysis. Looking at the installer logs you provided (both, the attached, and on the box Nikhil gave me), I see the same pattern: [ INFO 2019-01-08T15:35:32 main] /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]: Starting to evaluate the resource [DEBUG 2019-01-08T15:35:32 main] Exec[qpid-config ensure queue katello_event_queue](provider=posix): Executing check 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 queues katello_event_queue' [DEBUG 2019-01-08T15:35:32 main] Executing: 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 queues katello_event_queue' [DEBUG 2019-01-08T15:35:33 main] /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]/unless: Failed: ConnectError: [Errno 111] Connection refused [DEBUG 2019-01-08T15:35:33 main] Exec[qpid-config ensure queue katello_event_queue](provider=posix): Executing 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' [DEBUG 2019-01-08T15:35:33 main] Executing: 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' [ WARN 2019-01-08T15:35:33 main] /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]/returns: Failed: ConnectError: [Errno 111] Connection refused [ERROR 2019-01-08T15:35:33 main] 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' returned 1 instead of one of [0] We run "qpid-config … queues katello_event_queue" to verify the queue does already exist, and only add it via "qpid-config … add queue katello_event_queue --durable" when the first command exited non-zero. In both cases, it did exit non-zero, with a "Connection refused" error, indicating that qpidd was down at that point. Still need to investigate *why* qpidd wasn't listening at that time. Okay, I *think* this is because qpidd takes longer than 3 seconds to start but systemd already says it's started properly. The simplest reproducer for the underlying issue seems to be: #!/bin/sh i=0 systemctl restart qpidd while ! qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.com-qpid-broker.key -b amqps://localhost:5671 queues katello_event_queue; do echo $i i=$((i+1)) sleep 0.1 done this will loop until the "queues" command succeeds, which happens for me after 0.3-0.4 seconds, but not instantly :/ Created redmine issue https://projects.theforeman.org/issues/25909 from this bug Verified fix in Sat 6.5 snap 14 1. Installed Sat 6.4 2. Upgraded to 6.5 using 'foreman-maintain upgrade' tool 3. Ran satellite-installer --upgrade again. Also according to nkathole the automation issue from comment 3 is no longer happening duplicate of 1644201 ? :) Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:1222 |