Bug 1665466 - satellite-installer --upgrade qpid-config returned 1 instead of one of [0]
Summary: satellite-installer --upgrade qpid-config returned 1 instead of one of [0]
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Installer
Version: 6.5.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent vote
Target Milestone: Released
Assignee: Evgeni Golov
QA Contact: Perry Gagne
URL:
Whiteboard:
Depends On:
Blocks: 1653153
TreeView+ depends on / blocked
 
Reported: 2019-01-11 13:53 UTC by Ales Dujicek
Modified: 2019-10-07 17:19 UTC (History)
10 users (show)

Fixed In Version: katello-installer-base-3.10.0.1-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-05-14 12:39:45 UTC


Attachments (Terms of Use)
satellite.log (3.07 MB, text/plain)
2019-01-11 13:53 UTC, Ales Dujicek
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:1222 None None None 2019-05-14 12:39:54 UTC
Github theforeman puppet-qpid pull 113 None None None 2019-01-25 20:15:25 UTC
Foreman Issue Tracker 25909 None None None 2019-01-23 11:08:05 UTC

Description Ales Dujicek 2019-01-11 13:53:08 UTC
Created attachment 1520027 [details]
satellite.log

Description of problem:

After upgrade from 6.4 I run upgrade once more and hit the following error:

# satellite-installer --upgrade
Resetting puppet server version param...
Upgrading, to monitor the progress on all related services, please do:
  foreman-tail | tee upgrade-$(date +%Y-%m-%d-%H%M).log
Upgrade Step: stop_services...
Redirecting to 'foreman-maintain service'
Running Stop Services
...
Upgrade Step: Running installer...
 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' returned 1 instead of one of [0]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:164:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes'
...
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
 /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]/returns: change from 'notrun' to ['0'] failed: 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' returned 1 instead of one of [0]
Installing             Done                                               [100%]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade failed during the installation phase. Fix the error and re-run the upgrade.


it is described also in upstream
https://community.theforeman.org/t/1-19-1-20-upgrade-fail-add-queue-katello-event-queue-durable-returned-1-instead-of-one-of-0/11982

it is probably not serious as it does not happen again if you rerun installer one more time
and also it does not happen when you restart services first (foreman-maintain service restart) and then run upgrade

attaching installer log


Version-Release number of selected component (if applicable):
6.5 snap 10
satellite-6.5.0-5.beta.el7sat.noarch
foreman-1.20.1.3-1.el7sat.noarch
satellite-installer-6.5.0.3-1.beta.el7sat.noarch

Comment 4 Evgeni Golov 2019-01-22 08:29:28 UTC
Ales, Nikhil, thanks for raising this and the initial analysis.

Looking at the installer logs you provided (both, the attached, and on the box Nikhil gave me), I see the same pattern:

[ INFO 2019-01-08T15:35:32 main]  /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]: Starting to evaluate the resource
[DEBUG 2019-01-08T15:35:32 main]  Exec[qpid-config ensure queue katello_event_queue](provider=posix): Executing check 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 queues katello_event_queue'
[DEBUG 2019-01-08T15:35:32 main]  Executing: 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 queues katello_event_queue'
[DEBUG 2019-01-08T15:35:33 main]  /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]/unless: Failed: ConnectError: [Errno 111] Connection refused
[DEBUG 2019-01-08T15:35:33 main]  Exec[qpid-config ensure queue katello_event_queue](provider=posix): Executing 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable'
[DEBUG 2019-01-08T15:35:33 main]  Executing: 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable'
[ WARN 2019-01-08T15:35:33 main]  /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]/returns: Failed: ConnectError: [Errno 111] Connection refused
[ERROR 2019-01-08T15:35:33 main]  'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' returned 1 instead of one of [0]

We run "qpid-config … queues katello_event_queue" to verify the queue does already exist, and only add it via "qpid-config … add queue katello_event_queue --durable" when the first command exited non-zero.
In both cases, it did exit non-zero, with a "Connection refused" error, indicating that qpidd was down at that point.

Still need to investigate *why* qpidd wasn't listening at that time.

Comment 5 Evgeni Golov 2019-01-22 08:46:35 UTC
Okay, I *think* this is because qpidd takes longer than 3 seconds to start but systemd already says it's started properly.

Comment 6 Evgeni Golov 2019-01-22 13:21:30 UTC
The simplest reproducer for the underlying issue seems to be:

#!/bin/sh

i=0
systemctl restart qpidd
while ! qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.com-qpid-broker.key -b amqps://localhost:5671 queues katello_event_queue; do
  echo $i
  i=$((i+1))
  sleep 0.1
done

this will loop until the "queues" command succeeds, which happens for me after 0.3-0.4 seconds, but not instantly :/

Comment 7 Evgeni Golov 2019-01-23 11:08:05 UTC
Created redmine issue https://projects.theforeman.org/issues/25909 from this bug

Comment 11 Perry Gagne 2019-02-05 18:31:00 UTC
Verified fix in Sat 6.5 snap 14

1. Installed Sat 6.4
2. Upgraded to 6.5 using 'foreman-maintain upgrade' tool
3. Ran satellite-installer --upgrade again. 

Also according to nkathole the automation issue from comment 3 is no longer happening

Comment 13 Pavel Moravec 2019-03-03 16:41:46 UTC
duplicate of 1644201 ? :)

Comment 16 errata-xmlrpc 2019-05-14 12:39:45 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222


Note You need to log in before you can comment on or make changes to this bug.