Created attachment 1520027 [details] satellite.log Description of problem: After upgrade from 6.4 I run upgrade once more and hit the following error: # satellite-installer --upgrade Resetting puppet server version param... Upgrading, to monitor the progress on all related services, please do: foreman-tail | tee upgrade-$(date +%Y-%m-%d-%H%M).log Upgrade Step: stop_services... Redirecting to 'foreman-maintain service' Running Stop Services ... Upgrade Step: Running installer... 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' returned 1 instead of one of [0] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:164:in `sync' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `sync' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:88:in `block in perform_changes' ... /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute' /opt/puppetlabs/puppet/bin/puppet:5:in `<main>' /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]/returns: change from 'notrun' to ['0'] failed: 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' returned 1 instead of one of [0] Installing Done [100%] Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/satellite.log Upgrade failed during the installation phase. Fix the error and re-run the upgrade. it is described also in upstream https://community.theforeman.org/t/1-19-1-20-upgrade-fail-add-queue-katello-event-queue-durable-returned-1-instead-of-one-of-0/11982 it is probably not serious as it does not happen again if you rerun installer one more time and also it does not happen when you restart services first (foreman-maintain service restart) and then run upgrade attaching installer log Version-Release number of selected component (if applicable): 6.5 snap 10 satellite-6.5.0-5.beta.el7sat.noarch foreman-1.20.1.3-1.el7sat.noarch satellite-installer-6.5.0.3-1.beta.el7sat.noarch
Ales, Nikhil, thanks for raising this and the initial analysis. Looking at the installer logs you provided (both, the attached, and on the box Nikhil gave me), I see the same pattern: [ INFO 2019-01-08T15:35:32 main] /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]: Starting to evaluate the resource [DEBUG 2019-01-08T15:35:32 main] Exec[qpid-config ensure queue katello_event_queue](provider=posix): Executing check 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 queues katello_event_queue' [DEBUG 2019-01-08T15:35:32 main] Executing: 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 queues katello_event_queue' [DEBUG 2019-01-08T15:35:33 main] /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]/unless: Failed: ConnectError: [Errno 111] Connection refused [DEBUG 2019-01-08T15:35:33 main] Exec[qpid-config ensure queue katello_event_queue](provider=posix): Executing 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' [DEBUG 2019-01-08T15:35:33 main] Executing: 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' [ WARN 2019-01-08T15:35:33 main] /Stage[main]/Katello::Qpid/Qpid::Config::Queue[katello_event_queue]/Qpid::Config_cmd[ensure queue katello_event_queue]/Exec[qpid-config ensure queue katello_event_queue]/returns: Failed: ConnectError: [Errno 111] Connection refused [ERROR 2019-01-08T15:35:33 main] 'qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.redhat.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.redhat.com-qpid-broker.key -b amqps://localhost:5671 add queue katello_event_queue --durable' returned 1 instead of one of [0] We run "qpid-config … queues katello_event_queue" to verify the queue does already exist, and only add it via "qpid-config … add queue katello_event_queue --durable" when the first command exited non-zero. In both cases, it did exit non-zero, with a "Connection refused" error, indicating that qpidd was down at that point. Still need to investigate *why* qpidd wasn't listening at that time.
Okay, I *think* this is because qpidd takes longer than 3 seconds to start but systemd already says it's started properly.
The simplest reproducer for the underlying issue seems to be: #!/bin/sh i=0 systemctl restart qpidd while ! qpid-config --ssl-certificate /etc/pki/katello/certs/satellite.example.com-qpid-broker.crt --ssl-key /etc/pki/katello/private/satellite.example.com-qpid-broker.key -b amqps://localhost:5671 queues katello_event_queue; do echo $i i=$((i+1)) sleep 0.1 done this will loop until the "queues" command succeeds, which happens for me after 0.3-0.4 seconds, but not instantly :/
Created redmine issue https://projects.theforeman.org/issues/25909 from this bug
Verified fix in Sat 6.5 snap 14 1. Installed Sat 6.4 2. Upgraded to 6.5 using 'foreman-maintain upgrade' tool 3. Ran satellite-installer --upgrade again. Also according to nkathole the automation issue from comment 3 is no longer happening
duplicate of 1644201 ? :)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:1222