Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 1226770[details]
complete logs from console
Description of problem:
I was trying to upgrade a sat6.2.4 instance installed on rhel68 using 6.2.5 snap1 and installer throws following error at migrate_foreman step:
Upgrade Step: fix_katello_settings_file...
Upgrade Step: migrate_foreman...
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
true
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
E, [2016-12-01T05:57:33.203441 #11319] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:142:in `any_executor'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:134:in `find_receiver'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:86:in `handle_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:6:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:26:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
<tuncate>
E, [2016-12-01T05:57:33.203979 #11319] ERROR -- /client-dispatcher: No executor available (Dynflow::Error)
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:142:in `any_executor'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:134:in `find_receiver'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:86:in `handle_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:6:in `on_message'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:26:in `on_envelope'
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass'
even though installer throws above error, upgrade completed successfully. So first thing is we need to ensure if above errors are harmful and then we should fail the installer as soon as some error appears instead of completing it.
Version-Release number of selected component (if applicable):
upgrade sat6.2.4 -> Sat6.2.5 on rhel68
How reproducible:
Steps to Reproduce:
1. satellite-installer --scenario satellite --upgrade
2.
3.
Actual results:
error at 'migrate_foreman' step
Expected results:
no error should be raised and if something goes wrong installer should fail
Additional info:
How reproducible this issue is? How the insteance looked like before starting the upgrade?
At the beginning of the migration, the system runs a consistency check to clean up some invalid info about running worker processes, and tries to distribute the work elsewhere if it's the case. However, this logic was not touched in 6.2.5, so it should not be considered a regression. Also, the fact that the upgrade was successful at the end leads to assumption, that this issue was there before unnoticed.
I tried to reproduce the issue by upgrade sat6.2.4 -> Sat6.2.5 on rhel68. I started with fresh Sat 6.2.4 installation and the upgrade went fine including the migrate_foreman step.
Earlier in the provided upgrade log I noticed the pulp services were down before the upgrade. Is it possible that the Sat was in some unusual state? I'll check the foreman-debug if there is anything interesting there.
@Ivan: Before upgrade, 6.2.4 instance was alright. After `yum update` I rebooted the instance due to a kernel update, so you may see some services were failing before upgrade.
I tried upgrade only once, but state of instance before upgrade was stable.
I'll try to reproduce this again with 6.2.5 snap3
Ok, I tried upgrade from sat6.2.4 -> sat6.2.5 snap3 on rhel68. This time also I performed same steps as I did before. But I didn't see the reported issue.
This time I didn't get any error at migrate_foreman step. Please see the attached logs.
Created attachment 1226770 [details] complete logs from console Description of problem: I was trying to upgrade a sat6.2.4 instance installed on rhel68 using 6.2.5 snap1 and installer throws following error at migrate_foreman step: Upgrade Step: fix_katello_settings_file... Upgrade Step: migrate_foreman... API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache. true API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache. E, [2016-12-01T05:57:33.203441 #11319] ERROR -- /connector-database-core: No executor available (Dynflow::Error) /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:142:in `any_executor' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:134:in `find_receiver' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:86:in `handle_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:6:in `on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:26:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope' <tuncate> E, [2016-12-01T05:57:33.203979 #11319] ERROR -- /client-dispatcher: No executor available (Dynflow::Error) /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:142:in `any_executor' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:134:in `find_receiver' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:86:in `handle_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:6:in `on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:26:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' even though installer throws above error, upgrade completed successfully. So first thing is we need to ensure if above errors are harmful and then we should fail the installer as soon as some error appears instead of completing it. Version-Release number of selected component (if applicable): upgrade sat6.2.4 -> Sat6.2.5 on rhel68 How reproducible: Steps to Reproduce: 1. satellite-installer --scenario satellite --upgrade 2. 3. Actual results: error at 'migrate_foreman' step Expected results: no error should be raised and if something goes wrong installer should fail Additional info: