| Summary: | Upgrading sat6.2.4 -> 6.2.5 on rhel6 installer throws error at step: migrate_foreman. | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Sachin Ghai <sghai> | ||||||||
| Component: | Upgrades | Assignee: | satellite6-bugs <satellite6-bugs> | ||||||||
| Status: | CLOSED WORKSFORME | QA Contact: | Katello QA List <katello-qa-list> | ||||||||
| Severity: | high | Docs Contact: | |||||||||
| Priority: | unspecified | ||||||||||
| Version: | 6.2.5 | CC: | bbuckingham, inecas, mbacovsk, sghai | ||||||||
| Target Milestone: | Unspecified | Keywords: | Triaged | ||||||||
| Target Release: | Unused | ||||||||||
| Hardware: | Unspecified | ||||||||||
| OS: | Unspecified | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2016-12-06 16:17:11 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Attachments: |
|
||||||||||
Created attachment 1226786 [details]
foreman-debug
How reproducible this issue is? How the insteance looked like before starting the upgrade? At the beginning of the migration, the system runs a consistency check to clean up some invalid info about running worker processes, and tries to distribute the work elsewhere if it's the case. However, this logic was not touched in 6.2.5, so it should not be considered a regression. Also, the fact that the upgrade was successful at the end leads to assumption, that this issue was there before unnoticed. I tried to reproduce the issue by upgrade sat6.2.4 -> Sat6.2.5 on rhel68. I started with fresh Sat 6.2.4 installation and the upgrade went fine including the migrate_foreman step. Earlier in the provided upgrade log I noticed the pulp services were down before the upgrade. Is it possible that the Sat was in some unusual state? I'll check the foreman-debug if there is anything interesting there. @Ivan: Before upgrade, 6.2.4 instance was alright. After `yum update` I rebooted the instance due to a kernel update, so you may see some services were failing before upgrade. I tried upgrade only once, but state of instance before upgrade was stable. I'll try to reproduce this again with 6.2.5 snap3 Ok, I tried upgrade from sat6.2.4 -> sat6.2.5 snap3 on rhel68. This time also I performed same steps as I did before. But I didn't see the reported issue. This time I didn't get any error at migrate_foreman step. Please see the attached logs. Created attachment 1228620 [details]
complete logs from console while running upgrade
Closing this bz for now and will reopen If come across this issue again |
Created attachment 1226770 [details] complete logs from console Description of problem: I was trying to upgrade a sat6.2.4 instance installed on rhel68 using 6.2.5 snap1 and installer throws following error at migrate_foreman step: Upgrade Step: fix_katello_settings_file... Upgrade Step: migrate_foreman... API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache. true API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache. E, [2016-12-01T05:57:33.203441 #11319] ERROR -- /connector-database-core: No executor available (Dynflow::Error) /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:142:in `any_executor' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:134:in `find_receiver' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:86:in `handle_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:6:in `on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:26:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope' <tuncate> E, [2016-12-01T05:57:33.203979 #11319] ERROR -- /client-dispatcher: No executor available (Dynflow::Error) /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:142:in `any_executor' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:134:in `find_receiver' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/connectors/database.rb:86:in `handle_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:6:in `on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.1/lib/dynflow/actor.rb:26:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' even though installer throws above error, upgrade completed successfully. So first thing is we need to ensure if above errors are harmful and then we should fail the installer as soon as some error appears instead of completing it. Version-Release number of selected component (if applicable): upgrade sat6.2.4 -> Sat6.2.5 on rhel68 How reproducible: Steps to Reproduce: 1. satellite-installer --scenario satellite --upgrade 2. 3. Actual results: error at 'migrate_foreman' step Expected results: no error should be raised and if something goes wrong installer should fail Additional info: