Bug 1413966

Summary: Upgrade w/ customer db raises "No executor available (Dynflow::Error)" at db:seed step
Product: Red Hat Satellite Reporter: Sachin Ghai <sghai>
Component: Tasks PluginAssignee: Martin Bacovsky <mbacovsk>
Status: CLOSED ERRATA QA Contact: Roman Plevka <rplevka>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.2.7CC: bbuckingham, chrobert, egolov, hmore, inecas, mbacovsk, mmccune, oshtaier, pmoravec, rplevka
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-dynflow-0.8.13.5-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1446733 (view as bug list) Environment:
Last Closed: 2017-06-20 17:53:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1410795    
Attachments:
Description Flags
upgrade error dusing db:seed step
none
foreman-debug none

Description Sachin Ghai 2017-01-17 13:04:40 UTC
Description of problem:
Performed upgrade from sat6.2 -> 6.2.7 on rhel7 aw/ customer db. Upgrade was completed but I see errors on console at db_seed step :

-- 
Upgrade Step: db_seed...


[Sat_SERVER]  out: E, [2017-01-13T06:20:01.007010 #16653] ERROR -- /connector-database-core: No executor available (Dynflow::Error)


[Sat_SERVER]  out: E, [2017-01-13T06:20:01.007245 #16653] ERROR -- /client-dispatcher: No executor available (Dynflow::Error)
-- 



[Sat_SERVER] out: E, [2017-01-13T06:20:01.007010 #16653] ERROR -- /connector-database-core: No executor available (Dynflow::Error)
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/connectors/database.rb:142:in `any_executor'
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/connectors/database.rb:134:in `find_receiver'
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/connectors/database.rb:86:in `handle_envelope'
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/actor.rb:6:in `on_message'
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'



[Sat_SERVER] out: E, [2017-01-13T06:20:01.007245 #16653] ERROR -- /client-dispatcher: No executor available (Dynflow::Error)
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/connectors/database.rb:142:in `any_executor'
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/connectors/database.rb:134:in `find_receiver'
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/connectors/database.rb:86:in `handle_envelope'
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.13.3/lib/dynflow/actor.rb:6:in `on_message'
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in `on_envelope'
[Sat_SERVER] out: /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'

Version-Release number of selected component (if applicable):
upgrade sat6.2 to 6.2.7

How reproducible:
once w/ customer db

Steps to Reproduce:
1. used satellite-clone tool
2.
3.

Actual results:
Upgrade was completed however, got reported errors under "upgrade: db seed" step

Expected results:
no errors should be raised while performing upgrade

Additional info:

Comment 2 Sachin Ghai 2017-01-17 13:06:16 UTC
Created attachment 1241771 [details]
upgrade error dusing db:seed step

Comment 3 Sachin Ghai 2017-01-17 13:12:00 UTC
Created attachment 1241773 [details]
foreman-debug

Comment 4 Martin Bacovsky 2017-01-18 14:03:14 UTC
I was able to reproduce with the cust DB dump and rake db:migrate. I'll update once I have more details.

Comment 5 Martin Bacovsky 2017-01-23 16:35:05 UTC
The customer's DB contains Dynflow tasks that are in running state. We may assume the DB dump was done while foreman-tasks was running and the tasks were not finished properly.

During db:seed the Dynflow that is being initiated is trying to recover and is trying to restart the tasks. When no executor is running the error is risen. It is not harmful because the task will be resumed on next executor start, but the error should not be displayed.

Changing the componnent to the Tasks plugin.

Comment 6 Martin Bacovsky 2017-01-23 22:01:02 UTC
Created redmine issue http://projects.theforeman.org/issues/18206 from this bug

Comment 7 Martin Bacovsky 2017-01-24 08:22:40 UTC
Minimal reproducer for this issue:
1/ Have Foreman/Satellite with at least one Dynflow task running.    (Actions::Candlepin::ListenOnCandlepinEvents in Katello/Satellite is a good candidate)
2/ kill dynflow executor while the task is running: pkill -9 -f dynflow_executor
3/ foreman-rake db:migrate

You should see errors in the output

E, [2017-01-24T03:00:53.032499 #9544] ERROR -- /client-dispatcher: No executor available (Dynflow::Error)

Comment 10 Satellite Program 2017-01-24 17:20:05 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/18206 has been resolved.

Comment 11 Roman Plevka 2017-01-26 09:30:56 UTC
*** Bug 1413575 has been marked as a duplicate of this bug. ***

Comment 14 Roman Plevka 2017-06-01 11:48:34 UTC
VERIFIED
on 6.2.10-2

using the reproducer in Comment #7, I was able to reproduce the issue on 6.2.9 and can confirm, that it no longer occurs on 6.2.10-2

Comment 15 Bryan Kearney 2017-06-20 17:53:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1553

Comment 16 Red Hat Bugzilla 2023-09-14 03:52:21 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days