Bug 1416177

Created attachment 1244021 [details]
stacktrace

There are two known bugs related:

https://bugzilla.redhat.com/show_bug.cgi?id=1339231 - the join issue: this is just a symptom: not the root cause of the issue.

The issue with the future execution not starting was already reported in 
https://bugzilla.redhat.com/show_bug.cgi?id=1384214 and was closed in favor of 
https://bugzilla.redhat.com/show_bug.cgi?id=1390931, which was the root cause of the execution not running properly.

Could you provide us foreman-debug archive + answers from question in https://bugzilla.redhat.com/show_bug.cgi?id=1384214#c3, to make sure the root
cause is the same, which would mean we could close this bug as duplicite and make
sure the original fix is delivered sooner rather than later

I ran the cli command to checkout the worlds, and found the delayed_executor to be invalid.  I've restarted foreman-tasks multiple times, without it ever becoming valid again.

{"hostname"=>"myhostname.fqdn.com", "pid"=>6942, "delayed_executor"=>true, "valid"=>:invalid}

I have uploaded a foreman-debug to RH case# 01779823

Here's the full console output from the rake command

[root scripts]# cat <<-END | foreman-rake console
> states = ForemanTasks.dynflow.world.worlds_validity_check(false)
> ForemanTasks.dynflow.world.coordinator.find_worlds.map do |world|
>   world.meta.merge(:valid => states.fetch(world.id, :invalid))
> end
> END
/usr/share/foreman/lib/tasks/console.rake:4: warning: already initialized constant ARGV
Loading production environment (Rails 4.1.5)
Switch to inspect mode.
states = ForemanTasks.dynflow.world.worlds_validity_check(false)
{"f852629f-8919-489a-ad10-49ad1e52fa83"=>:valid, "38616092-3b64-4373-b6d9-823dc3fc38d7"=>:invalid, "38908017-bd16-4675-8bf2-54029540b9b5"=>:valid, "c605cdb5-fd45-4be8-9ab7-caba1a59bc06"=>:valid, "5e27ca82-677a-4569-9359-3b3264f04b91"=>:invalid, "462bd57d-4d54-42e6-b799-0024b4a648fd"=>:valid, "e081cf2d-00b5-4e2f-a8ad-dbf83ac645cd"=>:valid, "155a7ae5-1331-4da6-83ed-23b1990c88fc"=>:valid, "e9b8148c-798c-4326-bd04-ad8a2404251b"=>:invalid, "46dbc8b4-5b80-42f6-b996-c736a0c37a77"=>:invalid, "4281cf25-91ec-402a-89e1-a87c28f0e64c"=>:invalid, "d322208f-dba3-4009-a2c4-7ce565ee720d"=>:valid, "5188be1b-9d6d-4ae3-a95d-8ce36a474c41"=>:valid, "3615a141-9484-4641-a009-c51d65e83e24"=>:valid, "9d82bb58-0e26-4e68-a953-da5190eeb6bd"=>:invalid, "3f043e42-0a4b-45e1-9892-912b41a15666"=>:invalid, "d88ced49-5345-4a56-8daf-2060e8421a22"=>:invalid, "e1e560d2-6294-4296-8256-8395d3d73ca5"=>:valid, "af60b94c-d0d6-42a3-8bfd-45232a1f7e48"=>:invalid, "cc0f8421-203a-499e-9002-f9fadcf9e845"=>:invalid, "3fbe457e-49a7-473b-8d6e-9ba5beb38fb5"=>:invalid, "73fab36c-500f-4094-9110-b8252ba751c2"=>:valid, "1c468be6-8d59-4e3d-ac3d-bcaf660e68b5"=>:valid, "555cda66-a4d5-4b63-bd4d-8fa1ae7c9749"=>:valid, "7ad1620b-0fd6-463b-ade2-dbe57e731700"=>:valid}
ForemanTasks.dynflow.world.coordinator.find_worlds.map do |world|
  world.meta.merge(:valid => states.fetch(world.id, :invalid))
end
[{"hostname"=>"mysatl.fqdn.com", "pid"=>57730, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>6942, "delayed_executor"=>true, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>32886, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>33005, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>55741, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>33151, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>33300, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>59597, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>50290, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>50441, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>50657, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>34325, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>33055, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>33207, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>50323, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>50519, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>50741, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>33358, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>16641, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>50389, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>50577, "valid"=>:invalid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>32851, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>32961, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>33099, "valid"=>:valid}, {"hostname"=>"mysatl.fqdn.com", "pid"=>33262, "valid"=>:valid}]

Doing the patch mentioned in 1390931 fixed my execution issue.  

I also had to apply the change that fixed 1339231 to get the job task viewable before running.

Thanks Jason: I'm marking this a duplicate of 1390931 then.

*** This bug has been marked as a duplicate of bug 1390931 ***