Description of problem: While verifying if the issue reported on "https://bugzilla.redhat.com/show_bug.cgi?id=1727493" is whether or not reproducible on Satellite 6.8 beta, It's being observed that whereas the issue is not reproducible but it creates some tasks at backend which stays in planned state even if the result is success for those tasks. Version-Release number of selected component (if applicable): satellite-6.8.0-0.7.beta.el7sat.noarch How reproducible: Always Steps to Reproduce: 1. Content --> Subscriptions --> Manage Manifest 2. Change the CDN Url to point to "http://satellite.example.com/pub/sat-import/" 3. Click on Update. 4. Wait for few mins and Go back to Monitor --> Tasks 5. Search the tasks using this filter --> state != stopped && result = success Actual results: Number of tasks by the name "Updating repository authentication configuration" , label "Actions::Katello::Repository::UpdateCVRepoCertGuard " , stays in planned | stopped state. # hammer task list --search "state != stopped && result = success" -------------------------------------|---------------------------------------------------|---------|---------|---------------------|----------|-------|------------ ID | ACTION | STATE | RESULT | STARTED AT | ENDED AT | OWNER | TASK ERRORS -------------------------------------|---------------------------------------------------|---------|---------|---------------------|----------|-------|------------ d70da815-dc7b-4ed4-a78c-5d5194932992 | Updating repository authentication configuration | planned | success | 2020/08/28 08:15:17 | | admin | 8abfe90a-116c-4286-af27-78e2c548ff23 | Updating repository authentication configuration | planned | success | 2020/08/28 08:15:17 | | admin | 0b009f5a-16fa-46ca-b50a-234484dcf590 | Updating repository authentication configuration | planned | success | 2020/08/28 08:15:17 | | admin | 0c7585f9-208b-405e-9e08-ebddcda40a28 | Updating repository authentication configuration | planned | success | 2020/08/28 08:15:16 | | admin | -------------------------------------|---------------------------------------------------|---------|---------|---------------------|----------|-------|------------ # hammer task info --id d70da815-dc7b-4ed4-a78c-5d5194932992 ID: d70da815-dc7b-4ed4-a78c-5d5194932992 Action: Updating repository authentication configuration State: planned Result: success Started at: 2020/08/28 08:15:17 Ended at: Owner: admin Task errors: Expected results: If a task is successfully completed, It should not stay in planned state or else attempt of upgrade to next available minor\major release will fail. Additional info: NA
This is a funny one 1) Finalize phase of an action starts 2) The action kicks off a couple of Actions::Katello::Repository::UpdateCVRepoCertGuard tasks as async tasks from its finalize phase 3) The execution plan gets created *outside* of the transaction by Sequel 4) There is a hook which updates tasks when state of execution plans in dynflow changes, this hook gets triggered, but since it is still running in the same thread it is *inside* the transaction controlled by ActiveRecord 5) The hook moves the task to planned-success 6) Execution of the child task is handed off to the executor, possibly to another thread/process, definitely outside the still open ActiveRecord transaction 7) The execution of the child task finishes, but since it is outside the ActiveRecord transaction, from its point of view there is no task because the transaction wasn't commited yet 8) Finalize phase of the original action ends, changes get commited to the db, child task stays in planned-success Possible solutions: 1) Do not to kick off tasks from finalize phase, maybe make those tasks part of the main execution plan? 1a) Make them part of main execution plan 1b) Kick them off at the end of run 1c) Kick them off from execution plan hooks 1d) Kick them off from finalize, but set them to execute in the future to make sure the original transaction is commited by the time they start to run 2) Run the task update hooks in their own thread with their own active record connection. This is probably the correct way to go about this, but given the recent issues with db connection pools it also feels like a potential footgun CC oezr & jsherrill : Opinions on the above? Do you have any other idea how we could go about this? Holding off with setting devel_triaged until we decide which way we want to fix this, since it may involve a component change.
Couple of comments: 1) This really shouldn't be triggered as part of cdn url update, it was an unintended side effect. I think a small code bug here is causing it: https://github.com/Katello/katello/blob/master/app/lib/actions/katello/repository/update.rb#L11 in this case, its intended to check for a change, but is not handling the case when nil is being passed in (and thus no change needed) 2) I think it could either be kicked off in the run phase, or planned as part of the overall action. One reason to do it in the run phase is to reduce the amount of async work done as part of a repo update (as katello's endpoint is syncronous, even though an async task is done in the background..... not good, but thats another bug for another day). So for now my vote would be run phase
Created redmine issue https://projects.theforeman.org/issues/30736 from this bug
Upstream bug assigned to jsherril
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/30736 has been resolved.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.8 release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:4366