Bug 2047683

Summary:	Force cancel a paused task doesn't release the lock
Product:	Red Hat Satellite	Reporter:	Hao Chang Yu <hyu>
Component:	Tasks Plugin	Assignee:	Adam Ruzicka <aruzicka>
Status:	CLOSED ERRATA	QA Contact:	Peter Ondrejka <pondrejk>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	6.10.0	CC:	ahumbe, aruzicka, ehelms, peter.vreman, ramsingh
Target Milestone:	6.11.0	Keywords:	Triaged
Target Release:	Unused
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	tfm-rubygem-foreman-tasks-5.2.2	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2080335 (view as bug list)		Environment:
Last Closed:	2022-07-05 14:32:26 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Hao Chang Yu 2022-01-28 09:51:35 UTC

Description of problem:
Still getting the following error after "force cancel" the paused task that was holding the locks.

-----------------------------------------------
Conflicts with tasks: https://satellite.example.com/foreman_tasks/tasks/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX","errors":["Required lock is already taken by other running tasks.\nPlease inspect their state, fix their errors and resume them.
-----------------------------------------------

Steps to Reproduce:
1. Publish a content view and somehow make the task paused with error. Stop Pulp services in the middle of publish might help.
2. After that click "force cancel" the paused task in the Task page. Make sure the task status is set to stopped.
3. Publish the same content view again.

Actual results:
Fail with task conflict.

Expected results:
No conflict.

Additional info:

# Satellite 6.9
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/models/foreman_tasks/lock.rb
    ...
    scope :active, -> { joins(:task).where('foreman_tasks_tasks.state != ?', :stopped) }  <======= active scope got delete in 6.10
    ...
    ...
    # returns a scope of the locks colliding with this one
    def colliding_locks
      task_ids = task.self_and_parents.map(&:id)
      colliding_locks_scope = Lock.active.where(Lock.arel_table[:task_id].not_in(task_ids))  <============ has Lock.active scope
      colliding_locks_scope = colliding_locks_scope.where(name:          name,
                                                          resource_id:   resource_id,
                                                          resource_type: resource_type)
      unless exclusive?
        colliding_locks_scope = colliding_locks_scope.where(:exclusive => true)
      end
      colliding_locks_scope
    end


# Satellite 6.10
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/models/foreman_tasks/lock.rb
    # returns a scope of the locks colliding with this one
    def colliding_locks
      task_ids = task.self_and_parents.map(&:id)
      colliding_locks_scope = Lock.where(Lock.arel_table[:task_id].not_in(task_ids))  <========== Missing Lock.active scope
      colliding_locks_scope.where(resource_id:   resource_id,
                                  resource_type: resource_type)
    end

Comment 1 Adam Ruzicka 2022-01-28 14:06:59 UTC

Created redmine issue https://projects.theforeman.org/issues/34343 from this bug

Comment 2 Bryan Kearney 2022-01-28 16:05:27 UTC

Upstream bug assigned to aruzicka

Comment 3 Bryan Kearney 2022-01-28 16:05:28 UTC

Upstream bug assigned to aruzicka

Comment 4 Peter Vreman 2022-01-28 16:41:56 UTC

Thanks for the quick fixing. The patch works great for me.
It even seems to solve the 'original' problem that created the paused task that after the unlock still held locks.

When i try the problematic task to do a promotion of a composite contentviews i still the following once and then task get paused:
~~~
2022-01-28T16:24:57 [I|bac|ab8bbddb] Task {label: Actions::Katello::ContentView::Publish, id: 14b87c8f-48c0-46eb-a15c-aa65519c86cb, execution_plan_id: 32870ddc-97b4
-40a6-9e78-652cffa1da3f} state changed: running
2022-01-28T16:24:57 [E|bac|ab8bbddb] Error message: the server returns an error
 ab8bbddb | HTTP status code: 404
 ab8bbddb | Response headers: {"Date"=>"Fri, 28 Jan 2022 16:24:57 GMT", "Server"=>"gunicorn", "Content-Type"=>"application/json", "Vary"=>"Accept,Cookie", "Allow"=>
"GET, PUT, PATCH, DELETE, HEAD, OPTIONS", "X-Frame-Options"=>"SAMEORIGIN", "Content-Length"=>"23", "Correlation-ID"=>"ab8bbddb-7403-4a61-adef-fcf1c9ad390f", "Access
-Control-Expose-Headers"=>"Correlation-ID", "Via"=>"1.1 li-lc-2222.hag.hilti.com"}
 ab8bbddb | Response body: {"detail":"Not found."} (PulpRpmClient::ApiError)
 ab8bbddb | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
 ab8bbddb | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/repositories_rpm_api.rb:438:in `read_with_http_info'
 ab8bbddb | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/repositories_rpm_api.rb:385:in `read'
 ab8bbddb | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/lib/actions/pulp3/repository/save_version.rb:45:in `fetch_version_href'
 ab8bbddb | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/lib/actions/pulp3/repository/save_version.rb:19:in `run'
~~~
but now with this patch i can cleanly cancel the paused task with the above error.
And suprise... after that i can restart the promotion task and it finishes successful.

Comment 5 Bryan Kearney 2022-03-04 16:05:16 UTC

Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34343 has been resolved.

Comment 8 Peter Ondrejka 2022-04-07 11:58:02 UTC

Verified on Satellite 6.11 snap 15, after unlocking a failed publish task the cv can be republished successfully.

Comment 11 errata-xmlrpc 2022-07-05 14:32:26 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498