Bug 1693440

Summary: Pulp's task cancellation does not work
Product: Red Hat Satellite Reporter: Dennis Kliban <dkliban>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: jcallaha
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.3.5CC: bmbouter, daviddavis, dkliban, ggainey, ipanova, mmccune, rchan, ttereshc, zhunting
Target Milestone: ReleasedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-14 19:57:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
verification video none

Description Dennis Kliban 2019-03-27 20:19:17 UTC
Description of problem:

Sync / publish tasks cancelled in the UI are not actually getting cancelled when the user performs a cancel. The worker proccess continues to execute the task.  Pulp's REST API says the cancellation is successful and the task status is marked as CANCELLED. However, the details of the task continue to get updated as a sync or publish continues to run. Users can experience delays in other tasks starting because the workers are busy executing cancelled tasks.

Version-Release number of selected component (if applicable):


Any release of Satellite 6 that uses python2-celery 4.0.2.


How reproducible:

Every time

Steps to Reproduce:

1) Start a sync of a single repository.
2) Look at output of 'top' and make note of CPU usage of a single celery process that is performing the sync. 
3) Go to the task details and click the cancel button. 
4) Look at the output of 'top' and observe that the same celery process is still using just as much CPU. 

Actual results:

The celery process continues executing the task.

Expected results:

Celery process stops executing the task. 

Additional info:

The actual problem is in the kombu package. There is already a patch for it upstream. https://github.com/celery/kombu/pull/1029/

Comment 3 pulp-infra@redhat.com 2019-03-28 08:31:35 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 4 pulp-infra@redhat.com 2019-03-28 08:31:35 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 6 Dennis Kliban 2019-03-28 17:09:05 UTC
Pulp is able to work with both Celery 3 and 4. Downgrading python2-celery is a good way to work around this problem.

Comment 7 pulp-infra@redhat.com 2019-04-02 13:04:10 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2019-04-02 13:31:22 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 12 jcallaha 2019-04-09 20:44:53 UTC
Verified in Satellite 6.5.0 Snap 23

Followed the reproducer steps. 

Upon canceling the sync, you can see the celery worker being killed and replaced.

See attached video for verification.

Comment 13 jcallaha 2019-04-09 20:45:11 UTC
Created attachment 1553940 [details]
verification video

Comment 14 Bryan Kearney 2019-05-14 19:57:24 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1222

Comment 15 pulp-infra@redhat.com 2019-05-22 19:31:47 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 16 pulp-infra@redhat.com 2019-05-30 17:02:08 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.