Bug 1468078

Summary: sync task does not ever complete if satellite is interrupted during run
Product: Red Hat Satellite 6 Reporter: Chris Duryee <cduryee>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED ERRATA QA Contact: Jitendra Yejare <jyejare>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.10CC: andrew.schofield, bbuckingham, bkearney, cduryee, dalley, daviddavis, dgross, dkliban, fgarciad, jcallaha, jsherril, jyejare, kabbott, lzap, mhrivnak, mmccune, pcreech, rchan, sreber, sthirugn, ttereshc
Target Milestone: GAKeywords: FieldEngineering, PrioBumpField, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1548167 (view as bug list) Environment:
Last Closed: 2018-02-21 16:54:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 1482208, 1548167    

Description Chris Duryee 2017-07-06 00:56:18 UTC
Description of problem:

If you are running a sat 6.2.10 and have a sync running, it may get confused if you restart the Satellite during the run. You'll end up with some sync tasks that do not ever complete, requiring manual intervention.

to repro:

* kick off a sync on a repo with lots of consumers
* wait for Actions::Pulp::Repository::RegenerateApplicability task to get going, you can watch for this in dynflow console
* run "pkill celery" (not sure if this is needed, it's just how I did it)
* stop and then start services

What will happen is the applicability regen finishes in pulp, but the task group  will have some stragglers that never get updated in the mongo DB. This causes Katello to wait forever for the task group to have zero tasks in waiting/running state.

Version-Release number of selected component (if applicable): 6.2.10

Here is how I repro'd with a plain pulp 2.8.7:

* POST to https://localhost/pulp/api/v2/repositories/actions/content/regenerate_applicability/ for a big repo with lots of bound consumers.
* you'll get a task group ID back, ensure that if you curl the task group ID's state summary, you get a number that's decreasing
* kill celery (just regular, not -9) and stop and start all pulp services
* watch state summary go down again, it will not quite get to zero, but no tasks will be running

Comment 1 Chris Duryee 2017-07-06 00:56:54 UTC
workaround: get the task group ID from the dynflow task output, and delete the task group via pulp API curl command

Comment 3 pulp-infra@redhat.com 2017-07-06 01:32:09 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 4 pulp-infra@redhat.com 2017-07-06 01:32:12 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 5 pulp-infra@redhat.com 2017-07-07 15:02:08 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2017-08-01 18:05:01 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 9 pulp-infra@redhat.com 2017-08-21 17:03:53 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 11 pulp-infra@redhat.com 2017-08-22 13:01:48 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 13 pulp-infra@redhat.com 2017-08-23 02:31:35 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 14 pulp-infra@redhat.com 2017-08-23 14:31:41 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 15 pulp-infra@redhat.com 2017-08-23 15:01:34 UTC
All upstream Pulp bugs are at MODIFIED+. Moving this bug to POST.

Comment 18 Justin Sherrill 2017-08-29 18:46:39 UTC
*** Bug 1376821 has been marked as a duplicate of this bug. ***

Comment 19 pulp-infra@redhat.com 2017-10-05 22:02:36 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 21 Jitendra Yejare 2017-12-20 05:37:33 UTC
(Thanks Chris for the help in verifying this bug)

Verified !

@ Satellite 6.3 snap 28

Steps:
1. Create lot of content hosts
2. Create(or Use existing) and attempt to sync a big repository
3. Wait for the ReGenerateApplicability task to appear in Dynflow console for repo sync.
4. Wait for some task to run under task group under RegenerateApplicability
5. As soon as some tasks in running state. kill the celery on sat server, and observe the task is in a suspended state.
# pkill celery
6. Now restart the pulp_workers service.
7. Go back, RegenerateApplicability and check the suspended task should be running


Observed:
1. The suspended RegenerateApplicability task in running state
2. The count of pending task went down to 0
3. The sync got completed successfully

Hence changing the state to Verified !

Comment 24 pm-sat@redhat.com 2018-02-21 16:54:17 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336