Bug 1375691

Summary: Repository syncs still run even when pulp is down
Product: Red Hat Satellite Reporter: John Mitsch <jomitsch>
Component: PulpAssignee: John Mitsch <jomitsch>
Status: CLOSED ERRATA QA Contact: Katello QA List <katello-qa-list>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2.0CC: bbuckingham, bkearney, bmbouter, dalley, daviddavis, dkliban, ggainey, ipanova, jcallaha, mhrivnak, mmccune, pcreech, rchan, ttereshc, zhunting
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/16542
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 16:51:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screenshot-pending_task.png none

Description John Mitsch 2016-09-13 17:47:18 UTC
Description of problem:

Repository syncs will run even if pulp services are stopped 

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1.  systemctl stop pulp_workers.service  pulp_celerybeat.service pulp_resource_manager.service 
2. Try to sync a repo
3. Repo will be stuck in "polling" stage

Actual results:

Repo will be stuck in "polling" stage

Expected results:

An error occurs in the task

Comment 1 John Mitsch 2016-09-13 18:34:12 UTC
Created redmine issue http://projects.theforeman.org/issues/16542 from this bug

Comment 2 Bryan Kearney 2016-09-13 20:18:11 UTC
Upstream bug assigned to jomitsch

Comment 4 Bryan Kearney 2016-09-14 22:20:04 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16542 has been resolved.

Comment 7 Peter Ondrejka 2016-11-28 10:03:26 UTC
Still reproducible in Satellite 6.3 snap 6:

satellite-6.3.0-6.1.beta.el7sat.noarch
katello-3.2.0-3.rc3.el7.noarch

With pulp_workers and pulp_celerybeat stopped, the sync task stays in pending state (see attached screenshot). No errors shown on the Errors tab.

/var/log/foreman/production.log keeps rolling:

2016-11-28 04:49:56 c6e719e4 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 2620:52:0:2282:2ad2:44ff:fef2:d8d2 at 2016-11-28 04:49:56 -0500
2016-11-28 04:49:56 c6e719e4 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
2016-11-28 04:49:56 c6e719e4 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"667bab5d-dee0-4584-88c2-6374c1341a09", "search_id"=>"1"}], "task"=>{}}
2016-11-28 04:49:56 c6e719e4 [app] [I] Completed 200 OK in 80ms (Views: 2.4ms | ActiveRecord: 1.7ms)

Comment 8 Peter Ondrejka 2016-11-28 10:04:28 UTC
Created attachment 1225218 [details]
screenshot-pending_task.png

Comment 9 Michael Hrivnak 2016-12-12 20:12:14 UTC
I'm changing the component since this is not an issue with pulp, but an issue with katello's use of pulp.

Comment 10 Bryan Kearney 2016-12-12 21:09:35 UTC
Upstream bug assigned to jomitsch

Comment 11 Bryan Kearney 2016-12-12 21:09:38 UTC
Upstream bug component is Pulp

Comment 12 John Mitsch 2016-12-14 21:56:26 UTC
I just discussed this with the pulp team, there is a bug in pulp that will show a faulty status report, which is the reason the check in katello isn't working. To summarize the issue:

pulp_celerybeat does the cleanup of worker entries in our database when they 
stop. If you stop pulp_celerybeat first, and then stop the workers, there's nobody around to clean up. So the records stay in the DB, and the status API might continue reporting that those workers exist. 

When an upstream bug is filed in pulp, we can associate it with this BZ

Comment 13 pulp-infra@redhat.com 2016-12-14 22:32:54 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 14 pulp-infra@redhat.com 2016-12-14 22:32:57 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 15 Bryan Kearney 2016-12-14 23:09:33 UTC
Upstream bug assigned to jomitsch

Comment 16 Bryan Kearney 2016-12-14 23:09:36 UTC
Upstream bug component is Pulp

Comment 17 pulp-infra@redhat.com 2016-12-16 17:30:54 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 18 pulp-infra@redhat.com 2016-12-16 22:33:49 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 19 pulp-infra@redhat.com 2016-12-16 23:01:01 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 20 pulp-infra@redhat.com 2016-12-16 23:01:04 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 21 pulp-infra@redhat.com 2016-12-17 18:02:59 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 22 pulp-infra@redhat.com 2016-12-20 20:01:42 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 23 pulp-infra@redhat.com 2017-01-06 16:31:38 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 24 pulp-infra@redhat.com 2017-01-06 18:02:00 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 25 pulp-infra@redhat.com 2017-01-10 02:01:37 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 26 pulp-infra@redhat.com 2017-01-10 17:31:53 UTC
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.

Comment 27 pulp-infra@redhat.com 2017-01-16 21:01:53 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 28 pulp-infra@redhat.com 2017-01-17 19:01:31 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 29 pulp-infra@redhat.com 2017-01-17 21:01:53 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 30 pulp-infra@redhat.com 2017-01-19 21:01:34 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 31 pulp-infra@redhat.com 2017-01-20 15:01:54 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 32 pulp-infra@redhat.com 2017-01-24 22:01:23 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 33 Dennis Kliban 2017-02-15 22:57:36 UTC
providing info for a test

Comment 34 Dennis Kliban 2017-02-15 22:58:51 UTC
Requesting needsinfo from upstream developer dkliban because 'FailQA' flag is set.

Comment 35 Dennis Kliban 2017-02-15 22:59:00 UTC
Requesting needsinfo from upstream developer daviddavis because 'FailQA' flag is set.

Comment 36 David Davis 2017-02-16 20:54:27 UTC
Clearing NEEDSINFO since it failed qa before I worked on it.

Comment 37 pulp-infra@redhat.com 2017-02-16 21:02:12 UTC
Requesting needsinfo from upstream developer dkliban because the 'FailedQA' flag is set.

Comment 38 pulp-infra@redhat.com 2017-02-16 21:02:20 UTC
Requesting needsinfo from upstream developer daviddavis because the 'FailedQA' flag is set.

Comment 39 Satellite Program 2017-02-16 23:09:58 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16542 has been resolved.

Comment 40 pulp-infra@redhat.com 2017-04-17 13:33:50 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 41 pulp-infra@redhat.com 2017-04-17 18:32:43 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 42 pulp-infra@redhat.com 2017-04-17 18:32:47 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 43 pulp-infra@redhat.com 2017-04-17 18:32:52 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 44 pulp-infra@redhat.com 2017-04-17 18:32:55 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 45 Daniel Alley 2017-04-17 20:37:47 UTC
Removing the trackers for backport tasks - not the intended workflow and breaks some automation

Comment 46 pulp-infra@redhat.com 2017-04-27 14:04:40 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 48 Zach Huntington-Meath 2017-08-16 17:19:35 UTC
Confirmed that the sync properly errors when trying to sync it while pulp is turned off in snap-11.0

My steps were:

1. systemctl stop pulp_workers.service  pulp_celerybeat.service pulp_resource_manager.service 

2. I tried to sync a repo in the UI and received an error warning.

Comment 49 Satellite Program 2018-02-21 16:51:07 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336