Bug 1375691 - Repository syncs still run even when pulp is down
Summary: Repository syncs still run even when pulp is down
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Pulp
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
medium
medium vote
Target Milestone: Unspecified
Assignee: John Mitsch
QA Contact: Katello QA List
URL: http://projects.theforeman.org/issues...
Whiteboard:
Keywords: Triaged
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-13 17:47 UTC by John Mitsch
Modified: 2019-04-01 20:27 UTC (History)
12 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2018-02-21 16:51:07 UTC


Attachments (Terms of Use)
screenshot-pending_task.png (29.73 KB, image/png)
2016-11-28 10:04 UTC, Peter Ondrejka
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Pulp Redmine 2491 High CLOSED - CURRENTRELEASE When stopping pulp_workers, pulp_celerybeat, and pulp_resource_manager gracefully, the status API still shows them as ru... 2017-01-16 21:01 UTC
Pulp Redmine 2496 Normal CLOSED - CURRENTRELEASE Killing pulp_workers, pulp_celerybeat, and pulp_resource_manager causes the status API still shows them as running 2017-04-27 14:04 UTC
Foreman Issue Tracker 16542 None None None 2016-09-13 19:04 UTC

Description John Mitsch 2016-09-13 17:47:18 UTC
Description of problem:

Repository syncs will run even if pulp services are stopped 

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1.  systemctl stop pulp_workers.service  pulp_celerybeat.service pulp_resource_manager.service 
2. Try to sync a repo
3. Repo will be stuck in "polling" stage

Actual results:

Repo will be stuck in "polling" stage

Expected results:

An error occurs in the task

Comment 1 John Mitsch 2016-09-13 18:34:12 UTC
Created redmine issue http://projects.theforeman.org/issues/16542 from this bug

Comment 2 Bryan Kearney 2016-09-13 20:18:11 UTC
Upstream bug assigned to jomitsch@redhat.com

Comment 4 Bryan Kearney 2016-09-14 22:20:04 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16542 has been resolved.

Comment 7 Peter Ondrejka 2016-11-28 10:03:26 UTC
Still reproducible in Satellite 6.3 snap 6:

satellite-6.3.0-6.1.beta.el7sat.noarch
katello-3.2.0-3.rc3.el7.noarch

With pulp_workers and pulp_celerybeat stopped, the sync task stays in pending state (see attached screenshot). No errors shown on the Errors tab.

/var/log/foreman/production.log keeps rolling:

2016-11-28 04:49:56 c6e719e4 [app] [I] Started POST "/foreman_tasks/api/tasks/bulk_search" for 2620:52:0:2282:2ad2:44ff:fef2:d8d2 at 2016-11-28 04:49:56 -0500
2016-11-28 04:49:56 c6e719e4 [app] [I] Processing by ForemanTasks::Api::TasksController#bulk_search as JSON
2016-11-28 04:49:56 c6e719e4 [app] [I]   Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"667bab5d-dee0-4584-88c2-6374c1341a09", "search_id"=>"1"}], "task"=>{}}
2016-11-28 04:49:56 c6e719e4 [app] [I] Completed 200 OK in 80ms (Views: 2.4ms | ActiveRecord: 1.7ms)

Comment 8 Peter Ondrejka 2016-11-28 10:04 UTC
Created attachment 1225218 [details]
screenshot-pending_task.png

Comment 9 Michael Hrivnak 2016-12-12 20:12:14 UTC
I'm changing the component since this is not an issue with pulp, but an issue with katello's use of pulp.

Comment 10 Bryan Kearney 2016-12-12 21:09:35 UTC
Upstream bug assigned to jomitsch@redhat.com

Comment 11 Bryan Kearney 2016-12-12 21:09:38 UTC
Upstream bug component is Pulp

Comment 12 John Mitsch 2016-12-14 21:56:26 UTC
I just discussed this with the pulp team, there is a bug in pulp that will show a faulty status report, which is the reason the check in katello isn't working. To summarize the issue:

pulp_celerybeat does the cleanup of worker entries in our database when they 
stop. If you stop pulp_celerybeat first, and then stop the workers, there's nobody around to clean up. So the records stay in the DB, and the status API might continue reporting that those workers exist. 

When an upstream bug is filed in pulp, we can associate it with this BZ

Comment 13 pulp-infra@redhat.com 2016-12-14 22:32:54 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 14 pulp-infra@redhat.com 2016-12-14 22:32:57 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 15 Bryan Kearney 2016-12-14 23:09:33 UTC
Upstream bug assigned to jomitsch@redhat.com

Comment 16 Bryan Kearney 2016-12-14 23:09:36 UTC
Upstream bug component is Pulp

Comment 17 pulp-infra@redhat.com 2016-12-16 17:30:54 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 18 pulp-infra@redhat.com 2016-12-16 22:33:49 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 19 pulp-infra@redhat.com 2016-12-16 23:01:01 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 20 pulp-infra@redhat.com 2016-12-16 23:01:04 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 21 pulp-infra@redhat.com 2016-12-17 18:02:59 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 22 pulp-infra@redhat.com 2016-12-20 20:01:42 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 23 pulp-infra@redhat.com 2017-01-06 16:31:38 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 24 pulp-infra@redhat.com 2017-01-06 18:02:00 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 25 pulp-infra@redhat.com 2017-01-10 02:01:37 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 26 pulp-infra@redhat.com 2017-01-10 17:31:53 UTC
The Pulp upstream bug status is at VERIFIED. Updating the external tracker on this bug.

Comment 27 pulp-infra@redhat.com 2017-01-16 21:01:53 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 28 pulp-infra@redhat.com 2017-01-17 19:01:31 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 29 pulp-infra@redhat.com 2017-01-17 21:01:53 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 30 pulp-infra@redhat.com 2017-01-19 21:01:34 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 31 pulp-infra@redhat.com 2017-01-20 15:01:54 UTC
The Pulp upstream bug status is at POST. Updating the external tracker on this bug.

Comment 32 pulp-infra@redhat.com 2017-01-24 22:01:23 UTC
The Pulp upstream bug status is at MODIFIED. Updating the external tracker on this bug.

Comment 33 Dennis Kliban 2017-02-15 22:57:36 UTC
providing info for a test

Comment 34 Dennis Kliban 2017-02-15 22:58:51 UTC
Requesting needsinfo from upstream developer dkliban@redhat.com because 'FailQA' flag is set.

Comment 35 Dennis Kliban 2017-02-15 22:59:00 UTC
Requesting needsinfo from upstream developer daviddavis@redhat.com because 'FailQA' flag is set.

Comment 36 David Davis 2017-02-16 20:54:27 UTC
Clearing NEEDSINFO since it failed qa before I worked on it.

Comment 37 pulp-infra@redhat.com 2017-02-16 21:02:12 UTC
Requesting needsinfo from upstream developer dkliban@redhat.com because the 'FailedQA' flag is set.

Comment 38 pulp-infra@redhat.com 2017-02-16 21:02:20 UTC
Requesting needsinfo from upstream developer daviddavis@redhat.com because the 'FailedQA' flag is set.

Comment 39 pm-sat@redhat.com 2017-02-16 23:09:58 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/16542 has been resolved.

Comment 40 pulp-infra@redhat.com 2017-04-17 13:33:50 UTC
The Pulp upstream bug status is at ON_QA. Updating the external tracker on this bug.

Comment 41 pulp-infra@redhat.com 2017-04-17 18:32:43 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 42 pulp-infra@redhat.com 2017-04-17 18:32:47 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 43 pulp-infra@redhat.com 2017-04-17 18:32:52 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 44 pulp-infra@redhat.com 2017-04-17 18:32:55 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 45 Daniel Alley 2017-04-17 20:37:47 UTC
Removing the trackers for backport tasks - not the intended workflow and breaks some automation

Comment 46 pulp-infra@redhat.com 2017-04-27 14:04:40 UTC
The Pulp upstream bug status is at CLOSED - CURRENTRELEASE. Updating the external tracker on this bug.

Comment 48 Zach Huntington-Meath 2017-08-16 17:19:35 UTC
Confirmed that the sync properly errors when trying to sync it while pulp is turned off in snap-11.0

My steps were:

1. systemctl stop pulp_workers.service  pulp_celerybeat.service pulp_resource_manager.service 

2. I tried to sync a repo in the UI and received an error warning.

Comment 49 pm-sat@redhat.com 2018-02-21 16:51:07 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> 
> For information on the advisory, and where to find the updated files, follow the link below.
> 
> If the solution does not work for you, open a new bug report.
> 
> https://access.redhat.com/errata/RHSA-2018:0336


Note You need to log in before you can comment on or make changes to this bug.