Bug 1959640 - katello:delete_orphaned_content fails on Satellite with "backend service pulp3: 503 Service Unavailable"
Summary: katello:delete_orphaned_content fails on Satellite with "backend service pulp...
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Repositories
Version: 6.9.0
Hardware: Unspecified
OS: Unspecified
medium vote
Target Milestone: 6.9.3
Assignee: Ian Ballou
QA Contact: Tasos Papaioannou
: 1957464 (view as bug list)
Depends On:
TreeView+ depends on / blocked
Reported: 2021-05-12 01:21 UTC by Dylan Gross
Modified: 2021-08-09 10:20 UTC (History)
10 users (show)

Fixed In Version: tfm-rubygem-katello-
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1962852 (view as bug list)
Last Closed: 2021-07-01 14:56:52 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 32623 0 Normal Closed katello:delete_orphaned_content fails on Katello 3.18 with "backend service pulp3: 503 Service Unavailable" 2021-05-20 15:36:31 UTC
Red Hat Knowledge Base (Solution) 6034731 0 None None None 2021-05-12 01:21:08 UTC
Red Hat Product Errata RHBA-2021:2636 0 None None None 2021-07-01 14:57:22 UTC

Description Dylan Gross 2021-05-12 01:21:09 UTC
Description of problem:

   After upgrading to v6.9.1, the weekly cron "foreman-rake katello:publish_unpublished_repositories RAILS_ENV=production" from /etc/cron.d/katello fails on the Satellite.  (And thus never executes against the rest of the capsules either)

Version-Release number of selected component (if applicable):

   Red Hat Satellite 6.9.1

How reproducible:   Unknown if this is widespread or isolated to a single satellite.

Steps to Reproduce:
1.   Either wait for the weekly execution or invoke the cleanup manually:
  #  foreman-rake katello:publish_unpublished_repositories RAILS_ENV=production 

2.   Check logs for backtrace (attachment

Actual results:

   Logs report that the pulp3 service on Smart proxy Id 1 may be down

   "[E|app|] Smart proxy with ID 1 may be down: There was an issue with the backend service pulp3: 503 Service Unavailable"

   The cleanup never proceeds to clean-up other smart proxies, however, if you specify a single non-satellite capsule with SMART_PROXY_ID= , it will succeed on that capsule.

Expected results:

    Clean-up job succeeds, and proceeds to clean up other capsules.

Additional info:

    To my knowledge, pulp3 should not be expected to be running in 6.9 yet.

Comment 2 Brad Buckingham 2021-05-12 20:51:39 UTC
Not certain the appropriate component for this rake task; however, Repositories may be a better fit.

Comment 3 Brad Buckingham 2021-05-12 20:52:17 UTC
Solution for this one may be same as for bug 1957464.

Comment 4 Ian Ballou 2021-05-18 21:08:50 UTC
Created redmine issue https://projects.theforeman.org/issues/32623 from this bug

Comment 5 Ian Ballou 2021-05-19 18:12:27 UTC
@dgross it isn't clear which rake task this BZ is targeting. In the description you mentioned `katello:publish_unpublished_repositories` but the "Smart proxy with ID 1 may be down" error matches the production log for `katello:delete_orphaned_content`.  I see the cause for the `katello:delete_orphaned_content` error so I'll start by fixing that, but if there's also an issue with `katello:publish_unpublished_repositories` we'll need more info about it.

Comment 6 Dylan Gross 2021-05-19 18:28:28 UTC
Apologies.  Title is incorrect.  This is 100% "katello:delete_orphaned_content"

Comment 7 Ian Ballou 2021-05-19 19:10:31 UTC
Thanks Dylan. In that case, the PR associated to the redmine attached to this BZ should directly fix the problem.

Comment 8 Ian Ballou 2021-05-20 15:38:20 UTC
*** Bug 1957464 has been marked as a duplicate of this bug. ***

Comment 9 Bryan Kearney 2021-05-20 20:00:53 UTC
Upstream bug assigned to iballou

Comment 10 Bryan Kearney 2021-05-20 20:00:55 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/32623 has been resolved.

Comment 11 Dylan Gross 2021-05-27 16:41:05 UTC
Just a follow-up:  The upstream fix from https://projects.theforeman.org/issues/32623 does indeed allow the task to be successfully invoked and run to completion in the Satellite (and capsules) where I first encountered the issue.

Comment 13 Tasos Papaioannou 2021-06-07 15:21:18 UTC
Verified on 6.9.3 snap 1.0.

# systemctl status pulpcore-api
● pulpcore-api.service - Pulp API Server
   Loaded: loaded (/etc/systemd/system/pulpcore-api.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

# foreman-rake katello:delete_orphaned_content
Orphaned content deletion started in background.


2021-06-07T10:51:31 [I|bac|] Task {label: Actions::Katello::OrphanCleanup::RemoveOrphans, id: 62b7c2cc-1f7c-419e-83a3-a695ed819337, execution_plan_id: 7d212dfa-d991-458c-92ae-d50f69de9f69} state changed: stopped  result: success

Comment 18 errata-xmlrpc 2021-07-01 14:56:52 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Satellite 6.9.3 Async Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.