Created attachment 1779963 [details] prodution log errors Description of problem: Remove Orphans Task encounters pulp3 Service down error Version-Release number of selected component (if applicable): 6.9.1 (upgraded from 6.8.3, had to re-run installer multiple times due to errors caused by /tmp/bundler) How reproducible: consistently Steps to Reproduce: 1. Run: foreman-rake katello:delete_orphaned_content RAILS_ENV=production 2. Find Orphan Remove task in the Task list in Satellite Actual results: Error: There was an issue with the backend service pulp3: 503 Service Unavailable There was an issue with the backend service pulp3: 503 Service Unavailable There was an issue with the backend service pulp3: 503 Service Unavailable Expected results: Job runs Additional info: Attaching production.log errors
pulp3 references in the production log entries attached. $ grep pulp3 production_log_RemoveOrphans_error.txt | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/models/katello/ping.rb:185:in `pulp3_without_auth' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/models/katello/ping.rb:75:in `block in ping_pulp3_without_auth' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/models/katello/ping.rb:74:in `ping_pulp3_without_auth' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/lib/actions/pulp3/orchestration/orphan_cleanup/remove_orphans.rb:9:in `block in plan' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/lib/actions/pulp3/orchestration/orphan_cleanup/remove_orphans.rb:8:in `plan' 2021-05-05T14:39:24 [E|bac|] There was an issue with the backend service pulp3: 503 Service Unavailable (RuntimeError) | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/lib/actions/pulp3/orchestration/orphan_cleanup/remove_orphans.rb:9:in `block in plan' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/lib/actions/pulp3/orchestration/orphan_cleanup/remove_orphans.rb:8:in `plan' 2021-05-05T14:39:24 [E|bac|] There was an issue with the backend service pulp3: 503 Service Unavailable (RuntimeError) | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/lib/actions/pulp3/orchestration/orphan_cleanup/remove_orphans.rb:9:in `block in plan' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/lib/actions/pulp3/orchestration/orphan_cleanup/remove_orphans.rb:8:in `plan' 2021-05-05T14:39:24 [E|bac|] There was an issue with the backend service pulp3: 503 Service Unavailable (RuntimeError) | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/lib/actions/pulp3/orchestration/orphan_cleanup/remove_orphans.rb:9:in `block in plan' | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/lib/actions/pulp3/orchestration/orphan_cleanup/remove_orphans.rb:8:in `plan' 2021-05-05T14:39:25 [E|app|] Smart proxy with ID 1 may be down: There was an issue with the backend service pulp3: 503 Service Unavailable katello ping is attempting to ping the pulp3 api. Notice: $ systemctl list-unit-files ... pulp_celerybeat.service enabled pulp_resource_manager.service enabled pulp_streamer.service enabled pulp_worker-0.service static pulp_worker-1.service static pulp_worker-2.service static pulp_worker-3.service static pulp_workers.service enabled pulpcore-api.service disabled pulpcore-content.service disabled pulpcore-resource-manager.service disabled pulpcore-worker@.service disabled ... I believe the intent is that pulp3 is at least partly installed in preparation for migration activities, but no code in katello should be referencing it as yet. Unfortunately, it seems some pulp3 code slipped into katello. I have not noticed anything else affected so far, other than this code.
You can work around the issue by running pulp-admin orphan remove --all --bg on the satellite and each capsule, instead of using the foreman-rake command in cron.d/katello on the satellite.
Not certain the appropriate component for this rake task; however, Repositories may be a better fit.
Solution for this one may be same as for bug 1959640.
Closing this as a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1959640. Please reopen this bug if the fix for that issue doesn't resolve the issue here. *** This bug has been marked as a duplicate of bug 1959640 ***