Bug 1993288

Summary:	Pulp 3 check fails if only running a single worker
Product:	Red Hat Satellite	Reporter:	Sayan Das <saydas>
Component:	Capsule - Content	Assignee:	Justin Sherrill <jsherril>
Status:	CLOSED ERRATA	QA Contact:	Vladimír Sedmík <vsedmik>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	6.10.0	CC:	aruzicka, inecas, jsherril, zhunting
Target Milestone:	6.10.0	Keywords:	Triaged
Target Release:	Unused
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	tfm-rubygem-katello-4.1.1.17-1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-11-16 14:13:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sayan Das 2021-08-12 17:14:52 UTC

Description of problem:

Sync or CreatePulpDiskSpaceNotifications fails for capsule 6.10 with error Not all necessary pulp workers running at https://capsule.example.com/pulp/api/v3


Version-Release number of selected component (if applicable):

Satellite and Satellite Capsule 6.10 [ satellite-6.10.0-0.5.beta.el7sat.noarch / satellite-capsule-6.10.0-0.5.beta.el7sat.noarch ]


How reproducible:

Always


Steps to Reproduce:
1. Install Satellite (satellite-6.10.0-0.5.beta) and Capsule (satellite-capsule-6.10.0-0.5.beta) server 
2. Sync some repos in satellite
3. Add Library to Capsule as an LCE to sync
4. Try to initial Optimized Sync

or, simply look at "Pulp storage" section for the capsule from Infrastructure --> Capsule --> capsule.example.com page


Actual results:

For CreatePulpDiskSpaceNotifications task

2021-08-12T18:07:10 [E|app|] Error performing CreatePulpDiskSpaceNotifications (Job ID: a48b9b04-71bc-4beb-a62d-b8bc9b6dca6b) from Dynflow(default) in 319.58ms: RuntimeError (Not all necessary pulp workers running at https://capsule.example.com/pulp/api/v3.):
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/models/katello/ping.rb:200:in `pulp3_without_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/models/katello/concerns/smart_proxy_extensions.rb:370:in `ping_pulp3'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/models/katello/concerns/smart_proxy_extensions.rb:169:in `pulp_disk_usage'


For Sync task

2021-08-12T22:29:27 [I|app|3b366271] Started POST "/katello/api/capsules/2-capsule-example-com/content/sync" for 10.67.xx.xx at 2021-08-12 22:29:27 +0530
2021-08-12T22:29:27 [I|app|3b366271] Processing by Katello::Api::V2::CapsuleContentController#sync as HTML
2021-08-12T22:29:27 [I|app|3b366271]   Parameters: {"id"=>"2-capsule-example-com", "skip_metadata_check"=>false, "api_version"=>"v2", "capsule_content"=>{"id"=>"2-capsule-example-com", "skip_metadata_check"=>false}}
2021-08-12T22:29:27 [I|bac|3b366271] Task {label: Actions::Katello::CapsuleContent::Sync, id: e3f4a17a-3298-42b6-8d4b-ce07c37d0ea2, execution_plan_id: 9ed96668-cb4a-4bc3-9fc0-bddad7377094} state changed: planning 
2021-08-12T22:29:29 [W|app|3b366271] Not all necessary pulp workers running at https://capsule.example.com/pulp/api/v3.
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/models/katello/ping.rb:200:in `pulp3_without_auth'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/models/katello/ping.rb:76:in `block in ping_pulp3_without_auth'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/models/katello/ping.rb:138:in `exception_watch'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/models/katello/ping.rb:75:in `ping_pulp3_without_auth'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/models/katello/ping.rb:228:in `ping_services_for_capsule'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/models/katello/ping.rb:21:in `ping'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/lib/actions/middleware/backend_services_check.rb:13:in `block in plan'
 3b366271 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
 3b366271 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/lib/actions/middleware/backend_services_check.rb:13:in `plan'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.1/app/lib/actions/middleware/remote_action.rb:9:in `plan'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 3b366271 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'


Expected results:

No such errors.


Additional info:

* Capsule is configured with only one vCPU and thus has only one pulpcore-worker .

* Also all capsule services are running just fine.

* In Satellite I keep on noticing the following lines for capsule.

2021-08-12T22:26:27 [I|app|ba04b151] Started POST "/api/config_reports" for 10.74.XX.XX at 2021-08-12 22:26:27 +0530
2021-08-12T22:26:27 [I|app|ba04b151] Processing by Api::V2::ConfigReportsController#create as JSON
2021-08-12T22:26:27 [I|app|ba04b151]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2021-08-12T22:26:27 [W|app|ba04b151] No smart proxy server found on ["capsule.example.com"] and is not in trusted_hosts
2021-08-12T22:26:27 [I|app|ba04b151]   Rendering api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout
2021-08-12T22:26:27 [I|app|ba04b151]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 19.2ms | Allocations: 7377)
2021-08-12T22:26:27 [I|app|ba04b151] Filter chain halted as #<Proc:0x000000001854ee68 /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14> rendered or redirected
2021-08-12T22:26:27 [I|app|ba04b151] Completed 403 Forbidden in 101ms (Views: 24.6ms | ActiveRecord: 2.4ms | Allocations: 34307)

Comment 9 Justin Sherrill 2021-09-07 17:13:00 UTC

It does look like the problem only happens for me when using a single pulpcore worker.  We can adjust the code to allow 1

Comment 10 Justin Sherrill 2021-09-07 17:18:18 UTC

Created redmine issue https://projects.theforeman.org/issues/33430 from this bug

Comment 11 Bryan Kearney 2021-09-07 20:05:33 UTC

Upstream bug assigned to jsherril

Comment 12 Bryan Kearney 2021-09-07 20:05:35 UTC

Upstream bug assigned to jsherril

Comment 14 Vladimír Sedmík 2021-10-05 18:54:14 UTC

Reproduced at 6.10.0 snap 18, verified at snap 21 (tfm-rubygem-katello-4.1.1.29-1.el7sat.noarch):
1) Had a SAT with an external CAPS.
2) Ran `satellite-installer --foreman-proxy-content-pulpcore-worker-count=1` on the CAPS to reduce count of the pulp workers to 1.
3) Synced some repos to the SAT and triggered optimized and complete sync to the CAPS.

The sync succeeded, no errors spotted in logs nor in UI. Also the 'Pulp storage' on Capsule details page reports properly.

Comment 17 errata-xmlrpc 2021-11-16 14:13:06 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:4702