Description of problem: Error message: the server returns an error (PulpCertguardClient::ApiError) while publishing content-view Actions::Pulp3::ContentGuard::Refresh PulpCertguardClient::ApiError: Error message: the server returns an error Error message: the server returns an errorError message: the server returns an error Version-Release number of selected component (if applicable): satellite-6.10.0-0.6.beta How reproducible: On a new installation of 6.10 Beta Steps to Reproduce: 1. Install new 6.10 Beta 2. Create a content view and add single repo with Docker Repository 3. Publish content-view Actual results: ==> /var/log/foreman/production.log <== 2021-08-20T12:39:54 [I|app|e64fef84] Started POST "/foreman_tasks/api/tasks/bulk_search" for 10.74.16.219 at 2021-08-20 12:39:54 +0530 2021-08-20T12:39:54 [I|app|e64fef84] Processing by ForemanTasks::Api::TasksController#bulk_search as HTML 2021-08-20T12:39:54 [I|app|e64fef84] Parameters: {"searches"=>[{"type"=>"task", "task_id"=>"af43d379-a4bb-4bed-9f15-4f58314f4ac8", "search_id"=>"1"}], "task"=>{}} 2021-08-20T12:39:55 [I|app|e64fef84] Completed 200 OK in 96ms (Views: 0.9ms | ActiveRecord: 6.0ms | Allocations: 47182) 2021-08-20T12:39:58 [I|app|8ac381e9] CONTENT_INDEX type=Katello::Rpm success=false duration=60123.59 2021-08-20T12:39:58 [E|bac|8ac381e9] Error message: the server returns an error (PulpRpmClient::ApiError) 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:90:in `rescue in call_api' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:68:in `call_api' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:236:in `list_with_http_info' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:130:in `list' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/services/katello/pulp3/pulp_content_unit.rb:93:in `content_unit_list' 8ac381e9 | [ sidekiq ] 8ac381e9 | [ concurrent-ruby ] 2021-08-20T12:39:58 [E|bac|8ac381e9] Error message: the server returns an error (PulpCertguardClient::ApiError) 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_certguard_client-1.4.0/lib/pulp_certguard_client/api_client.rb:90:in `rescue in call_api' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_certguard_client-1.4.0/lib/pulp_certguard_client/api_client.rb:68:in `call_api' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_certguard_client-1.4.0/lib/pulp_certguard_client/api/contentguards_rhsm_api.rb:225:in `list_with_http_info' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_certguard_client-1.4.0/lib/pulp_certguard_client/api/contentguards_rhsm_api.rb:161:in `list' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/services/katello/pulp3/api/content_guard.rb:65:in `list' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/services/katello/pulp3/api/content_guard.rb:32:in `refresh' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.8/app/lib/actions/pulp3/content_guard/refresh.rb:10:in `run' 8ac381e9 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:572:in `block (3 levels) in execute_run' 2021-08-20T12:57:04 [I|bac|8ac381e9] Task {label: Actions::Katello::ContentView::Publish, id: af43d379-a4bb-4bed-9f15-4f58314f4ac8, execution_plan_id: 91946f89-c59a-4575-9974-964e2c051dda} state changed: paused result: error /var/log/messages: Aug 20 12:56:01 pulpcore-api: pulp [8ac381e9-f27c-48a8-9df1-46c4dd2fad40]: - - [20/Aug/2021:07:26:01 +0000] "GET /pulp/api/v3/content/rpm/packages/?arch__ne=src&fields=pulp_href&limit=2000&offset=2000&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2F1efe5081-5c58-4202-b54c-a6fc9dea0882%2Fversions%2F1%2F HTTP/1.1" 200 176545 "-" "OpenAPI-Generator/3.13.3/ruby" Aug 20 12:57:09 pulpcore-api: pulp [8ac381e9-f27c-48a8-9df1-46c4dd2fad40]: - - [20/Aug/2021:07:27:09 +0000] "GET /pulp/api/v3/content/rpm/packages/?arch__ne=src&fields=pulp_href&limit=2000&offset=4000&repository_version=%2Fpulp%2Fapi%2Fv3%2Frepositories%2Frpm%2Frpm%2F1efe5081-5c58-4202-b54c-a6fc9dea0882%2Fversions%2F1%2F HTTP/1.1" 200 176557 "-" "OpenAPI-Generator/3.13.3/ruby" Expected results: Additional info: # hammer ping database: Status: ok Server Response: Duration: 0ms candlepin: Status: ok Server Response: Duration: 36ms candlepin_auth: Status: ok Server Response: Duration: 27ms candlepin_events: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 0ms katello_events: Status: ok message: 0 Processed, 0 Failed Server Response: Duration: 0ms pulp3: Status: ok Server Response: Duration: 92ms foreman_tasks: Status: ok Server Response: Duration: 5ms # satellite-maintain service status -b Running Status Services ================================================================================ Get status of applicable services: Displaying the following service(s): rh-redis5-redis, postgresql, pulpcore-api, pulpcore-content, rh-redis5-redis, pulpcore-worker, pulpcore-worker, pulpcore-worker, pulpcore-worker, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, puppetserver, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy, foreman-cockpit | displaying rh-redis5-redis [OK] / displaying postgresql [OK] / displaying pulpcore-api [OK] / displaying pulpcore-content [OK] / displaying rh-redis5-redis [OK] / displaying pulpcore-worker [OK] / displaying pulpcore-worker [OK] / displaying pulpcore-worker [OK] / displaying pulpcore-worker [OK] / displaying tomcat [OK] / displaying dynflow-sidekiq@orchestrator [OK] / displaying foreman [OK] / displaying httpd [OK] / displaying puppetserver [OK] / displaying dynflow-sidekiq@worker-1 [OK] / displaying dynflow-sidekiq@worker-hosts-queue-1 [OK] / displaying foreman-proxy [OK] - displaying foreman-cockpit [OK] - All services are running [OK] -------------------------------------------------------------------------------- Id:af43d379-a4bb-4bed-9f15-4f58314f4ac8 Label:Actions::Katello::ContentView::Publish Duration:14 minutes { "content_view": { "id": 2, "name": "RHEL7", "label": "RHEL7" }, "organization": { "id": 1, "name": "Default Organization", "label": "Default_Organization" }, "services_checked": [ "pulp3", "candlepin", "candlepin_auth" ], "history_id": 1, "content_view_id": 2, "auto_publish_composite_ids": [], "content_view_version_name": "RHEL7 1.0", "content_view_version_id": 2, "environment_id": 1, "user_id": 4, "skip_promotion": null, "current_request_id": "8ac381e9-f27c-48a8-9df1-46c4dd2fad40", "current_timezone": "Asia/Kolkata", "current_organization_id": 1, "current_location_id": 2, "current_user_id": 4 }
The katello traceback doesn't seem to be relevant to docker content at all. It looks like there is rpm content involved. The logs provided also do not look relevant (from 15 minutes after the error), is the box still up for us to investigate? or could you provide more logs?
Can you try increasing the number of pulpcore-api workers? edit /etc/systemd/system/pulpcore-api.service change -w 1 \ to -w 3 \ and then run: systemctl daemon-reload systemctl restart pulpcore-api and see if you can reproduce on the same system? This configuration isn't available via the installer, but this would be a good bz to cause that to b done. There is another option you may be able to adjust, but i'd like you to try this one first.
*** Bug 1996137 has been marked as a duplicate of this bug. ***
This looks like a dupe but with more details about the setup https://bugzilla.redhat.com/show_bug.cgi?id=1996485
I encountered this issue while publishing a content view containing RHEL 6, 7, and 8 RPM repos. I worked around by setting `-w 3`: [root@ci-vm-10-0-149-5 ~]# systemctl edit pulpcore-api.service [root@ci-vm-10-0-149-5 ~]# systemctl restart pulpcore-api [root@ci-vm-10-0-149-5 ~]# cat /etc/systemd/system/pulpcore-api.service.d/override.conf [Service] ExecStart= ExecStart=/usr/libexec/pulpcore/gunicorn pulpcore.app.wsgi:application \ --timeout 90 \ -w 3 \ --access-logfile - \ --access-logformat 'pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"' ...and then resuming the paused tasks. They eventually succeeded. The satellite in question may be somewhat resource starved, as it is a VM.
Thanks! we're working on increasing the api worker count (what that -w controls) here: https://bugzilla.redhat.com/show_bug.cgi?id=2002693
gonna close this as a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=2002693 since the solution is there *** This bug has been marked as a duplicate of bug 2002693 ***
*** Bug 1996485 has been marked as a duplicate of this bug. ***