Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2033853

Summary: Publish content view failed with "PulpRpmClient::ApiError Error message: the server returns an error"
Product: Red Hat Satellite Reporter: Hao Chang Yu <hyu>
Component: PulpAssignee: Justin Sherrill <jsherril>
Status: CLOSED ERRATA QA Contact: Lai <ltran>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.10.0CC: ahumbe, alsouza, casl, dkliban, gsulliva, iballou, jbhatia, jsherril, jstormshak, kkinge, matthew.lesieur, mjia, mkalyat, patalber, pcreech, pmendezh, pmoravec, pratshar, risantam, saydas, sraut, tharring, wpinheir
Target Milestone: 6.11.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: pulpcore-client-3.16.7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2080350 (view as bug list) Environment:
Last Closed: 2022-07-05 14:31:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hao Chang Yu 2021-12-18 06:00:46 UTC 
Description of problem:
Publish a content view sometimes failed to index repo content with the following error. This is a timeout error when connecting to Pulp 3 due to 60 seconds non-adjustable timeout in the api client.

When publishing a content view, Satellite sends requests to Pulp to fetch 2000 rpm records per batch. If it can't finish the request in 60 seconds, the request will timeout.

Error:
PulpRpmClient::ApiError
Error message: the server returns an error
---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:90:in
  `rescue in call_api'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:68:in
  `call_api'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:236:in
  `list_with_http_info'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:130:in
  `list'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:93:in
  `content_unit_list'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:106:in
  `fetch_content_list'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:75:in
  `block (2 levels) in pulp_units_batch_for_repo'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:69:in
  `loop'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:69:in
  `block in pulp_units_batch_for_repo'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/concerns/pulp_database_unit.rb:164:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/concerns/pulp_database_unit.rb:164:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/concerns/pulp_database_unit.rb:164:in
  `import_for_repository'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:916:in
  `block (2 levels) in index_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/lib/katello/logging.rb:8:in
  `time'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:915:in
  `block in index_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:914:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:914:in
  `index_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/lib/actions/katello/repository/index_content.rb:22:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:572:in
  `block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass


Root cause:
Excon has default 60 second timeout.

/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb
   ...
    def build_request(http_method, path, request, opts = {})
      url = build_request_url(path)
      http_method = http_method.to_sym.downcase

      header_params = @default_headers.merge(opts[:header_params] || {})
      query_params = opts[:query_params] || {}
      form_params = opts[:form_params] || {}

      update_params_for_auth! header_params, query_params, opts[:auth_names]

      req_opts = {
        :method => http_method,
        :headers => header_params,
        :params => query_params,
        :params_encoding => @config.params_encoding,
        :timeout => @config.timeout,   <============= This timeout value is 0 and it doesn't pass to the excon adapter.
        :verbose => @config.debugging
      }

Note: According to my test, setting timeout=0 in excon means timeout immediately.
Comment 2 Justin Sherrill 2022-01-06 15:55:26 UTC 
Hao, 

There's a bulk_load_size under Administer > Settings > Content that can be adjusted down from 2000, to 1000 or 500.  Does that help?

Justin
Comment 3 Hao Chang Yu 2022-01-11 07:24:11 UTC 
(In reply to Justin Sherrill from comment #2)
> Hao, 
> 
> There's a bulk_load_size under Administer > Settings > Content that can be
> adjusted down from 2000, to 1000 or 500.  Does that help?
> 
> Justin

Yeah, that seems to help. Thanks. How could I missed that?
Comment 4 Hao Chang Yu 2022-01-12 02:33:26 UTC 
(In reply to Hao Chang Yu from comment #3)
> (In reply to Justin Sherrill from comment #2)
> > Hao, 
> > 
> > There's a bulk_load_size under Administer > Settings > Content that can be
> > adjusted down from 2000, to 1000 or 500.  Does that help?
> > 
> > Justin
> 
> Yeah, that seems to help. Thanks. How could I missed that?

It seems like you might need to restart the Satellite to take effect? I only test it in foreman-rake console (calling repo.index_content). It seems like after adjusting the value from GUI, the setting doesn't get updated in the console until I exit and re-enter the console
Comment 5 Justin Sherrill 2022-01-12 13:37:22 UTC 
Hao, 

Yes, settings aren't reloaded in the console automatically, but are reloaded on every api request.  So you shouldn't need to restart the server, but would need to restart the console.

Justin
Comment 12 Sayan Das 2022-02-14 09:01:31 UTC 
FYI, This is affecting Satellite 7.0 (Beta) as well. Even if I have different sync plans for different products scheduled to be executed 2 or 3 hours apart from each other, atleast two Big repos fail to sync and the error is at the IndexContent step only.
Comment 15 Justin Sherrill 2022-02-21 14:37:41 UTC 
Yes, we're gonna take a look at this.  I believe the timeout is actually 60 seconds, which is lower than intended.
Comment 16 Justin Sherrill 2022-02-21 20:22:41 UTC 
Connecting redmine issue https://projects.theforeman.org/issues/34411 from this bug
Comment 17 Justin Sherrill 2022-02-23 14:24:46 UTC 
After some investigation, this is a bit more complicated than i had thought. 

After making the proper change in katello: https://github.com/Katello/katello/pull/9930

I discovered that there is a bug in the pulp bindings.  This is fixed upstream: https://github.com/OpenAPITools/openapi-generator/commit/0eb7189faea45cd83384119cbfd7180909066ead

But to fix it in our pulp bindings would require all the ruby bindings to be rebuilt as well.

Moving to the pulp component to figure that out.
Comment 18 Justin Sherrill 2022-02-23 14:28:31 UTC 
Due to this, i'm suggesting we remove this from 6.9.z, since there is a workaround for migrations and the complexity to achieving this in those old version is probably fairly high.
Comment 19 Bryan Kearney 2022-02-23 16:04:51 UTC 
Upstream bug assigned to jsherril
Comment 20 Bryan Kearney 2022-02-23 16:04:54 UTC 
Upstream bug assigned to jsherril
Comment 21 Justin Sherrill 2022-02-24 20:12:22 UTC 
*** Bug 2052333 has been marked as a duplicate of this bug. ***
Comment 23 Bryan Kearney 2022-03-01 00:05:02 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34411 has been resolved.
Comment 25 Dennis Kliban 2022-04-26 15:57:37 UTC 
The timeout setting was exposed in pulp-rpm-client 3.14.14.
Comment 26 Justin Sherrill 2022-04-29 13:53:52 UTC 
Moving to post, but keep in mind this needs a katello cherry pick AND rubygem-pulp-rpm-client 3.14.14.
Comment 37 errata-xmlrpc 2022-07-05 14:31:10 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498