Bug 2033853 - Publish content view failed with "PulpRpmClient::ApiError Error message: the server returns an error"
Summary: Publish content view failed with "PulpRpmClient::ApiError Error message: the ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.10.0
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: 6.11.0
Assignee: Justin Sherrill
QA Contact: Griffin Sullivan
URL:
Whiteboard:
: 2052333 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-18 06:00 UTC by Hao Chang Yu
Modified: 2022-08-25 05:17 UTC (History)
20 users (show)

Fixed In Version: pulpcore-client-3.16.7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2080350 (view as bug list)
Environment:
Last Closed: 2022-07-05 14:31:10 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 34411 0 Normal Ready For Testing rest_client_timeout not respected as part of pulp3 api usage 2022-02-28 14:31:44 UTC
Red Hat Knowledge Base (Solution) 6667331 0 None None None 2022-02-07 10:29:19 UTC
Red Hat Product Errata RHSA-2022:5498 0 None None None 2022-07-05 14:31:47 UTC

Description Hao Chang Yu 2021-12-18 06:00:46 UTC
Description of problem:
Publish a content view sometimes failed to index repo content with the following error. This is a timeout error when connecting to Pulp 3 due to 60 seconds non-adjustable timeout in the api client.

When publishing a content view, Satellite sends requests to Pulp to fetch 2000 rpm records per batch. If it can't finish the request in 60 seconds, the request will timeout.

Error:
PulpRpmClient::ApiError
Error message: the server returns an error
---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:90:in
  `rescue in call_api'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:68:in
  `call_api'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:236:in
  `list_with_http_info'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:130:in
  `list'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:93:in
  `content_unit_list'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:106:in
  `fetch_content_list'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:75:in
  `block (2 levels) in pulp_units_batch_for_repo'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:69:in
  `loop'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:69:in
  `block in pulp_units_batch_for_repo'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/concerns/pulp_database_unit.rb:164:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/concerns/pulp_database_unit.rb:164:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/concerns/pulp_database_unit.rb:164:in
  `import_for_repository'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:916:in
  `block (2 levels) in index_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/lib/katello/logging.rb:8:in
  `time'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:915:in
  `block in index_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:914:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:914:in
  `index_content'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/lib/actions/katello/repository/index_content.rb:22:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:572:in
  `block (3 levels) in execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in
  `pass'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in
  `pass


Root cause:
Excon has default 60 second timeout.

/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb
   ...
    def build_request(http_method, path, request, opts = {})
      url = build_request_url(path)
      http_method = http_method.to_sym.downcase

      header_params = @default_headers.merge(opts[:header_params] || {})
      query_params = opts[:query_params] || {}
      form_params = opts[:form_params] || {}

      update_params_for_auth! header_params, query_params, opts[:auth_names]

      req_opts = {
        :method => http_method,
        :headers => header_params,
        :params => query_params,
        :params_encoding => @config.params_encoding,
        :timeout => @config.timeout,   <============= This timeout value is 0 and it doesn't pass to the excon adapter.
        :verbose => @config.debugging
      }

Note: According to my test, setting timeout=0 in excon means timeout immediately.

Comment 2 Justin Sherrill 2022-01-06 15:55:26 UTC
Hao, 

There's a bulk_load_size under Administer > Settings > Content that can be adjusted down from 2000, to 1000 or 500.  Does that help?

Justin

Comment 3 Hao Chang Yu 2022-01-11 07:24:11 UTC
(In reply to Justin Sherrill from comment #2)
> Hao, 
> 
> There's a bulk_load_size under Administer > Settings > Content that can be
> adjusted down from 2000, to 1000 or 500.  Does that help?
> 
> Justin

Yeah, that seems to help. Thanks. How could I missed that?

Comment 4 Hao Chang Yu 2022-01-12 02:33:26 UTC
(In reply to Hao Chang Yu from comment #3)
> (In reply to Justin Sherrill from comment #2)
> > Hao, 
> > 
> > There's a bulk_load_size under Administer > Settings > Content that can be
> > adjusted down from 2000, to 1000 or 500.  Does that help?
> > 
> > Justin
> 
> Yeah, that seems to help. Thanks. How could I missed that?

It seems like you might need to restart the Satellite to take effect? I only test it in foreman-rake console (calling repo.index_content). It seems like after adjusting the value from GUI, the setting doesn't get updated in the console until I exit and re-enter the console

Comment 5 Justin Sherrill 2022-01-12 13:37:22 UTC
Hao, 

Yes, settings aren't reloaded in the console automatically, but are reloaded on every api request.  So you shouldn't need to restart the server, but would need to restart the console.

Justin

Comment 12 Sayan Das 2022-02-14 09:01:31 UTC
FYI, This is affecting Satellite 7.0 (Beta) as well. Even if I have different sync plans for different products scheduled to be executed 2 or 3 hours apart from each other, atleast two Big repos fail to sync and the error is at the IndexContent step only.

Comment 15 Justin Sherrill 2022-02-21 14:37:41 UTC
Yes, we're gonna take a look at this.  I believe the timeout is actually 60 seconds, which is lower than intended.

Comment 16 Justin Sherrill 2022-02-21 20:22:41 UTC
Connecting redmine issue https://projects.theforeman.org/issues/34411 from this bug

Comment 17 Justin Sherrill 2022-02-23 14:24:46 UTC
After some investigation, this is a bit more complicated than i had thought. 

After making the proper change in katello: https://github.com/Katello/katello/pull/9930

I discovered that there is a bug in the pulp bindings.  This is fixed upstream: https://github.com/OpenAPITools/openapi-generator/commit/0eb7189faea45cd83384119cbfd7180909066ead

But to fix it in our pulp bindings would require all the ruby bindings to be rebuilt as well.

Moving to the pulp component to figure that out.

Comment 18 Justin Sherrill 2022-02-23 14:28:31 UTC
Due to this, i'm suggesting we remove this from 6.9.z, since there is a workaround for migrations and the complexity to achieving this in those old version is probably fairly high.

Comment 19 Bryan Kearney 2022-02-23 16:04:51 UTC
Upstream bug assigned to jsherril

Comment 20 Bryan Kearney 2022-02-23 16:04:54 UTC
Upstream bug assigned to jsherril

Comment 21 Justin Sherrill 2022-02-24 20:12:22 UTC
*** Bug 2052333 has been marked as a duplicate of this bug. ***

Comment 23 Bryan Kearney 2022-03-01 00:05:02 UTC
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34411 has been resolved.

Comment 25 Dennis Kliban 2022-04-26 15:57:37 UTC
The timeout setting was exposed in pulp-rpm-client 3.14.14.

Comment 26 Justin Sherrill 2022-04-29 13:53:52 UTC
Moving to post, but keep in mind this needs a katello cherry pick AND rubygem-pulp-rpm-client 3.14.14.

Comment 37 errata-xmlrpc 2022-07-05 14:31:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498


Note You need to log in before you can comment on or make changes to this bug.