Bug 2033853
| Summary: | Publish content view failed with "PulpRpmClient::ApiError Error message: the server returns an error" | |||
|---|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Hao Chang Yu <hyu> | |
| Component: | Pulp | Assignee: | Justin Sherrill <jsherril> | |
| Status: | CLOSED ERRATA | QA Contact: | Lai <ltran> | |
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 6.10.0 | CC: | ahumbe, alsouza, casl, dkliban, gsulliva, iballou, jbhatia, jsherril, jstormshak, kkinge, matthew.lesieur, mjia, mkalyat, patalber, pcreech, pmendezh, pmoravec, pratshar, risantam, saydas, sraut, tharring, wpinheir | |
| Target Milestone: | 6.11.0 | Keywords: | Triaged | |
| Target Release: | Unused | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | pulpcore-client-3.16.7 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2080350 (view as bug list) | Environment: | ||
| Last Closed: | 2022-07-05 14:31:10 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
Hao, There's a bulk_load_size under Administer > Settings > Content that can be adjusted down from 2000, to 1000 or 500. Does that help? Justin (In reply to Justin Sherrill from comment #2) > Hao, > > There's a bulk_load_size under Administer > Settings > Content that can be > adjusted down from 2000, to 1000 or 500. Does that help? > > Justin Yeah, that seems to help. Thanks. How could I missed that? (In reply to Hao Chang Yu from comment #3) > (In reply to Justin Sherrill from comment #2) > > Hao, > > > > There's a bulk_load_size under Administer > Settings > Content that can be > > adjusted down from 2000, to 1000 or 500. Does that help? > > > > Justin > > Yeah, that seems to help. Thanks. How could I missed that? It seems like you might need to restart the Satellite to take effect? I only test it in foreman-rake console (calling repo.index_content). It seems like after adjusting the value from GUI, the setting doesn't get updated in the console until I exit and re-enter the console Hao, Yes, settings aren't reloaded in the console automatically, but are reloaded on every api request. So you shouldn't need to restart the server, but would need to restart the console. Justin FYI, This is affecting Satellite 7.0 (Beta) as well. Even if I have different sync plans for different products scheduled to be executed 2 or 3 hours apart from each other, atleast two Big repos fail to sync and the error is at the IndexContent step only. Yes, we're gonna take a look at this. I believe the timeout is actually 60 seconds, which is lower than intended. Connecting redmine issue https://projects.theforeman.org/issues/34411 from this bug After some investigation, this is a bit more complicated than i had thought. After making the proper change in katello: https://github.com/Katello/katello/pull/9930 I discovered that there is a bug in the pulp bindings. This is fixed upstream: https://github.com/OpenAPITools/openapi-generator/commit/0eb7189faea45cd83384119cbfd7180909066ead But to fix it in our pulp bindings would require all the ruby bindings to be rebuilt as well. Moving to the pulp component to figure that out. Due to this, i'm suggesting we remove this from 6.9.z, since there is a workaround for migrations and the complexity to achieving this in those old version is probably fairly high. Upstream bug assigned to jsherril Upstream bug assigned to jsherril *** Bug 2052333 has been marked as a duplicate of this bug. *** Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34411 has been resolved. The timeout setting was exposed in pulp-rpm-client 3.14.14. Moving to post, but keep in mind this needs a katello cherry pick AND rubygem-pulp-rpm-client 3.14.14. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:5498 |
Description of problem: Publish a content view sometimes failed to index repo content with the following error. This is a timeout error when connecting to Pulp 3 due to 60 seconds non-adjustable timeout in the api client. When publishing a content view, Satellite sends requests to Pulp to fetch 2000 rpm records per batch. If it can't finish the request in 60 seconds, the request will timeout. Error: PulpRpmClient::ApiError Error message: the server returns an error --- - "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:90:in `rescue in call_api'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb:68:in `call_api'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:236:in `list_with_http_info'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api/content_packages_api.rb:130:in `list'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:93:in `content_unit_list'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:106:in `fetch_content_list'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:75:in `block (2 levels) in pulp_units_batch_for_repo'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:69:in `loop'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/services/katello/pulp3/pulp_content_unit.rb:69:in `block in pulp_units_batch_for_repo'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/concerns/pulp_database_unit.rb:164:in `each'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/concerns/pulp_database_unit.rb:164:in `each'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/concerns/pulp_database_unit.rb:164:in `import_for_repository'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:916:in `block (2 levels) in index_content'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/lib/katello/logging.rb:8:in `time'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:915:in `block in index_content'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:914:in `each'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/models/katello/repository.rb:914:in `index_content'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/app/lib/actions/katello/repository/index_content.rb:22:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:572:in `block (3 levels) in execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass Root cause: Excon has default 60 second timeout. /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.13.3/lib/pulp_rpm_client/api_client.rb ... def build_request(http_method, path, request, opts = {}) url = build_request_url(path) http_method = http_method.to_sym.downcase header_params = @default_headers.merge(opts[:header_params] || {}) query_params = opts[:query_params] || {} form_params = opts[:form_params] || {} update_params_for_auth! header_params, query_params, opts[:auth_names] req_opts = { :method => http_method, :headers => header_params, :params => query_params, :params_encoding => @config.params_encoding, :timeout => @config.timeout, <============= This timeout value is 0 and it doesn't pass to the excon adapter. :verbose => @config.debugging } Note: According to my test, setting timeout=0 in excon means timeout immediately.