Description of problem: When trying to sync content from RH CDN with an HTTP proxy enabled, the sync fails with an error "'_UnixSelectorEventLoop' object has no attribute 'start_tls' (Katello::Errors::Pulp3Error)" Version-Release number of selected component (if applicable): Sat 6.10 Snap 21 How reproducible: always Steps to Reproduce: 1. Configure an HTTP proxy which can access RH CDN (generally the public Internet) on a Satellite instance (Infrastructure - HTTP Proxies). 2. Set this proxy as default for content syncing (Administer - Settings - Content tab - Default HTTP proxy) 3. Upload a manifest with some CDN content subscriptions. 4. Enable a RH repository (e.g. "Red Hat Enterprise Linux 7 Server RPMs x86_64 7Server" in Content - Red Hat Repositories) 5. Try to sync the repo you have just enabled Actual results: The task fails with the following error "'_UnixSelectorEventLoop' object has no attribute 'start_tls'" Expected results: The sync is performed successfully. Additional info: Syncing without a content HTTP proxy works ok. /var/log/foreman/production.log contains following error: 2021-10-06T05:32:02 [E|bac|5b2228a4] '_UnixSelectorEventLoop' object has no attribute 'start_tls' (Katello::Errors::Pulp3Error) 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/lib/actions/pulp3/abstract_async_task.rb:108:in `block in check_for_errors' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/lib/actions/pulp3/abstract_async_task.rb:106:in `each' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/lib/actions/pulp3/abstract_async_task.rb:106:in `check_for_errors' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/lib/actions/pulp3/abstract_async_task.rb:139:in `poll_external_task' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/polling.rb:22:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/cancellable.rb:14:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:572:in `block (3 levels) in execute_run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:32:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/lib/actions/middleware/remote_action.rb:16:in `block in run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/models/katello/concerns/user_extensions.rb:21:in `cp_config' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.29/app/lib/actions/middleware/remote_action.rb:16:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:31:in `with_progress_calculation' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:17:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/load_setting_values.rb:20:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:32:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_user.rb:15:in `run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/world.rb:31:in `execute' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:571:in `block (2 levels) in execute_run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:570:in `catch' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:570:in `block in execute_run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `block in with_error_handling' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `catch' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `with_error_handling' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:565:in `execute_run' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:286:in `execute' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director.rb:94:in `execute' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors.rb:18:in `run_user_code' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform' 5b2228a4 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform' 5b2228a4 | [ sidekiq ] 5b2228a4 | [ concurrent-ruby ]
I bet this comes from https://github.com/theforeman/pulpcore-packaging/pull/288/files
One thing that I noticed: https://github.com/aio-libs/aiohttp/pull/5992 (where the patch we applied comes from, essentially) mentions Python 3.7+, but we're running this on 3.6!
@Evgeni is correct in that this patch does not work on Python 3.6. CPython 3.6 does not have support for TLS in TLS. This feature was added in Python 3.7.
For the context, see https://github.com/aio-libs/aiohttp/pull/5992.patch — this is the upstream patch that I've got merged into the aiohttp master last night. master corresponds to v4.0+ and has already dropped support for Python 3.6. There's a backporting attempt for this change at https://github.com/aio-libs/aiohttp/pull/6049 to get it into the upcoming release of aiohttp v3.8.0. It hasn't yet dropped support for Python 3.6 but there's a chance it will because CPython 3.6 will reach EOL in two months and it's rather pointless to support it upstream. The reason the backport is not yet merged is that this specific error is happening in the CI jobs for Python 3.6 (that are still present in the branch `3.8` but haven't been in the master branch for over a year at least). The current plan is to recover the guard expression in the code, just for aiohttp v3.8 that would tell the users when they attempt to use this feature in an unsupported environment (which is anything below Python 3.7, for `start_tls()`). So for Sattelite to use this feature, it'll be necessary to upgrade the Python runtime to 3.7 or higher and migrate to using aiohttp v3.8 once it's out. I would recommend using something newer, though. Like Python 3.10 or 3.9: over the years, since this feature has been added, there's been talk about it being having a lot of corner cases that got addressed over time. So the ideal case for this is to use Python 3.10+ really. If you attempt to backport those upstream patches on top of CPython 3.6, I foresee that it may end up being rather complicated and I wouldn't have any confidence in such a setup.
I believe the behavior described by this bugzilla is a result of including bug 1993917. We will likely need to revert that change from the current 6.10 SNAP, at which time this one can go to ON_QA to confirm the issue no longer exists.
@ssydoren Thanks for the info. We won't go to a Python 3.9 or 3.10 as we are reliant on the SCL from RHEL. We will move forward with an option to get to python 3.8. Here are some more details on the decision. https://docs.google.com/document/d/1nusVTz4Kdx-HPb05kUHA-HMF0MBMX7vZxETGZ1YoPhA/edit?usp=sharing
I have successfully synced a RH repository using an HTTP proxy on a Sat 6.10 Snap 23.0 in the IPv6 only environment. I will leave IPv4 testing to Jameer, but I believe there shouldn't be much of a difference.
Verified Verified with: - Satellite 6.10.0 snap 23 Test steps: - Create HTTP Proxy (Infrastructure > Http proxies) - Set it as content default http proxy in setting - Try to sync redhat repository. Observations: - Repository sync completed successfully.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Satellite 6.10 Release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:4702