Bug 1993917 - Syncing repositories with https proxy set ends with warning Katello::Errors::Pulp3 Error Only http proxies are supported
Summary: Syncing repositories with https proxy set ends with warning Katello::Errors::...
Keywords:
Status: ASSIGNED
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.10.0
Hardware: Unspecified
OS: Unspecified
high
high vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: addubey
URL:
Whiteboard:
: 2054148 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-08-16 12:10 UTC by Ahmed Eladawy
Modified: 2022-09-23 18:27 UTC (History)
37 users (show)

Fixed In Version: pulpcore-3.16.6,aiohttp-3.8.1
Doc Type: Known Issue
Doc Text:
Pulp - Syncing repositories with HTTPS proxy set ends with warning Syncing repositories with HTTPS proxy set ends with warning Katello::Errors::Pulp3 Error Only HTTP proxies are supported. A workaround is to use plain HTTP proxy or an HTTP proxy that is upgraded to HTTPS using the HTTP CONNECT method.
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github pulp pulpcore issues 3036 0 None open HTTPs Proxies not working. 2022-09-09 17:27:52 UTC
Github theforeman pulpcore-packaging pull 288 0 None Merged Adds aiohttp secure proxy support 2022-09-09 17:27:43 UTC
Pulp Redmine 9304 0 High CLOSED - NOTABUG As a user, I can configure https proxy using https:// 2021-09-29 21:07:36 UTC
Red Hat Issue Tracker SAT-12304 0 None None None 2022-08-18 21:10:57 UTC

Description Ahmed Eladawy 2021-08-16 12:10:27 UTC
Description of problem:

Syncing repositories with https proxy set ends with warning Katello::Errors::Pulp3  Error Only http proxies are supported

Version-Release number of selected component (if applicable):

6.10

How reproducible:

100

Steps to Reproduce:
1. Create HTTP proxy with HTTPS URL
2. Set the Default HTTP Proxy for syncing content from setting to the created one.
3. Sync any Red Hat repository.

Actual results:

The syncing task finishes with warning with the following message :

Katello::Errors::Pulp3Error

Only http proxies are supported

Expected results:

Both HTTP and HTTPS proxies should work without issues.

Additional info:

- Changing the proxy URL from https to http works.
- No hints to use only http proxy.
- The example URL of the URL section when creating a proxy contains https example "URL of the proxy including schema (https://proxy.example.com:8080)"

Comment 1 Ahmed Eladawy 2021-08-16 12:14:11 UTC
Log trace :

pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/c3aadbfe-2b3d-449d-8e65-9666951476d2/"
  pulp_created: '2021-08-16T11:20:14.160+00:00'
  state: failed
  name: pulp_rpm.app.tasks.synchronizing.synchronize
  logging_cid: 0f189218-3ac5-4a3f-b183-0d6b916aa4f2
  started_at: '2021-08-16T11:20:14.249+00:00'
  finished_at: '2021-08-16T11:20:14.374+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 363, in synchronize
          remote_url = fetch_remote_url(remote)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 242, in fetch_remote_url
          get_repomd_file(remote, normalized_remote_url)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 200, in get_repomd_file
          return downloader.fetch()
        File "/usr/lib/python3.6/site-packages/pulpcore/download/base.py", line 176, in fetch
          return done.pop().result()
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 258, in run
          return await download_wrapper()
        File "/usr/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
          ret = await target(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 256, in download_wrapper
          return await self._run(extra_data=extra_data)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 92, in _run
          url, proxy=self.proxy, proxy_auth=self.proxy_auth, auth=self.auth
        File "/usr/lib64/python3.6/site-packages/aiohttp/client.py", line 1117, in __aenter__
          self._resp = await self._coro
        File "/usr/lib64/python3.6/site-packages/aiohttp/client.py", line 513, in _request
          traces=traces,
        File "/usr/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 311, in __init__
          self.update_proxy(proxy, proxy_auth, proxy_headers)
        File "/usr/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 551, in update_proxy
          raise ValueError("Only http proxies are supported")
    description: Only http proxies are supported
  worker: "/pulp/api/v3/workers/204569e0-9f35-48ca-9e22-5cbfd8828dcc/"
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/abc9dd0c-d198-433c-af59-47b40de82db6/"
  - "/pulp/api/v3/remotes/rpm/rpm/d86a3ede-74f5-4f11-b979-34e9f0e6dd27/"
create_version: true
task_groups: []
poll_attempts:
  total: 1
  failed: 1

Comment 2 Tanya Tereshchenko 2021-08-22 10:44:51 UTC
The underlying library aiohttp doesn't support https proxies, specified with https://.
https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support
"aiohttp supports plain HTTP proxies and HTTP proxies that can be upgraded to HTTPS via the HTTP CONNECT method. aiohttp does not support proxies that must be connected to via https://."

HTTP CONNECT method https://www.ietf.org/rfc/rfc2817.txt.

Was it supported in Satellite 6.9?

Comment 4 Ahmed Eladawy 2021-08-27 11:45:51 UTC
(In reply to Tanya Tereshchenko from comment #2)
> The underlying library aiohttp doesn't support https proxies, specified with
> https://.
> https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support
> "aiohttp supports plain HTTP proxies and HTTP proxies that can be upgraded
> to HTTPS via the HTTP CONNECT method. aiohttp does not support proxies that
> must be connected to via https://."
> 
> HTTP CONNECT method https://www.ietf.org/rfc/rfc2817.txt.
> 
> Was it supported in Satellite 6.9?

Yes , Https proxy worked without issues on satellite 6.9

Comment 5 pulp-infra@redhat.com 2021-08-27 12:12:06 UTC
The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

Comment 6 pulp-infra@redhat.com 2021-08-27 12:12:07 UTC
The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

Comment 7 pulp-infra@redhat.com 2021-08-30 18:09:01 UTC
The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

Comment 8 pulp-infra@redhat.com 2021-09-28 19:07:14 UTC
The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

Comment 10 pulp-infra@redhat.com 2021-09-29 21:07:37 UTC
The Pulp upstream bug status is at CLOSED - NOTABUG. Updating the external tracker on this bug.

Comment 15 Ian Ballou 2021-10-19 21:48:22 UTC
I found that simply using `https_port 3128 cert=/etc/squid/ssl_cert/myCA.pem` and disabling `http_port 3128` reliably produces a proxy that can only be accessed via HTTPS, so unless anyone has any objections, I'm going to go with that for my example.  Here's my full config for the HTTPS proxy (Squid 3.5 on EL7):

```
[root@rhel7 ssl_cert]# cat /etc/squid/squid.conf
#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8	# RFC1918 possible internal network
acl localnet src 172.16.0.0/12	# RFC1918 possible internal network
acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http
acl SSL_ports port 80		# http
acl SSL_ports port 21		# ftp
acl SSL_ports port 443		# https
acl SSL_ports port 70		# gopher
acl SSL_ports port 210		# wais
acl SSL_ports port 1025-65535	# unregistered ports
acl SSL_ports port 280		# http-mgmt
acl SSL_ports port 488		# gss-http
acl SSL_ports port 591		# filemaker
acl SSL_ports port 777		# multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
#http_port 3128
https_port 3128 cert=/etc/squid/ssl_cert/myCA.pem

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /var/spool/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern .		0	20%	4320

```

Comment 27 Brian Bouterse 2022-01-14 16:11:56 UTC
I looked at the comment 24 and it all seems fine. Can someone confirm the version of Python and aiohttp on this system?

Comment 29 Grant Gainey 2022-01-20 17:04:24 UTC
(In reply to addubey from comment #28)
> Python - 2.7.5
> 
> Metadata-Version: 2.1
> Name: aiohttp
> Version: 3.7.4
> Summary: Async http client/server framework (asyncio)

Pulp3 uses Python3 - what's the pyhton3 version?

Comment 30 Brian Bouterse 2022-01-20 17:06:08 UTC
Thanks addubey. This explains why it isn't working then. Aiohttp carries the secure proxy feature, and I believe it is only in 3.8.z. Additionally my understanding is that it requires Python 3.8 in order to receive the TLS fixes. We should confirm the versions Satellite needs with ssydoren.

Comment 31 addubey 2022-01-21 06:46:23 UTC
Python3  - 3.6.8

Comment 38 addubey 2022-01-25 17:28:16 UTC
(In reply to Dennis Kliban from comment #37)
> The package name is actually  tfm-pulpcore-python3-aiohttp.

~]# rpm -q tfm-pulpcore-python3-aiohttp
tfm-pulpcore-python3-aiohttp-3.7.4-4.el7pc.x86_64

Comment 44 Odilon Sousa 2022-02-07 17:26:31 UTC
The line where we applied the patch changed from aiohttp 3.7.4 to 3.8.1, we will need to rewrite the patch

https://github.com/aio-libs/aiohttp/compare/v3.7.4...v3.8.1#diff-fa9062813a095af1b8f81504f078fedfc4440c1ffbcb29e4fc8ce6e713cf8cd9R81

Comment 48 addubey 2022-02-21 13:52:09 UTC
@dkliban should we move it back to the assigned state?

Comment 52 Stephen Wadeley 2022-03-10 13:48:30 UTC
(In reply to addubey from comment #48)
> @dkliban should we move it back to the assigned state?

yes

Comment 53 Grant Gainey 2022-04-05 18:21:09 UTC
*** Bug 2054148 has been marked as a duplicate of this bug. ***

Comment 55 addubey 2022-04-19 10:23:07 UTC
Hello Dennis, Though the aiothttp version has been updated to 3.8+ the content sync is failing.

rhel8 - rpm -q python38-aiohttp
python38-aiohttp-3.8.1-2.el8pc.x86_64

rhel7 - rpm -q tfm-pulpcore-python3-aiohttp
tfm-pulpcore-python3-aiohttp-3.8.1-2.el7pc.x86_64

Comment 58 Brian Bouterse 2022-04-19 13:58:33 UTC
It would be good to verify the SSL proxy is configured correctly. Please demonstrate that using curl commands similar to this one https://github.com/aio-libs/aiohttp/pull/5992#issuecomment-917174572

Also then please output the configuration of the Pulp remote that mimics the certs being handed to Pulp to use in a similar way.

Comment 66 Lukas Pramuk 2022-04-26 06:10:28 UTC
@

Comment 75 Evgeni Golov 2022-05-09 10:29:13 UTC
Finally found the Ruby upstream issue: https://bugs.ruby-lang.org/issues/16482

🤷

Comment 76 Evgeni Golov 2022-05-09 11:34:25 UTC
The Ruby issue does affect other Satellite releases too (I tested 6.8, but see no reason for it to be different on 6.9/6.10 either).

I'd argue this BZ is about "You can sync a repo via an HTTPS proxy" which was a regression in 6.10 (due to the switch to Pulp3) and this might as well work now with the new aiohttp and everything.
If Pulp3 can sync via an HTTP proxy, I'd mark this particular BZ as verified and track the issues in the Ruby part of the application in another BZ.

@bbuckingham what do you think?

Comment 77 Brad Buckingham 2022-05-12 11:41:05 UTC
I agree that it sounds like 1 issue has been resolved (pulp3/aiohttp) and another issue found (ruby).  If the original cause of the failure has been resolved, then I agree with moving to VERIFIED with another bugzilla being created to track the next issue (ruby).  In the new bugzilla, it needs to be clear as to the Satellite impact.  For example, if Satellite still unable to sync content with an http proxy (and it was in Satellite 6.9), it should remain a High severity regression.

Comment 79 Brad Buckingham 2022-05-17 12:24:44 UTC
What is the difference between an 'old' and 'new' proxy?

From the scenarios, 3 out of 4 fail.  Are all failures related to ruby issue?

Based upon the extent of the failures, I am ok with either of the 2 approaches:

1) Move this BZ to VERIFIED; however, state in the open the 1 passing scenario, 3 failed scenarios and that there is an additional issue to be addressed.  Use the BZ 'clone', to create the second bugzilla to track that issue.  The clone will ensure that customer cases follow the bugzilla.

2) Move this BZ to ASSIGNED/failedQA and re-assign the component appropriately to track the ruby issue.

Comment 80 Lukas Pramuk 2022-05-17 13:49:36 UTC
Please can we file a separate BZ for ruby ? 

I wouldn't introduce another complexity by authed / unauthed https proxy

Lets just stick to unauthed https proxy (= plain https functionality) , there can be filed another BZ for authed if needed - now you can't say when https doesn't work at all.

Comment 86 Evgeni Golov 2022-05-18 07:31:20 UTC
Yeah, kinda sure that's it:

https://github.com/pulp/pulpcore/blob/90f8643969421b5be3cd8922fd4d20166345b5ac/pulpcore/download/factory.py#L107-L109
        sslcontext = None
        if self._remote.ca_cert:
            sslcontext = ssl.create_default_context(cadata=self._remote.ca_cert)

That sslcontext is used for both, talking to the Proxy *and* the Remote, which fails if the two are signed by different CAs (which will be true for 99.9999999% of the setups)

Comment 87 Brian Bouterse 2022-05-18 13:58:54 UTC
Yes that one config is how to configure Pulp to trust the TLS connection from both the proxy and the endpoint. We probably should add additional separate configs, but at least for now it doesn't work like that. What you can do to work with that is append one cert to the other and it should work fine.

Comment 88 Evgeni Golov 2022-05-19 09:47:49 UTC
(In reply to Brian Bouterse from comment #87)
> Yes that one config is how to configure Pulp to trust the TLS connection
> from both the proxy and the endpoint. We probably should add additional
> separate configs, but at least for now it doesn't work like that. What you
> can do to work with that is append one cert to the other and it should work
> fine.

Right, but Katello feeds the "RedHat CDN cert" into that field, and the user has no way of influencing it.

https://github.com/Katello/katello/blob/9f4273cf9d853709eeb56b922b66ea815d4c1705/app/services/katello/pulp3/repository.rb#L397-L419

Comment 90 Grant Gainey 2022-06-06 18:19:55 UTC
@bmbouter catching up post-PTO - this looks like we need to figure out how to make this work for the satellite use-case? Or is this something that katello needs to handle, in the proxy case? Apologies, I'm not clear on the path forward here.

Comment 91 Brian Bouterse 2022-06-14 01:54:13 UTC
Either the certs should be concatenated in the single field, or PM in a future version of Pulp can ask for additional fields for the proxy versus end-site configuration.

Comment 93 Partha Aji 2022-07-29 21:01:32 UTC
(In reply to Evgeni Golov from comment #86)
> Yeah, kinda sure that's it:
> 
> https://github.com/pulp/pulpcore/blob/
> 90f8643969421b5be3cd8922fd4d20166345b5ac/pulpcore/download/factory.py#L107-
> L109
>         sslcontext = None
>         if self._remote.ca_cert:
>             sslcontext =
> ssl.create_default_context(cadata=self._remote.ca_cert)
> 
> That sslcontext is used for both, talking to the Proxy *and* the Remote,
> which fails if the two are signed by different CAs (which will be true for
> 99.9999999% of the setups)

Couldn't pulp include other ca certs from its certificate ca-trust store along with the remote cacert? That way we could have the user add the proxy ca certs in the ca trust store? 

I m guessing it did that in pulp2 and hence it worked fine in 6.9

Comment 95 Partha Aji 2022-08-04 17:41:18 UTC
Created a Pulp PR upstream to at least make pulp3 sync things using the system default trust store. This should at least put it on part with pulp2 behavior.
https://github.com/pulp/pulpcore/issues/3036

Comment 96 Grant Gainey 2022-08-09 14:55:27 UTC
Made as many comments public in this BZ as possible (modulo comments w/ machine-names or release-discussions). A reminder to all that there is upstream interest in this support, let's try to keep things as public as we can.


Note You need to log in before you can comment on or make changes to this bug.