Bug 1993917

Log trace :

pulp_tasks:
- pulp_href: "/pulp/api/v3/tasks/c3aadbfe-2b3d-449d-8e65-9666951476d2/"
  pulp_created: '2021-08-16T11:20:14.160+00:00'
  state: failed
  name: pulp_rpm.app.tasks.synchronizing.synchronize
  logging_cid: 0f189218-3ac5-4a3f-b183-0d6b916aa4f2
  started_at: '2021-08-16T11:20:14.249+00:00'
  finished_at: '2021-08-16T11:20:14.374+00:00'
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 272, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 363, in synchronize
          remote_url = fetch_remote_url(remote)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 242, in fetch_remote_url
          get_repomd_file(remote, normalized_remote_url)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 200, in get_repomd_file
          return downloader.fetch()
        File "/usr/lib/python3.6/site-packages/pulpcore/download/base.py", line 176, in fetch
          return done.pop().result()
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 258, in run
          return await download_wrapper()
        File "/usr/lib/python3.6/site-packages/backoff/_async.py", line 133, in retry
          ret = await target(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulpcore/download/http.py", line 256, in download_wrapper
          return await self._run(extra_data=extra_data)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py", line 92, in _run
          url, proxy=self.proxy, proxy_auth=self.proxy_auth, auth=self.auth
        File "/usr/lib64/python3.6/site-packages/aiohttp/client.py", line 1117, in __aenter__
          self._resp = await self._coro
        File "/usr/lib64/python3.6/site-packages/aiohttp/client.py", line 513, in _request
          traces=traces,
        File "/usr/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 311, in __init__
          self.update_proxy(proxy, proxy_auth, proxy_headers)
        File "/usr/lib64/python3.6/site-packages/aiohttp/client_reqrep.py", line 551, in update_proxy
          raise ValueError("Only http proxies are supported")
    description: Only http proxies are supported
  worker: "/pulp/api/v3/workers/204569e0-9f35-48ca-9e22-5cbfd8828dcc/"
  child_tasks: []
  progress_reports: []
  created_resources: []
  reserved_resources_record:
  - "/pulp/api/v3/repositories/rpm/rpm/abc9dd0c-d198-433c-af59-47b40de82db6/"
  - "/pulp/api/v3/remotes/rpm/rpm/d86a3ede-74f5-4f11-b979-34e9f0e6dd27/"
create_version: true
task_groups: []
poll_attempts:
  total: 1
  failed: 1

The underlying library aiohttp doesn't support https proxies, specified with https://.
https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support
"aiohttp supports plain HTTP proxies and HTTP proxies that can be upgraded to HTTPS via the HTTP CONNECT method. aiohttp does not support proxies that must be connected to via https://."

HTTP CONNECT method https://www.ietf.org/rfc/rfc2817.txt.

Was it supported in Satellite 6.9?

(In reply to Tanya Tereshchenko from comment #2)
> The underlying library aiohttp doesn't support https proxies, specified with
> https://.
> https://docs.aiohttp.org/en/stable/client_advanced.html#proxy-support
> "aiohttp supports plain HTTP proxies and HTTP proxies that can be upgraded
> to HTTPS via the HTTP CONNECT method. aiohttp does not support proxies that
> must be connected to via https://."
> 
> HTTP CONNECT method https://www.ietf.org/rfc/rfc2817.txt.
> 
> Was it supported in Satellite 6.9?

Yes , Https proxy worked without issues on satellite 6.9

The Pulp upstream bug status is at NEW. Updating the external tracker on this bug.

The Pulp upstream bug priority is at Normal. Updating the external tracker on this bug.

The Pulp upstream bug status is at ASSIGNED. Updating the external tracker on this bug.

The Pulp upstream bug priority is at High. Updating the external tracker on this bug.

The Pulp upstream bug status is at CLOSED - NOTABUG. Updating the external tracker on this bug.

I found that simply using `https_port 3128 cert=/etc/squid/ssl_cert/myCA.pem` and disabling `http_port 3128` reliably produces a proxy that can only be accessed via HTTPS, so unless anyone has any objections, I'm going to go with that for my example.  Here's my full config for the HTTPS proxy (Squid 3.5 on EL7):

```
[root@rhel7 ssl_cert]# cat /etc/squid/squid.conf
#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 10.0.0.0/8	# RFC1918 possible internal network
acl localnet src 172.16.0.0/12	# RFC1918 possible internal network
acl localnet src 192.168.0.0/16	# RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http
acl SSL_ports port 80		# http
acl SSL_ports port 21		# ftp
acl SSL_ports port 443		# https
acl SSL_ports port 70		# gopher
acl SSL_ports port 210		# wais
acl SSL_ports port 1025-65535	# unregistered ports
acl SSL_ports port 280		# http-mgmt
acl SSL_ports port 488		# gss-http
acl SSL_ports port 591		# filemaker
acl SSL_ports port 777		# multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
#http_port 3128
https_port 3128 cert=/etc/squid/ssl_cert/myCA.pem

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /var/spool/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern .		0	20%	4320

```

I looked at the comment 24 and it all seems fine. Can someone confirm the version of Python and aiohttp on this system?

(In reply to addubey from comment #28)
> Python - 2.7.5
> 
> Metadata-Version: 2.1
> Name: aiohttp
> Version: 3.7.4
> Summary: Async http client/server framework (asyncio)

Pulp3 uses Python3 - what's the pyhton3 version?

Thanks addubey. This explains why it isn't working then. Aiohttp carries the secure proxy feature, and I believe it is only in 3.8.z. Additionally my understanding is that it requires Python 3.8 in order to receive the TLS fixes. We should confirm the versions Satellite needs with ssydoren.

Python3  - 3.6.8

(In reply to Dennis Kliban from comment #37)
> The package name is actually  tfm-pulpcore-python3-aiohttp.

~]# rpm -q tfm-pulpcore-python3-aiohttp
tfm-pulpcore-python3-aiohttp-3.7.4-4.el7pc.x86_64

The line where we applied the patch changed from aiohttp 3.7.4 to 3.8.1, we will need to rewrite the patch

https://github.com/aio-libs/aiohttp/compare/v3.7.4...v3.8.1#diff-fa9062813a095af1b8f81504f078fedfc4440c1ffbcb29e4fc8ce6e713cf8cd9R81

@dkliban should we move it back to the assigned state?

(In reply to addubey from comment #48)
> @dkliban should we move it back to the assigned state?

yes

*** Bug 2054148 has been marked as a duplicate of this bug. ***

Hello Dennis, Though the aiothttp version has been updated to 3.8+ the content sync is failing.

rhel8 - rpm -q python38-aiohttp
python38-aiohttp-3.8.1-2.el8pc.x86_64

rhel7 - rpm -q tfm-pulpcore-python3-aiohttp
tfm-pulpcore-python3-aiohttp-3.8.1-2.el7pc.x86_64

It would be good to verify the SSL proxy is configured correctly. Please demonstrate that using curl commands similar to this one https://github.com/aio-libs/aiohttp/pull/5992#issuecomment-917174572

Also then please output the configuration of the Pulp remote that mimics the certs being handed to Pulp to use in a similar way.

@

Finally found the Ruby upstream issue: https://bugs.ruby-lang.org/issues/16482

🤷

The Ruby issue does affect other Satellite releases too (I tested 6.8, but see no reason for it to be different on 6.9/6.10 either).

I'd argue this BZ is about "You can sync a repo via an HTTPS proxy" which was a regression in 6.10 (due to the switch to Pulp3) and this might as well work now with the new aiohttp and everything.
If Pulp3 can sync via an HTTP proxy, I'd mark this particular BZ as verified and track the issues in the Ruby part of the application in another BZ.

@bbuckingham what do you think?

I agree that it sounds like 1 issue has been resolved (pulp3/aiohttp) and another issue found (ruby).  If the original cause of the failure has been resolved, then I agree with moving to VERIFIED with another bugzilla being created to track the next issue (ruby).  In the new bugzilla, it needs to be clear as to the Satellite impact.  For example, if Satellite still unable to sync content with an http proxy (and it was in Satellite 6.9), it should remain a High severity regression.

What is the difference between an 'old' and 'new' proxy?

From the scenarios, 3 out of 4 fail.  Are all failures related to ruby issue?

Based upon the extent of the failures, I am ok with either of the 2 approaches:

1) Move this BZ to VERIFIED; however, state in the open the 1 passing scenario, 3 failed scenarios and that there is an additional issue to be addressed.  Use the BZ 'clone', to create the second bugzilla to track that issue.  The clone will ensure that customer cases follow the bugzilla.

2) Move this BZ to ASSIGNED/failedQA and re-assign the component appropriately to track the ruby issue.

Please can we file a separate BZ for ruby ? 

I wouldn't introduce another complexity by authed / unauthed https proxy

Lets just stick to unauthed https proxy (= plain https functionality) , there can be filed another BZ for authed if needed - now you can't say when https doesn't work at all.

Yeah, kinda sure that's it:

https://github.com/pulp/pulpcore/blob/90f8643969421b5be3cd8922fd4d20166345b5ac/pulpcore/download/factory.py#L107-L109
        sslcontext = None
        if self._remote.ca_cert:
            sslcontext = ssl.create_default_context(cadata=self._remote.ca_cert)

That sslcontext is used for both, talking to the Proxy *and* the Remote, which fails if the two are signed by different CAs (which will be true for 99.9999999% of the setups)

Yes that one config is how to configure Pulp to trust the TLS connection from both the proxy and the endpoint. We probably should add additional separate configs, but at least for now it doesn't work like that. What you can do to work with that is append one cert to the other and it should work fine.

(In reply to Brian Bouterse from comment #87)
> Yes that one config is how to configure Pulp to trust the TLS connection
> from both the proxy and the endpoint. We probably should add additional
> separate configs, but at least for now it doesn't work like that. What you
> can do to work with that is append one cert to the other and it should work
> fine.

Right, but Katello feeds the "RedHat CDN cert" into that field, and the user has no way of influencing it.

https://github.com/Katello/katello/blob/9f4273cf9d853709eeb56b922b66ea815d4c1705/app/services/katello/pulp3/repository.rb#L397-L419

@bmbouter catching up post-PTO - this looks like we need to figure out how to make this work for the satellite use-case? Or is this something that katello needs to handle, in the proxy case? Apologies, I'm not clear on the path forward here.

Either the certs should be concatenated in the single field, or PM in a future version of Pulp can ask for additional fields for the proxy versus end-site configuration.

(In reply to Evgeni Golov from comment #86)
> Yeah, kinda sure that's it:
> 
> https://github.com/pulp/pulpcore/blob/
> 90f8643969421b5be3cd8922fd4d20166345b5ac/pulpcore/download/factory.py#L107-
> L109
>         sslcontext = None
>         if self._remote.ca_cert:
>             sslcontext =
> ssl.create_default_context(cadata=self._remote.ca_cert)
> 
> That sslcontext is used for both, talking to the Proxy *and* the Remote,
> which fails if the two are signed by different CAs (which will be true for
> 99.9999999% of the setups)

Couldn't pulp include other ca certs from its certificate ca-trust store along with the remote cacert? That way we could have the user add the proxy ca certs in the ca trust store? 

I m guessing it did that in pulp2 and hence it worked fine in 6.9

Created a Pulp PR upstream to at least make pulp3 sync things using the system default trust store. This should at least put it on part with pulp2 behavior.
https://github.com/pulp/pulpcore/issues/3036

Made as many comments public in this BZ as possible (modulo comments w/ machine-names or release-discussions). A reminder to all that there is upstream interest in this support, let's try to keep things as public as we can.

My current understanding of this issue is - and please correct me if I am wrong, it is effectively blocked on Python 3.11.  As such I'm going to remove the current flags, because it doesn't seem that we could fix it in the current releases.

Bulk setting Target Milestone = 6.15.0 where sat-6.15.0+ is set.

Created attachment 2015212 [details]
manifest refresh issue

Tested on Satellite 6.15.0 Snap 8.0

Tried this with both authenticated and un-authenticated proxies, on Satellite I had the manifest imported and repos enabled, after that when https proxy is set for setting content_default_http_proxy/http_proxy and updated enabled repos to use specific https proxies then repository sync works with https proxy as expected for this BZ.

But, I ran into condition where I can't refresh the manifest when https proxy is set for setting content_default_http_proxy/http_proxy and when I set normal http proxy then refresh button appears and refresh works.

It's unclear to me whether this is a regression or a known issue with https, so I was wondering if we should move it back to ASSIGNED state?

Is it the result of a failure on the Pulp side (bubbling up indirectly - since Pulp doesn't deal with manifests at all), or does Katello perform some additional validation?  If it's the latter then we shouldn't move this back to ASSIGNED, we'd need to file a new BZ for the Katello portion of the BZ. 

@iballou wdyt?

Daniel is correct here. Manifest refresh is not connected to pulp. We should open a new bug. The main point of this issue is ensure sync works ok if you appended the correct certs to the repo. 

"""

Tried this with both authenticated and un-authenticated proxies, on Satellite I had the manifest imported and repos enabled, after that when https proxy is set for setting content_default_http_proxy/http_proxy and updated enabled repos to use specific https proxies then repository sync works with https proxy as expected for this BZ.
"""
Indicates to me that it is working correctly.

Sure, I've opened a new BZ to track it separately and CC'ed you there https://bugzilla.redhat.com/show_bug.cgi?id=2263159

Therefore, I think we're good to verify this BZ as repository sync works with https_proxies for both authenticated and un-authenticated proxies as expected with pulpcore-3.39.7-1

Also, I hope you don't mind me making above comments public, because I think this part should be openly discussed and considered as known limitation for 6.15.0
 
Thank you for confirming it and for your assistance.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:2010

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days