Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1677765

Summary: subscription-manager commands fail when configured to use an ssl proxy server
Product: Red Hat Enterprise Linux 8 Reporter: John Sefler <jsefler>
Component: subscription-managerAssignee: Jiri Hnidek <jhnidek>
Status: CLOSED WONTFIX QA Contact: Red Hat subscription-manager QE Team <rhsm-qe>
Severity: medium Docs Contact:
Priority: low    
Version: 8.0CC: cdonnell, csnyder, jhnidek, khowell, redakkan, rjerrido
Target Milestone: rcKeywords: Reopened, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-01 07:32:49 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1673062    

Description John Sefler 2019-02-15 20:03:58 UTC
Description of problem:
As a result of the work done in Bug 1654531, the rhsm.conf file can now be configured to communicate over dnf to the CDN through an ssl proxy.  This was demonstrated in https://bugzilla.redhat.com/show_bug.cgi?id=1654531#c39 and some more below.

The problem now is that with the same ssl proxy configured and working to access CDN content, the subscription-manager commands are now failing to reach the entitlement server.  We can not register nor execute subscription-manager list --available commands.  We get a Traceback with a ConnectionResetError: [Errno 104] Connection reset by peer


Version-Release number of selected component (if applicable):
[root@kvm-01-guest19 ~]# rpm -q subscription-manager
subscription-manager-1.23.8-26.el8.x86_64


How reproducible:


Steps to Reproduce:
First you have to setup an ssl proxy server and fetch/trust the proxy's ssl cert.  (I won't show all of these steps, but I will fetch the proxy's ssl cert and configure rhsm.conf...

Starting with an unregistered RHEL system...

[root@kvm-01-guest19 ~]# subscription-manager config --server.proxy_hostname=auto-services.usersys.redhat.com --server.proxy_user=redhat --server.proxy_password=redhat --server.proxy_port=3131 --server.proxy_scheme=https

[root@kvm-01-guest19 ~]# scp root.redhat.com:/etc/squid/ssl_cert/squid_https_cert.pem  /etc/pki/ca-trust/source/anchors/squid_https_cert.pem
root.redhat.com's password: 
squid_https_cert.pem                          100% 3063   146.9KB/s   00:00   

[root@kvm-01-guest19 ~]# update-ca-trust extract

[root@kvm-01-guest19 ~]# subscription-manager register --serverurl=subscription.rhsm.stage.redhat.com:443/subscription --username=stage_auto_testuser --auto-attach
Unable to reach the server at subscription.rhsm.stage.redhat.com:443/subscription

[root@kvm-01-guest19 ~]# tail -f /var/log/rhsm/rhsm.log
2019-02-15 14:29:46,950 [INFO] subscription-manager:9376:MainThread @connection.py:924 - Connection built: http_proxy=auto-services.usersys.redhat.com:3131 host=subscription.rhsm.stage.redhat.com port=443 handler=/subscription auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
2019-02-15 14:29:46,951 [INFO] subscription-manager:9376:MainThread @connection.py:924 - Connection built: http_proxy=auto-services.usersys.redhat.com:3131 host=subscription.rhsm.stage.redhat.com port=443 handler=/subscription auth=none
2019-02-15 14:29:46,976 [DEBUG] subscription-manager:9376:MainThread @connection.py:543 - Loaded CA certificates from /etc/rhsm/ca/: redhat-uep.pem, redhat-entitlement-authority.pem
2019-02-15 14:29:46,976 [DEBUG] subscription-manager:9376:MainThread @connection.py:573 - Using proxy: auto-services.usersys.redhat.com:3131
2019-02-15 14:29:46,976 [DEBUG] subscription-manager:9376:MainThread @connection.py:591 - Making request: GET /subscription/status/
2019-02-15 14:29:47,020 [ERROR] subscription-manager:9376:MainThread @utils.py:160 - [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/utils.py", line 143, in is_valid_server_info
    conn.ping()
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 985, in ping
    return self.conn.request_get("/status/")
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 746, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 772, in _request
    info=info, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 603, in _request
    conn.request(request_type, handler, body=body, headers=final_headers)
  File "/usr/lib64/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/lib64/python3.6/http/client.py", line 1392, in connect
    super().connect()
  File "/usr/lib64/python3.6/http/client.py", line 940, in connect
    self._tunnel()
  File "/usr/lib64/python3.6/http/client.py", line 914, in _tunnel
    (version, code, message) = response._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer


BANG ^^^ registration fails


Temporarily changing to a non-ssl configure proxy port just to get past the registration...


[root@kvm-01-guest19 ~]# subscription-manager config --server.proxy_port=3127 --server.proxy_scheme=http

[root@kvm-01-guest19 ~]# subscription-manager register --serverurl=subscription.rhsm.stage.redhat.com:443/subscription --username=stage_auto_testuser --auto-attach
Registering to: subscription.rhsm.stage.redhat.com:443/subscription
Password: 
The system has been registered with ID: 3656ba39-2cdb-409c-93cb-49d8ebda9520
The registered system name is: kvm-01-guest19.lab.eng.rdu2.redhat.com
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for x86_64 High Touch Beta
Status:       Subscribed

Now changing back to the ssl configure proxy..

[root@kvm-01-guest19 ~]# subscription-manager config --server.proxy_port=3131 --server.proxy_scheme=https

[root@kvm-01-guest19 ~]# subscription-manager list --available
Traceback (most recent call last):
  File "/usr/sbin/subscription-manager", line 11, in <module>
    load_entry_point('subscription-manager==1.23.8', 'console_scripts', 'subscription-manager')()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/scripts/subscription_manager.py", line 85, in main
    return managercli.ManagerCLI().main()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 2932, in main
    ret = CLI.main(self)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/cli.py", line 183, in main
    return cmd.main()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 506, in main
    return_code = self._do_command()
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managercli.py", line 2587, in _do_command
    after_date=after_date,
  File "/usr/lib64/python3.6/site-packages/rhsmlib/services/entitlement.py", line 264, in get_available_pools
    after_date=after_date,
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managerlib.py", line 324, in get_available_entitlements
    overlapping, uninstalled, text, filter_string, future=future, after_date=after_date)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managerlib.py", line 530, in get_filtered_pools_list
    filter_string=filter_string, future=future, after_date=after_date):
  File "/usr/lib64/python3.6/site-packages/subscription_manager/managerlib.py", line 282, in list_pools
    profile_mgr.update_check(uep, consumer_uuid)
  File "/usr/lib64/python3.6/site-packages/subscription_manager/cache.py", line 452, in update_check
    if not uep.supports_resource(PACKAGES_RESOURCE):
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 950, in supports_resource
    self._load_supported_resources()
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 937, in _load_supported_resources
    resources_list = self.conn.request_get("/")
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 746, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 772, in _request
    info=info, headers=headers)
  File "/usr/lib64/python3.6/site-packages/rhsm/connection.py", line 603, in _request
    conn.request(request_type, handler, body=body, headers=final_headers)
  File "/usr/lib64/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib64/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib64/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/lib64/python3.6/http/client.py", line 1392, in connect
    super().connect()
  File "/usr/lib64/python3.6/http/client.py", line 940, in connect
    self._tunnel()
  File "/usr/lib64/python3.6/http/client.py", line 914, in _tunnel
    (version, code, message) = response._read_status()
  File "/usr/lib64/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

[root@kvm-01-guest19 ~]# tail -f /var/log/rhsm/rhsm.log 
2019-02-15 14:42:59,230 [DEBUG] subscription-manager:10007:MainThread @connection.py:543 - Loaded CA certificates from /etc/rhsm/ca/: redhat-uep.pem, redhat-entitlement-authority.pem
2019-02-15 14:42:59,231 [DEBUG] subscription-manager:10007:MainThread @connection.py:573 - Using proxy: auto-services.usersys.redhat.com:3131
2019-02-15 14:42:59,231 [DEBUG] subscription-manager:10007:MainThread @connection.py:591 - Making request: GET /subscription/

BANG ^^^ list --available pools fails

Yet I know the ssl proxy server is good, because the entitled repo reflects the same proxy configuration and successfully reads available package count from the cdn....


[root@kvm-01-guest19 ~]# grep rhel-8-for-x86_64-baseos-htb-rpms /etc/yum.repos.d/redhat.repo -A15
[rhel-8-for-x86_64-baseos-htb-rpms]
name = Red Hat Enterprise Linux 8 for x86_64 - BaseOS HTB (RPMs)
baseurl = https://cdn.stage.redhat.com/content/htb/rhel8/8/x86_64/baseos/os
enabled = 1
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta,file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
sslverify = 1
sslcacert = /etc/rhsm/ca/redhat-uep.pem
sslclientkey = /etc/pki/entitlement/4656323524632987744-key.pem
sslclientcert = /etc/pki/entitlement/4656323524632987744.pem
metadata_expire = 86400
enable_metadata = 1
proxy = https://auto-services.usersys.redhat.com:3131
proxy_username = redhat
proxy_password = redhat


[root@kvm-01-guest19 ~]# dnf repolist --disablerepo=beaker*
Updating Subscription Management repositories.
Red Hat Enterprise Linux 8 for x86_64 - AppStream HTB (RPMs)            1.9 MB/s |  14 MB     00:07    
Red Hat Enterprise Linux 8 for x86_64 - BaseOS HTB (RPMs)               1.9 MB/s | 8.4 MB     00:04    
Last metadata expiration check: 0:00:02 ago on Fri 15 Feb 2019 02:48:54 PM EST.
repo id                              repo name                                                    status
rhel-8-for-x86_64-appstream-htb-rpms Red Hat Enterprise Linux 8 for x86_64 - AppStream HTB (RPMs) 8,287
rhel-8-for-x86_64-baseos-htb-rpms    Red Hat Enterprise Linux 8 for x86_64 - BaseOS HTB (RPMs)    3,567


Actual results:
  above

Expected results:
  registration and listing available pools should succeed when configured with an ssl proxy

Additional info:

Comment 2 Craig Donnelly 2019-03-07 21:08:31 UTC
Testing:

I ruled out networking issues between my RHEL 8 RC1 host and the lab by utilizing a new host with a ncat listener that is hosted on the same hypervisor as the RHEL 8 host. (Difference between auto-services location for squid connection.)

RHEL 8 RC1 Client:

[root@unused rhsm]# rpm -q subscription-manager
subscription-manager-1.23.8-33.el8.x86_64

[root@unused rhsm]# subscription-manager config | grep proxy
   no_proxy = []
   proxy_hostname = 10.12.208.90
   proxy_password = redhat
   proxy_port = 4433
   proxy_scheme = https
   proxy_user = redhat

[root@unused rhsm]# subscription-manager register --serverurl=subscription.rhsm.stage.redhat.com:443/subscription --username=stage_auto_testuser --auto-attach
<hangs waiting for response from ncat>
Unable to reach the server at subscription.rhsm.stage.redhat.com:443/subscription



=====================================================

Fedora 28 Listen Server:

[root@qedocs ~]# ncat -k -l 4433
CONNECT subscription.rhsm.stage.redhat.com:443 HTTP/1.0    <<<------------------ Not HTTPS...
User-Agent: RHSM/1.0 (cmd=subscription-manager)
Host: subscription.rhsm.stage.redhat.com:443
Proxy-Authorization: Basic cmVkaGF0OnJlZGhhdA==

no <me killing connection>

=====================================================

The same results from the connection to ncat listener with 'https_proxy' set on the RHEL 8 Client.

It seems that the proxy_scheme is not having an effect on the connection type or something fundamental about RHEL 8 has changed that is not allowing us to create an https based connection via python.

Comment 6 Jiri Hnidek 2019-09-11 12:52:26 UTC
Closed after discussion with Chris.

Comment 7 Kevin Howell 2019-10-01 18:17:17 UTC
I looked into this with the assistance of cdonnell, and we determined that although support is limited to newer versions of specific programs, curl at least supports the configuration where the proxy server uses TLS for the initial connection and the proxied resource is TLS as well. Specifically, HTTP CONNECT proxy where the proxy connection is through TLS.

Unfortunately, this scenario is not well supported within Python itself (e.g. see https://bugs.python.org/issue29394). `http.client` assumes that the proxy connection is plain HTTP. I did a bit of POC-work and determined that it is *possible* to work-around by overriding the initial proxy connection within http.client to use an SSL-wrapped socket. However, there is added complexity as communication with the proxied server also needs TLS, and so it's necessary to use the openssl bio interfaces with `wrap_bio` (trying to `wrap_socket` an already wrapped socket doesn't work, as the same socket is then referenced by two different SSL contexts -> bad time). In the POC, I worked around by emulating a socket (including the `makefile` function) with `ssl.SSLObject` (backed by `wrap_bio`); see https://github.com/candlepin/subscription-manager/tree/khowell/ssl_proxy_support_poc

Another possible option would be to use pycurl, as curl seems to have implemented this in https://github.com/curl/curl/commit/cb4e2be7c6d42ca0780f8e0a747cecf9ba45f151 , but this has some additional downstream complexities involving moving pycurl from AppStream to BaseOS, and adding another dependency to subscription-manager.

Comment 12 RHEL Program Management 2021-02-01 07:32:49 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.