Bug 1434876
| Summary: | subscription-manager does not work behind cascaded proxies | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Akash <asakpal> |
| Component: | subscription-manager | Assignee: | candlepin-bugs |
| Status: | CLOSED WONTFIX | QA Contact: | John Sefler <jsefler> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.3 | CC: | bkearney, candlepin-bugs, khowell, redakkan, skallesh |
| Target Milestone: | rc | Keywords: | Reopened, Triaged |
| Target Release: | 7.3 | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-04-23 21:30:00 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
We will take a look at the Host header, and try to fix any bad behavior on the behalf of subscription-manager. It is worth noting that as of RHEL 7.4, the HTTPs functionality will be using python standard libraries, and not M2Crypto, so the behavior may change (hopefully for the better :-)) Though RHEL 6.* still uses M2Crypto. However, the use case described is not one that we can support, because, there are multiple client certificates at play for a given system (ex. client ID cert to RHSM, entitlement certs for CDN access), and these are unique for each system. If the proxy is configured to terminate SSL and is configured to use one of these certificates, it will appear to RHSM as if one system is connecting, rather than the full number of systems behind the proxy, and consequently rate limiting will happen. There are other considerations that are difficult to address (ex. rhsmcertd on the system will update these certificates as necessary), which complicates trying to keep a proxy configured with the right certificates. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: subscription-manager talks to internal HTTP(s) proxy. subscription-manager uses, as it always does, TLS client cert authentication. subscription-manager also performs certificate validation. Customer is intercepting proxy re-encrypts the traffic. Therefore, subscription-manager is configured to check the provided server certificate against our site CA instead of Red Hat certificate. Customer proxy server is configured to accept Red Hat's own CA for the relevant RHSM hostnames and does not perform the usual Browser-like public CA validation, but validates only this single CA. Also, our proxy is provided with the necessary client certificate. Therefore, when a client (subscription-manager) contacts the proxy, the proxy will verify that the client possesses the proper certificate. If it does, then the proxy will authenticate to RHSM using its own copy of the client certificate. How reproducible: 100% reproducible Steps to Reproduce: - yum install subscription-manager squid httpd mod_ssl - make apache return HTTP 403: echo $'<Location />\nDeny from all\n</Location>\n' > /etc/httpd/conf.d/welcome.conf - configure subscription-manager to use this setup instead of RHN # subscription-manager config --server.hostname=127.0.0.99 # subscription-manager config --server.proxy_hostname=127.0.0.11 # subscription-manager config --server.proxy_port=3128 # subscription-manager config --server.insecure=1 - watch it fail # subscription-manager register --user foo --password bar Registering to: 127.0.0.99:443/subscription Forbidden: Invalid credentials for request. - further details can be analyzed with the following hack: by adding 'print data' to /usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py:221 we can intercept the unencrypted HTTP traffic: GET /subscription/users/foo/owners HTTP/1.1 Host: 127.0.0.11:3128 Accept-Encoding: identity Content-Length: 0 Accept-Language: en-us x-subscription-manager-version: 1.17.15-1.el7.centos x-python-rhsm-version: 1.17.9-1.el7 Accept: application/json User-Agent: RHSM/1.0 (cmd=subscription-manager) Content-type: application/json Authorization: Basic Zm9vOmJhcg== This shows that the request contains a Host: header with the proxy's name/address. At this place, the hostname of the target address is (127.0.0.99 ) is supposed to show up. I assume that this issue technically affects all customers using subscription-manager. However, when there is only one proxy in-between, the only side effect will be that you will see unexpected Host: headers on the web server at subscription.rhsm.redhat.com. In other words, this would break for all proxy-using customers if you decided to enable named-based virtual hosting on this server. Here, subscription-manager will send a CONNECT verb to our first proxy and will embed the above-cited request. First proxy is configured to send the request to a second proxy for content security reasons. Customer assume that at least one of these proxies has to rely on the request to contain a proper Host: statement. However, the request generated by subscription-manager does not contain a sane Host: header. Therefore, the request ends up being served (instead of forwarded) by our proxy. The proxy will of course refuse this request with HTTP status code 403. This leads to subscription-manager exiting with an error. Due to this bug, Customer is unable to use subscription-manager with RHN. Customer believes that there are two code issues: (1) The invoking code in rhsm.connection only sometimes adds a (correct) Host: header. For example, when running 'subscription-manager register', no Host: header is added. This part could probably be fixed as part of the subscription-manager / python rhsm package. (2) In any case, some code in-between (I assume M2Crypto httplib wrapping code) always adds a wrong Host: header containing the address of the used proxy. This part would probably need a change in M2Crypto. Actual results: Peer's Certificate issuer is not recognized" Expected results: Systems should be able to get registered without any issues Additional info: Adding a pdb breakpoint will show that the actual introduction of the Host: header happens in python's standard httplib: # subscription-manager register --user foo --password bar Registering to: 127.0.0.99:443/subscription > /usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py(179)putheader() -> if header.lower() == self._UA_HEADER.lower(): (Pdb) print header, value Host 127.0.0.11:3128 (Pdb) bt /usr/sbin/subscription-manager(81)<module>() -> sys.exit(abs(main() or 0)) /usr/sbin/subscription-manager(72)main() -> return managercli.ManagerCLI().main() /usr/lib/python2.7/site-packages/subscription_manager/managercli.py(2744)main() -> return CLI.main(self) /usr/lib/python2.7/site-packages/subscription_manager/cli.py(160)main() -> return cmd.main() /usr/lib/python2.7/site-packages/subscription_manager/managercli.py(526)main() -> return_code = self._do_command() /usr/lib/python2.7/site-packages/subscription_manager/managercli.py(1126)_do_command() -> owner_key = self._determine_owner_key(admin_cp) /usr/lib/python2.7/site-packages/subscription_manager/managercli.py(1271)_determine_owner_key() -> owners = cp.getOwnerList(self.username) /usr/lib64/python2.7/site-packages/rhsm/connection.py(1126)getOwnerList() -> return self.conn.request_get(method) /usr/lib64/python2.7/site-packages/rhsm/connection.py(694)request_get() -> return self._request("GET", method) /usr/lib64/python2.7/site-packages/rhsm/connection.py(584)_request() -> conn.request(request_type, handler, body=body, headers=headers) /usr/lib64/python2.7/httplib.py(973)request() -> self._send_request(method, url, body, headers) /usr/lib64/python2.7/httplib.py(1001)_send_request() -> self.putrequest(method, url, **skips) /usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py(174)putrequest() -> HTTPSConnection.putrequest(self, method, rest, skip_host, skip_accept_encoding) /usr/lib64/python2.7/httplib.py(920)putrequest() -> self.putheader('Host', "%s:%s" % (host_enc, self.port)) > /usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py(179)putheader() -> if header.lower() == self._UA_HEADER.lower(): (Pdb) When simulation the subscription-manager HTTP traffic using curl, everything works as expected, as curl adds the correct Host: header. The following patch makes the subscription-manager register call succeed against RHN. I believe that it might have unwanted side-effects for other M2Crypto user. The proper fix might have to be done in some subscription-manager-specific code path. --- /usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py.orig 2017-02-01 22:54:55.827000000 +0100 +++ /usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py 2017-02-01 22:55:31.089000000 +0100 @@ -142,7 +142,7 @@ self._proxy_auth = None self._proxy_UA = None - def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0): + def putrequest(self, method, url, skip_host=1, skip_accept_encoding=0): #putrequest is called before connect, so can interpret url and get #real host/port to be used to make CONNECT request to proxy proto, netloc, path, query, fragment = urlsplit(url) @@ -172,6 +172,7 @@ HTTPSConnection.putrequest(self, method, rest, skip_host) else: HTTPSConnection.putrequest(self, method, rest, skip_host, skip_accept_encoding) + self.putheader('Host', netloc) def putheader(self, header, value): # Store the auth header if passed in. Customer have resolved all certificate-related issues in our real-world setup (or rather, our Proxy Team has). Customer's setup uses features which are officially provided by our Proxy vendor. Basically, subscription-manager talks to our internal HTTP(s) proxy. subscription-manager uses, as it always does, TLS client cert authentication. subscription-manager also performs certificate validation. In our case, our intercepting proxy re-encrypts the traffic. Therefore, subscription-manager is configured to check the provided server certificate against our site CA instead of Red Hat's. Customer's proxy server is configured to accept Red Hat's own CA für the relevant RHSM hostnames and does not perform the usual Browser-like public CA validation, but validates only this single CA. Also, our proxy is provided with the necessary client certificate. Therefore, when a client (subscription-manager) contacts the proxy, the proxy will verify that the client possesses the proper certificate. If it does, then the proxy will authenticate to RHSM using its own copy of the client certificate. All in all, everything TLS-related works. Our issue is that subscription-manager generates semantically invalid HTTP requests, which are sent using this TLS connection. The wrong content (more specific, the wrong Host: header) makes it impossible for a proxy to forward the request to the correct destination (RHSM). Instead, the request ends up being targeted at our proxy itself!