Description of problem:
subscription-manager talks to internal HTTP(s) proxy. subscription-manager uses, as it always does, TLS client cert authentication. subscription-manager also performs certificate validation. Customer is intercepting proxy re-encrypts the traffic. Therefore, subscription-manager is configured to check the provided server certificate against our site CA instead of Red Hat certificate.

Customer proxy server is configured to accept Red Hat's own CA for the relevant RHSM hostnames and does not perform the usual Browser-like public CA validation, but validates only this single CA. Also, our proxy is provided with the necessary client certificate.

Therefore, when a client (subscription-manager) contacts the proxy, the proxy will verify that the client possesses the proper certificate. If it does, then the proxy will authenticate to RHSM using its own copy of the client certificate. 

How reproducible:
100% reproducible

Steps to Reproduce:
- yum install subscription-manager squid httpd mod_ssl
- make apache return HTTP 403:
  echo $'<Location />\nDeny from all\n</Location>\n' > /etc/httpd/conf.d/welcome.conf
- configure subscription-manager to use this setup instead of RHN
        # subscription-manager config --server.hostname=127.0.0.99
        # subscription-manager config --server.proxy_hostname=127.0.0.11
        # subscription-manager config --server.proxy_port=3128
        # subscription-manager config --server.insecure=1
- watch it fail
        # subscription-manager register  --user foo --password bar
        Registering to: 127.0.0.99:443/subscription
        Forbidden: Invalid credentials for request.

- further details can be analyzed with the following hack:
  by adding 'print data' to /usr/lib64/python2.7/site-packages/M2Crypto/SSL/Connection.py:221
  we can intercept the unencrypted HTTP traffic:

       GET /subscription/users/foo/owners HTTP/1.1
        Host: 127.0.0.11:3128
        Accept-Encoding: identity
        Content-Length: 0
        Accept-Language: en-us
        x-subscription-manager-version: 1.17.15-1.el7.centos
        x-python-rhsm-version: 1.17.9-1.el7
        Accept: application/json
        User-Agent: RHSM/1.0 (cmd=subscription-manager)
        Content-type: application/json
        Authorization: Basic Zm9vOmJhcg==

This shows that the request contains a Host: header with the proxy's name/address.
  At this place, the hostname of the target address is (127.0.0.99 ) is
  supposed to show up.

  I assume that this issue technically affects all customers using subscription-manager.
  However, when there is only one proxy in-between, the only side effect will be that
  you will see unexpected Host: headers on the web server at subscription.rhsm.redhat.com.
  In other words, this would break for all proxy-using customers if you decided to enable named-based
  virtual hosting on this server.

  Here, subscription-manager will send a CONNECT verb to our first proxy and will embed the above-cited request.
  First proxy is configured to send the request to a second proxy for content security reasons.
  Customer assume that at least one of these proxies has to rely on the request to contain a proper Host: statement.
  However, the request generated by subscription-manager does not contain a sane Host: header.
  Therefore, the request ends up being served (instead of forwarded) by our proxy. The proxy will of course
  refuse this request with HTTP status code 403. This leads to subscription-manager exiting with an error.

  Due to this bug, Customer is unable to use subscription-manager with RHN.

  Customer believes that there are two code issues:
  (1) The invoking code in rhsm.connection only sometimes adds a (correct) Host: header. For example, when running
  'subscription-manager register', no Host: header is added.
  This part could probably be fixed as part of the subscription-manager / python rhsm package.

  (2) In any case, some code in-between (I assume M2Crypto httplib wrapping code) always adds a wrong Host: header
  containing the address of the used proxy.
  This part would probably need a change in M2Crypto.

Actual results:
Peer's Certificate issuer is not recognized"

Expected results:
Systems should be able to get registered without any issues

Additional info:
Adding a pdb breakpoint will show that the actual introduction of the Host: header happens in python's standard httplib:

# subscription-manager register  --user foo --password bar
Registering to: 127.0.0.99:443/subscription
> /usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py(179)putheader()
-> if header.lower() == self._UA_HEADER.lower():
(Pdb) print header, value
Host 127.0.0.11:3128
(Pdb) bt
  /usr/sbin/subscription-manager(81)<module>()
-> sys.exit(abs(main() or 0))
  /usr/sbin/subscription-manager(72)main()
-> return managercli.ManagerCLI().main()
  /usr/lib/python2.7/site-packages/subscription_manager/managercli.py(2744)main()
-> return CLI.main(self)
  /usr/lib/python2.7/site-packages/subscription_manager/cli.py(160)main()
-> return cmd.main()
  /usr/lib/python2.7/site-packages/subscription_manager/managercli.py(526)main()
-> return_code = self._do_command()
  /usr/lib/python2.7/site-packages/subscription_manager/managercli.py(1126)_do_command()
-> owner_key = self._determine_owner_key(admin_cp)
  /usr/lib/python2.7/site-packages/subscription_manager/managercli.py(1271)_determine_owner_key()
-> owners = cp.getOwnerList(self.username)
  /usr/lib64/python2.7/site-packages/rhsm/connection.py(1126)getOwnerList()
-> return self.conn.request_get(method)
  /usr/lib64/python2.7/site-packages/rhsm/connection.py(694)request_get()
-> return self._request("GET", method)
  /usr/lib64/python2.7/site-packages/rhsm/connection.py(584)_request()
-> conn.request(request_type, handler, body=body, headers=headers)
  /usr/lib64/python2.7/httplib.py(973)request()
-> self._send_request(method, url, body, headers)
  /usr/lib64/python2.7/httplib.py(1001)_send_request()
-> self.putrequest(method, url, **skips)
  /usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py(174)putrequest()
-> HTTPSConnection.putrequest(self, method, rest, skip_host, skip_accept_encoding)
  /usr/lib64/python2.7/httplib.py(920)putrequest()
-> self.putheader('Host', "%s:%s" % (host_enc, self.port))
> /usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py(179)putheader()
-> if header.lower() == self._UA_HEADER.lower():
(Pdb)

When simulation the subscription-manager HTTP traffic using curl, everything works as expected,
as curl adds the correct Host: header.

The following patch makes the subscription-manager register call succeed against RHN.
I believe that it might have unwanted side-effects for other M2Crypto user.
The proper fix might have to be done in some subscription-manager-specific
code path.

--- /usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py.orig        2017-02-01 22:54:55.827000000 +0100
+++ /usr/lib64/python2.7/site-packages/M2Crypto/httpslib.py     2017-02-01 22:55:31.089000000 +0100
@@ -142,7 +142,7 @@
         self._proxy_auth = None
         self._proxy_UA = None

-    def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0):
+    def putrequest(self, method, url, skip_host=1, skip_accept_encoding=0):
         #putrequest is called before connect, so can interpret url and get
         #real host/port to be used to make CONNECT request to proxy
         proto, netloc, path, query, fragment = urlsplit(url)
@@ -172,6 +172,7 @@
             HTTPSConnection.putrequest(self, method, rest, skip_host)
         else:
             HTTPSConnection.putrequest(self, method, rest, skip_host, skip_accept_encoding)
+       self.putheader('Host', netloc)

     def putheader(self, header, value):
         # Store the auth header if passed in.

Customer have resolved all certificate-related issues in our real-world setup (or rather, our Proxy Team has). Customer's setup uses features which are officially provided by our Proxy vendor.

Basically, subscription-manager talks to our internal HTTP(s) proxy. subscription-manager uses, as it always does, TLS client cert authentication. subscription-manager also performs certificate validation. In our case, our intercepting proxy re-encrypts the traffic. Therefore, subscription-manager is configured to check the provided server certificate against our site CA instead of Red Hat's.
Customer's proxy server is configured to accept Red Hat's own CA für the relevant RHSM hostnames and does not perform the usual Browser-like public CA validation, but validates only this single CA. Also, our proxy is provided with the necessary client certificate.
Therefore, when a client (subscription-manager) contacts the proxy, the proxy will verify that the client possesses the proper certificate. If it does, then the proxy will authenticate to RHSM using its own copy of the client certificate.
All in all, everything TLS-related works.

Our issue is that subscription-manager generates semantically invalid HTTP requests, which are sent using this TLS connection. The wrong content (more specific, the wrong Host: header) makes it impossible for a proxy to forward the request to the correct destination (RHSM). Instead, the request ends up being targeted at our proxy itself!