Description of problem: With 2.3 the syncing of RHN Repositories via our company Proxy stopped working. Version-Release number of selected component (if applicable): RHEL 6.5 Pulp 2.3 How reproducible: Create Pulprepository with feed from RHN via Proxy. Start sync. Steps to Reproduce: 1. pulp-admin rpm repo create --repo-id=rhel-6-server-cf-tools-1-rpms \ --feed=https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/cf-tools/1/os \ --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem \ --feed-cert=/etc/pki/entitlement/XXXXX.pem \ --feed-key=/etc/pki/entitlement/XXXX-key.pem \ --description="Red Hat CloudForms Tools for RHEL 6 (RPMs)" \ --serve-http=True \ --proxy-host=http://proxy.mycompany.com \ --proxy-port=3128 \ --retain-old-count=1 \ --remove-missing=True \ --validate=True 2. pulp-admin rpm repo sync run --repo-id=rhel-6-server-cf-tools-1-rpms Actual results: Sync does not work. /var/log/pulp/nectar.log: 2013-12-09 09:18:49,226 requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com 2013-12-09 09:18:49,758 nectar.downloaders.threaded:ERROR: Cannot connect to proxy. Socket error: [Errno 104] Connection reset by peer. Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/nectar/downloaders/threaded.py", line 161, in _fetch response = session.get(request.url, headers=headers) File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 369, in get return self.request('GET', url, **kwargs) File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 357, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 460, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 319, in send timeout=timeout File "/usr/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py", line 583, in urlopen 'Socket error: %s.' % e) ProxyError: Cannot connect to proxy. Socket error: [Errno 104] Connection reset by peer. Expected results: Repository should sync. Additional info: * Worked with 2.1 * Does neither not work with "migrated" Repositories from 2.1 to 2.3 nor with new created ones * Non-RHN Repositories (like EPEL) work
We are not able to reproduce this problem. Can you tell us more about the proxy server itself? Are you using any kind of auth with it? It looks like the proxy server is resetting the TCP connection between itself and pulp, so perhaps it has some log output explaining why.
Our proxy is a Bluecoat and we do not use authentication. I tried with tinyproxy in a special environment an can confirm that it works with tiny proxy. I tried to investigate and think it is somehow connected to urllib3's connectionpooling... Our Proxyadmin will return tomorrow and I hope we find some time to get around the problem. I will keep this bugreport updated..
TLDR: I think the Bluecoat does not support TLS 1.2 Clients correctly. I investigated further and used this simple script to achieve the same results: ======================= from os.path import abspath, dirname, join import requests BASEDIR=abspath(dirname(__file__)) rhn_url = "https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/cf-tools/1/os/repodata/repomd.xml" cert = join(BASEDIR, "248437169715645491.pem") cacert = join(BASEDIR, "redhat-uep.pem") key = join(BASEDIR, "248437169715645491-key.pem") proxy = "http://mycompany.proxy.com:3128/" param_proxies = {'http': proxy, 'https': proxy} s = requests.session() s.proxies = param_proxies s.cert = (cert, key) s.verify = cacert print s.get(rhn_url) ======================= I used my Fedora 19 Machine with python 2.7 to run the tests, as the results were the same. Using wireshark I could see, that the Proxy (Bluecoat) sends the [ACK, RST] instantely after the ClientHello. Looking into the ClientHello and comparing it with the working one from the curl-command, I could see, that requests/urllib3 uses TLS1.2 within the Handshake Protocol, whereas curl used TLS 1.0. Googling around I found the openssl changelog entry "Changes between 1.0.1 and 1.0.1a [19 Apr 2012]". IT introduces "Workarounds for some broken servers that "hang" if a client hello record length exceeds 255 bytes." (http://www.openssl.org/news/changelog.html) The Bluecoat KnowledgeBase points into the direction of openssl's workaround: https://kb.bluecoat.com/index?page=content&id=KB4015&actp=LIST I compiled openssl on my Fedora with -DOPENSSL_MAX_TLS1_2_CIPHER_LENGTH=50 *and* -DOPENSSL_NO_TLS1_2_CLIENT , created a virtual env, compiled pyOpenSSL linked to my new openssl Library and the script worked! Yeah! As the ClientHello Package has a length now of 262 Bytes, I don't think it is connected to the packet length, but to the usage of TLS1.2 in the handshake. Tomorrow I will try to reproduce it on my RHEL Machine and will follow up here. As I couldn't find how to tell requests/urllib3/openSSL not to use TLS1.2 - fact is, that only TLSv1 is mentioned within the python files - I have no idea how to solve this issue in a proper way. Update tomorrow...
The problem is not the length of the packet but the usage of TLS 1.2, which does not work correctly with our Bluecoat. I finally found a proper way to make it work by setting the Default SSL Protocol of urllib3 to TLSv1. After two and a half days of messing around with requests, urllib3, openssl and pyopenssl I was able to find a rather simple solution. Setting TLSv1 explicitly, the Client Handshake will not involve TLS1.2, which is involved when the default setting is SSLv23. The default setting ist set in urllib3/util.py. --- /usr/lib/python2.6/site-packages/requests/packages/urllib3/util.py 2013-09-24 20:09:11.000000000 +0200 +++ /tmp/util.py 2013-12-12 14:58:45.539748660 +0100 @@ -26,7 +26,7 @@ HAS_SNI = False import ssl - from ssl import wrap_socket, CERT_NONE, PROTOCOL_SSLv23 + from ssl import wrap_socket, CERT_NONE, PROTOCOL_SSLv23, PROTOCOL_TLSv1 from ssl import SSLContext # Modern SSL? from ssl import HAS_SNI # Has SNI? except ImportError: @@ -528,7 +528,7 @@ like resolve_cert_reqs """ if candidate is None: - return PROTOCOL_SSLv23 + return PROTOCOL_TLSv1 if isinstance(candidate, str): res = getattr(ssl, candidate, None) I will post a follow up to my post on the pulp-Mailinglist in case someone has a similar problem, although is related to urllib3... Ticket can be closed.
The same bug and Florian's workaround can be reproduced with SGOS 6.2.13.2. We'll report the bug to the manufacturer. Thanks you.
I think the Bluecoat KB Entry https://kb.bluecoat.com/index?page=content&id=KB5493 applies. "TLS 1.2 is not supported in SGOS 6.4.x or earlier. TLS 1.2 support is a feature request for SGOS 6.5.x." Florian
It turned out that bluecoat did not support TLS 1.2. A workaround was established, and bluecoat is planning future support for 1.2.
The vendor advised us to update to the latest version of Blue Coat which should fix the problem.