Bug 1039471 - Syncing RHN Repositories via PROXY no longer working
Summary: Syncing RHN Repositories via PROXY no longer working
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Pulp
Classification: Retired
Component: rpm-support
Version: 2.3
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: pulp-bugs
QA Contact: pulp-qe-list
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-12-09 08:31 UTC by Florian Sachs
Modified: 2013-12-18 16:45 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-12-18 16:40:50 UTC
Embargoed:


Attachments (Terms of Use)

Description Florian Sachs 2013-12-09 08:31:26 UTC
Description of problem:
With 2.3 the syncing of RHN Repositories via our company Proxy stopped working.

Version-Release number of selected component (if applicable):
RHEL 6.5
Pulp 2.3

How reproducible:
Create Pulprepository with feed from RHN via Proxy. Start sync.

Steps to Reproduce:
1. pulp-admin rpm repo create --repo-id=rhel-6-server-cf-tools-1-rpms \
    --feed=https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/cf-tools/1/os \
    --feed-ca-cert=/etc/rhsm/ca/redhat-uep.pem \
    --feed-cert=/etc/pki/entitlement/XXXXX.pem \
    --feed-key=/etc/pki/entitlement/XXXX-key.pem \
    --description="Red Hat CloudForms Tools for RHEL 6 (RPMs)" \
    --serve-http=True \
    --proxy-host=http://proxy.mycompany.com \
    --proxy-port=3128 \
    --retain-old-count=1 \
    --remove-missing=True \
    --validate=True

2. pulp-admin rpm repo sync run --repo-id=rhel-6-server-cf-tools-1-rpms

Actual results:
Sync does not work. 

/var/log/pulp/nectar.log:
2013-12-09 09:18:49,226 requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): cdn.redhat.com
2013-12-09 09:18:49,758 nectar.downloaders.threaded:ERROR: Cannot connect to proxy. Socket error: [Errno 104] Connection reset by peer.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nectar/downloaders/threaded.py", line 161, in _fetch
    response = session.get(request.url, headers=headers)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 369, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 357, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 460, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 319, in send
    timeout=timeout
  File "/usr/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py", line 583, in urlopen
    'Socket error: %s.' % e)
ProxyError: Cannot connect to proxy. Socket error: [Errno 104] Connection reset by peer.



Expected results:
Repository should sync.

Additional info:
* Worked with 2.1
* Does neither not work with "migrated" Repositories from 2.1 to 2.3 nor with new created ones
* Non-RHN Repositories (like EPEL) work

Comment 1 Michael Hrivnak 2013-12-09 15:13:35 UTC
We are not able to reproduce this problem. Can you tell us more about the proxy server itself? Are you using any kind of auth with it?

It looks like the proxy server is resetting the TCP connection between itself and pulp, so perhaps it has some log output explaining why.

Comment 2 Florian Sachs 2013-12-10 16:14:15 UTC
Our proxy is a Bluecoat and we do not use authentication. I tried with tinyproxy in a special environment an can confirm that it works with tiny proxy.

I tried to investigate and think it is somehow connected to urllib3's connectionpooling...

Our Proxyadmin will return tomorrow and I hope we find some time to get around the problem.

I will keep this bugreport updated..

Comment 3 Florian Sachs 2013-12-11 14:57:18 UTC
TLDR:
I think the Bluecoat does not support TLS 1.2 Clients correctly.

I investigated further and used this simple script to achieve the same results:
=======================
from os.path import abspath, dirname, join
import requests


BASEDIR=abspath(dirname(__file__))

rhn_url     = "https://cdn.redhat.com/content/dist/rhel/server/6/6Server/x86_64/cf-tools/1/os/repodata/repomd.xml"
cert        = join(BASEDIR, "248437169715645491.pem")
cacert      = join(BASEDIR, "redhat-uep.pem")
key         = join(BASEDIR, "248437169715645491-key.pem")
proxy       = "http://mycompany.proxy.com:3128/"

param_proxies = {'http': proxy, 'https': proxy}

s = requests.session()
s.proxies = param_proxies
s.cert = (cert, key)
s.verify = cacert

print s.get(rhn_url)
=======================

I used my Fedora 19 Machine with python 2.7 to run the tests, as the results were the same.

Using wireshark I could see, that the Proxy (Bluecoat) sends the [ACK, RST] instantely after the ClientHello. Looking into the ClientHello and comparing it with the working one from the curl-command, I could see, that requests/urllib3 uses TLS1.2 within the Handshake Protocol, whereas curl used TLS 1.0.

Googling around I found the openssl changelog entry "Changes between 1.0.1 and 1.0.1a [19 Apr 2012]". IT introduces "Workarounds for some broken servers that "hang" if a client hello record length exceeds 255 bytes." (http://www.openssl.org/news/changelog.html)

The Bluecoat KnowledgeBase points into the direction of openssl's workaround: https://kb.bluecoat.com/index?page=content&id=KB4015&actp=LIST

I compiled openssl on my Fedora with -DOPENSSL_MAX_TLS1_2_CIPHER_LENGTH=50 *and* -DOPENSSL_NO_TLS1_2_CLIENT , created a virtual env, compiled pyOpenSSL linked to my new openssl Library and the script worked! Yeah!

As the ClientHello Package has a length now of 262 Bytes, I don't think it is connected to the packet length, but to the usage of TLS1.2 in the handshake.

Tomorrow I will try to reproduce it on my RHEL Machine and will follow up here. As I couldn't find how to tell requests/urllib3/openSSL not to use TLS1.2 - fact is, that only TLSv1 is mentioned within the python files - I have no idea how to solve this issue in a proper way.

Update tomorrow...

Comment 4 Florian Sachs 2013-12-12 14:31:01 UTC
The problem is not the length of the packet but the usage of TLS 1.2, which does not work correctly with our Bluecoat.

I finally found a proper way to make it work by setting the Default SSL Protocol of urllib3 to TLSv1.

After two and a half days of messing around with requests, urllib3, openssl and pyopenssl I was able to find a rather simple solution. Setting TLSv1 explicitly, the Client Handshake will not involve TLS1.2, which is involved when the default setting is SSLv23. The default setting ist set in urllib3/util.py.

--- /usr/lib/python2.6/site-packages/requests/packages/urllib3/util.py  2013-09-24 20:09:11.000000000 +0200
+++ /tmp/util.py        2013-12-12 14:58:45.539748660 +0100
@@ -26,7 +26,7 @@
     HAS_SNI = False

     import ssl
-    from ssl import wrap_socket, CERT_NONE, PROTOCOL_SSLv23
+    from ssl import wrap_socket, CERT_NONE, PROTOCOL_SSLv23, PROTOCOL_TLSv1
     from ssl import SSLContext  # Modern SSL?
     from ssl import HAS_SNI  # Has SNI?
 except ImportError:
@@ -528,7 +528,7 @@
     like resolve_cert_reqs
     """
     if candidate is None:
-        return PROTOCOL_SSLv23
+        return PROTOCOL_TLSv1

     if isinstance(candidate, str):
         res = getattr(ssl, candidate, None)

		 
I will post a follow up to my post on the pulp-Mailinglist in case someone has a similar problem, although is related to urllib3...

Ticket can be closed.

Comment 5 Arnold Bechtoldt 2013-12-12 14:46:45 UTC
The same bug and Florian's workaround can be reproduced with SGOS 6.2.13.2.

We'll report the bug to the manufacturer.

Thanks you.

Comment 6 Florian Sachs 2013-12-12 15:05:48 UTC
I think the Bluecoat KB Entry https://kb.bluecoat.com/index?page=content&id=KB5493 applies.

"TLS 1.2 is not supported in SGOS 6.4.x or earlier. TLS 1.2 support is a feature request for SGOS 6.5.x."

Florian

Comment 7 Michael Hrivnak 2013-12-18 16:40:50 UTC
It turned out that bluecoat did not support TLS 1.2. A workaround was established, and bluecoat is planning future support for 1.2.

Comment 8 Arnold Bechtoldt 2013-12-18 16:45:36 UTC
The vendor advised us to update to the latest version of Blue Coat which should fix the problem.


Note You need to log in before you can comment on or make changes to this bug.