Bug 1750650 - [HTTPS_PROXY] Can not visit github.com when do build in https_proxy cluster
Summary: [HTTPS_PROXY] Can not visit github.com when do build in https_proxy cluster
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.3.0
Assignee: Gabe Montero
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-10 07:31 UTC by wewang
Modified: 2020-09-09 18:34 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-08 14:39:38 UTC
Target Upstream Version:


Attachments (Terms of Use)

Comment 39 Adam Kaplan 2019-09-18 02:19:37 UTC
> Technically, the git clone does not come from that image.  However, looking at quay.io/openshift/origin-docker-builder:4.2 it is also at 1.8.3.1

I suspect this is the version of git on RHEL 7 - if so anything running on OpenShift using RHEL-7 based images will run into this issue. We may not be able to do anything until we move OpenShift images to be ubi8 based, at which point we will hopefully get the necessary updates to libcurl and git to make the things work as intended.

Until then I propose that we mark this as a known issue - "Builds: git clone fails if SslBump/peek-and-slice is enabled on Squid proxies".

Comment 40 Ben Parees 2019-09-18 02:22:11 UTC
Is this specific to "sslbump" or is it any MITM-configured proxy?  (I'm not clear on the distinction).

also we should stop marking comments private unless there is a good reason.  this thread will likely be useful information for support to be able to find+refer customers to.

Comment 41 Adam Kaplan 2019-09-19 02:43:24 UTC
@Ben I think sslbump is Squid's implementation for a MITM proxy - there are two variants depending on version [1][2]. I don't think anyone has sufficient expertise to know if this is an issue specific to Squid or a general issue with MITM proxies. I suspect the latter.

[1] https://wiki.squid-cache.org/Features/SslBump
[2] https://wiki.squid-cache.org/Features/SslPeekAndSplice

Comment 42 Ricardo Maraschini 2019-09-19 08:19:29 UTC
It seems like curl(libcurl) added support for https proxies on version 7.52 [1][2][3] while the default version on rhel7 seems to be 7.29. As git depends on libcurl it does not work.

[1] https://github.com/curl/curl/pull/1127
[2] https://github.com/curl/curl/commit/cb4e2be7c6d42ca0780f8e0a747cecf9ba45f151
[3] https://curl.haxx.se/changes.html (search for "Fixed in 7.52.0")

Comment 43 Ricardo Maraschini 2019-09-19 08:36:30 UTC
It seems like curl(libcurl) added support for https proxies on version 7.52 [1][2][3] while version on RHEL7 seems to be 7.29. I have noticed that other tools(such as yum) were not working behind the HTTPS proxy as well, very likely due to the same dependency on libcurl. What made me this far without noticing this problem was the fact that I was setting HTTPS_PROXY to http://proxy_ip:3128 and my proxy was configured to forward CONNECT requests directly to the remote servers(on this scenario I think that the local ca trust bundle for the proxy is not used as there is no need for it, no bump).

My derived conclusion is that any tool linked to libcurl on images based on RHEL7 are not going to communicate through HTTPS proxies. I am going to add this e-mail content to the BZ as well.

[1] https://github.com/curl/curl/pull/1127
[2] https://github.com/curl/curl/commit/cb4e2be7c6d42ca0780f8e0a747cecf9ba45f151
[3] https://curl.haxx.se/changes.html (search for "Fixed in 7.52.0")

Comment 44 Ricardo Maraschini 2019-09-19 15:04:39 UTC
Follow my git clone attempt now using an openshift/origin-docker-builder:v3.11 container. Long story short, same behavior as with a raw centos7. 


[root@5e80d34f4072 /]# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 
[root@5e80d34f4072 /]# git --version
git version 1.8.3.1
[root@5e80d34f4072 /]# vi /etc/pki/ca-trust/source/anchors/proxy.pem
[root@5e80d34f4072 /]# update-ca-trust 
[root@5e80d34f4072 /]# export http_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3128 
[root@5e80d34f4072 /]# export HTTPS_PROXY=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3130
[root@5e80d34f4072 /]# export HTTPS_PROXY=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3130
[root@5e80d34f4072 /]# export https_proxy=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3130
[root@5e80d34f4072 /]# export HTTP_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3128
[root@5e80d34f4072 /]# git clone https://github.com/openshift/cluster-image-registry-operator.git
Cloning into 'cluster-image-registry-operator'...
fatal: unable to access 'https://github.com/openshift/cluster-image-registry-operator.git/': Recv failure: Connection reset by peer
[root@5e80d34f4072 /]# rpm -q {curl,libcurl}
curl-7.29.0-51.el7_6.3.x86_64
libcurl-7.29.0-51.el7_6.3.x86_64
[root@5e80d34f4072 /]# yum -y install  https://centos7.iuscommunity.org/ius-release.rpm
Loaded plugins: fastestmirror, ovl
Cannot open: https://centos7.iuscommunity.org/ius-release.rpm. Skipping.
Error: Nothing to do
[root@5e80d34f4072 /]# unset http_proxy
[root@5e80d34f4072 /]# unset https_proxy
[root@5e80d34f4072 /]# unset HTTPS_PROXY
[root@5e80d34f4072 /]# unset HTTP_PROXY
[root@5e80d34f4072 /]# yum -y install https://centos7.iuscommunity.org/ius-release.rpm
<redacted>
Complete!
[root@5e80d34f4072 /]# rpm -Uvh http://www.city-fan.org/ftp/contrib/yum-repo/city-fan.org-release-2-1.rhel7.noarch.rpm
<redacted>
Complete!
[root@5e80d34f4072 /]# yum --enablerepo=city-fan.org update curl
<redacted>
Complete!
[root@5e80d34f4072 /]# rpm -q {curl,libcurl}
curl-7.66.0-1.1.cf.rhel7.x86_64
libcurl-7.66.0-1.1.cf.rhel7.x86_64
[root@5e80d34f4072 /]# export http_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3128
[root@5e80d34f4072 /]# export HTTPS_PROXY=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3130
[root@5e80d34f4072 /]# export https_proxy=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3130
[root@5e80d34f4072 /]# export HTTP_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3128
[root@5e80d34f4072 /]# git clone https://github.com/openshift/cluster-image-registry-operator.git
Cloning into 'cluster-image-registry-operator'...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 71386 (delta 0), reused 1 (delta 0), pack-reused 71379
Receiving objects: 100% (71386/71386), 116.64 MiB | 1.45 MiB/s, done.
Resolving deltas: 100% (34488/34488), done.
Checking out files: 100% (28675/28675), done.
[root@5e80d34f4072 /]#

Comment 45 Daneyon Hansen 2019-09-19 15:55:30 UTC
ssl_bump supports different actions with acl's to provide ssl bumping flexibility. The following example may be a workaround for this bug that allows bumping for all destinations except "broken sites" (i.e. github):

# squid.conf
acl broken_sites ssl::server_name .github.com
ssl_bump splice broken_sites
ssl_bump bump all

The 'splice' action allows the proxy to become a TCP tunnel without decrypting proxied traffic. Note: This is the default action for ssl_bump.

See the following for additional ssl_bump details:

http://www.squid-cache.org/Doc/config/ssl_bump/

Comment 46 Ricardo Maraschini 2019-09-19 16:24:07 UTC
Tested also against dev proxy(ec2-52-73-102-120.compute-1.amazonaws.com) provided by @Daneyon, same problem. Only works after libcurl update.

Comment 47 Adam Kaplan 2019-09-27 20:39:23 UTC
Treating this as an RFE - moving to JIRA: https://jira.coreos.com/browse/DEVEXP-440


Note You need to log in before you can comment on or make changes to this bug.