Bug 1750650

> Technically, the git clone does not come from that image.  However, looking at quay.io/openshift/origin-docker-builder:4.2 it is also at 1.8.3.1

I suspect this is the version of git on RHEL 7 - if so anything running on OpenShift using RHEL-7 based images will run into this issue. We may not be able to do anything until we move OpenShift images to be ubi8 based, at which point we will hopefully get the necessary updates to libcurl and git to make the things work as intended.

Until then I propose that we mark this as a known issue - "Builds: git clone fails if SslBump/peek-and-slice is enabled on Squid proxies".

Is this specific to "sslbump" or is it any MITM-configured proxy?  (I'm not clear on the distinction).

also we should stop marking comments private unless there is a good reason.  this thread will likely be useful information for support to be able to find+refer customers to.

@Ben I think sslbump is Squid's implementation for a MITM proxy - there are two variants depending on version [1][2]. I don't think anyone has sufficient expertise to know if this is an issue specific to Squid or a general issue with MITM proxies. I suspect the latter.

[1] https://wiki.squid-cache.org/Features/SslBump
[2] https://wiki.squid-cache.org/Features/SslPeekAndSplice

It seems like curl(libcurl) added support for https proxies on version 7.52 [1][2][3] while the default version on rhel7 seems to be 7.29. As git depends on libcurl it does not work.

[1] https://github.com/curl/curl/pull/1127
[2] https://github.com/curl/curl/commit/cb4e2be7c6d42ca0780f8e0a747cecf9ba45f151
[3] https://curl.haxx.se/changes.html (search for "Fixed in 7.52.0")

It seems like curl(libcurl) added support for https proxies on version 7.52 [1][2][3] while version on RHEL7 seems to be 7.29. I have noticed that other tools(such as yum) were not working behind the HTTPS proxy as well, very likely due to the same dependency on libcurl. What made me this far without noticing this problem was the fact that I was setting HTTPS_PROXY to http://proxy_ip:3128 and my proxy was configured to forward CONNECT requests directly to the remote servers(on this scenario I think that the local ca trust bundle for the proxy is not used as there is no need for it, no bump).

My derived conclusion is that any tool linked to libcurl on images based on RHEL7 are not going to communicate through HTTPS proxies. I am going to add this e-mail content to the BZ as well.

[1] https://github.com/curl/curl/pull/1127
[2] https://github.com/curl/curl/commit/cb4e2be7c6d42ca0780f8e0a747cecf9ba45f151
[3] https://curl.haxx.se/changes.html (search for "Fixed in 7.52.0")

Follow my git clone attempt now using an openshift/origin-docker-builder:v3.11 container. Long story short, same behavior as with a raw centos7. 


[root@5e80d34f4072 /]# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 
[root@5e80d34f4072 /]# git --version
git version 1.8.3.1
[root@5e80d34f4072 /]# vi /etc/pki/ca-trust/source/anchors/proxy.pem
[root@5e80d34f4072 /]# update-ca-trust 
[root@5e80d34f4072 /]# export http_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3128 
[root@5e80d34f4072 /]# export HTTPS_PROXY=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3130
[root@5e80d34f4072 /]# export HTTPS_PROXY=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3130
[root@5e80d34f4072 /]# export https_proxy=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3130
[root@5e80d34f4072 /]# export HTTP_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3128
[root@5e80d34f4072 /]# git clone https://github.com/openshift/cluster-image-registry-operator.git
Cloning into 'cluster-image-registry-operator'...
fatal: unable to access 'https://github.com/openshift/cluster-image-registry-operator.git/': Recv failure: Connection reset by peer
[root@5e80d34f4072 /]# rpm -q {curl,libcurl}
curl-7.29.0-51.el7_6.3.x86_64
libcurl-7.29.0-51.el7_6.3.x86_64
[root@5e80d34f4072 /]# yum -y install  https://centos7.iuscommunity.org/ius-release.rpm
Loaded plugins: fastestmirror, ovl
Cannot open: https://centos7.iuscommunity.org/ius-release.rpm. Skipping.
Error: Nothing to do
[root@5e80d34f4072 /]# unset http_proxy
[root@5e80d34f4072 /]# unset https_proxy
[root@5e80d34f4072 /]# unset HTTPS_PROXY
[root@5e80d34f4072 /]# unset HTTP_PROXY
[root@5e80d34f4072 /]# yum -y install https://centos7.iuscommunity.org/ius-release.rpm
<redacted>
Complete!
[root@5e80d34f4072 /]# rpm -Uvh http://www.city-fan.org/ftp/contrib/yum-repo/city-fan.org-release-2-1.rhel7.noarch.rpm
<redacted>
Complete!
[root@5e80d34f4072 /]# yum --enablerepo=city-fan.org update curl
<redacted>
Complete!
[root@5e80d34f4072 /]# rpm -q {curl,libcurl}
curl-7.66.0-1.1.cf.rhel7.x86_64
libcurl-7.66.0-1.1.cf.rhel7.x86_64
[root@5e80d34f4072 /]# export http_proxy=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3128
[root@5e80d34f4072 /]# export HTTPS_PROXY=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3130
[root@5e80d34f4072 /]# export https_proxy=https://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3130
[root@5e80d34f4072 /]# export HTTP_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.76.148:3128
[root@5e80d34f4072 /]# git clone https://github.com/openshift/cluster-image-registry-operator.git
Cloning into 'cluster-image-registry-operator'...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 71386 (delta 0), reused 1 (delta 0), pack-reused 71379
Receiving objects: 100% (71386/71386), 116.64 MiB | 1.45 MiB/s, done.
Resolving deltas: 100% (34488/34488), done.
Checking out files: 100% (28675/28675), done.
[root@5e80d34f4072 /]#

ssl_bump supports different actions with acl's to provide ssl bumping flexibility. The following example may be a workaround for this bug that allows bumping for all destinations except "broken sites" (i.e. github):

# squid.conf
acl broken_sites ssl::server_name .github.com
ssl_bump splice broken_sites
ssl_bump bump all

The 'splice' action allows the proxy to become a TCP tunnel without decrypting proxied traffic. Note: This is the default action for ssl_bump.

See the following for additional ssl_bump details:

http://www.squid-cache.org/Doc/config/ssl_bump/

Tested also against dev proxy(ec2-52-73-102-120.compute-1.amazonaws.com) provided by @Daneyon, same problem. Only works after libcurl update.

Treating this as an RFE - moving to JIRA: https://jira.coreos.com/browse/DEVEXP-440

Old bug, but I looked at it again recently and couldn't remember if it had been resolved since being deferred.  Recording for posterity: OpenShift 4.6 brought in a UBI bump with Git v2, which supports https:// values for HTTPS_PROXY.  That should cover this issue [1].

[1]: https://docs.openshift.com/container-platform/4.6/release_notes/ocp-4-6-release-notes.html#ocp-4-6-builds-git-clones-http-proxy

We had resolved this issue in https://issues.redhat.com/browse/BUILD-68 since 4.6, but still got failure to download dependences in https proxy cluster. Track the new issue in bug https://bugzilla.redhat.com/show_bug.cgi?id=1881790