Bug 1217477

Summary: Git over HTTPS doesn't work with TLSv1.1 or TLSv1.2
Product: Red Hat Enterprise Linux 7 Reporter: Viktor B <bviktor>
Component: gitAssignee: Petr Stodulka <pstodulk>
Status: CLOSED DUPLICATE QA Contact: qe-baseos-daemons
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.1CC: emaldona, fweimer, hkario, kdudka, kengert, lpol, manuel.wolfshant, rrelyea, spider, tsorense
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: curl-7.29.0-24.el7 nss-3.19.1-14.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-07 15:39:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1171318    
Bug Blocks:    
Attachments:
Description Flags
Allow the use of TLS 1.2 in NSS by default none

Description Viktor B 2015-04-30 13:01:45 UTC
If using Git over HTTPS, it only accepts TLSv1. Host version is CentOS 7.1.1503 with EL7 packages only. All updates are installed, thus Git version is 1.8.3.1-4.el7. But I got reports that it works (err, fails) the same way on EL6 e.g. CentOS 6.6, too. I've also tried to install the GhettoForge version that is git-2.3.1-1.gf.el7. I've also tried to build Git from sources (2.3.7). I also got a report with a custom build, git-2.3.0+20150206042556-1.1.x86_64. They all fail the same way. 

If I enforce 1.1 or 1.2 on the server side, the CentOS client says:

$ GIT_CURL_VERBOSE=1 git clone https://tlstest.foobar.net:8443/r/test.git
Cloning into 'test'...
* Couldn't find host tlstest.foobar.net in the .netrc file; using defaults
* About to connect() to tlstest.foobar.net port 8443 (#0)
*   Trying 104.40.152.x...
* Connected to tlstest.foobar.net (104.40.152.x) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0
fatal: unable to access 'https://tlstest.foobar.net:8443/r/test.git/': TCP connection reset by peer

The Git frontend server that handles HTTPS over the Git server is Nginx with the following SSL-related settings:

ssl_protocols TLSv1.2;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_prefer_server_ciphers on;
ssl_ciphers 'AES256+EECDH:AES256+EDH';
ssl_dhparam dh4096.pem;
ssl_stapling on;
ssl_stapling_verify on;
resolver 208.67.222.222 208.67.220.220 valid=300s;
resolver_timeout 10s;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
add_header X-Content-Type-Options nosniff;
add_header X-Frame-Options DENY;

The moment I add TLSv1 to ssl_protocols, it starts to work.

This same server works at least with:

* Windows + MsysGit 2.3.7.1
* OS X 10.10 + Git 2.1.0
* Ubuntu 14.10 + Git 2.1.0
* Ubuntu 14.04 + Git 1.9.1

I.e. everything non-CentOS we have.

I've tested the same site with cURL, Wget and the OpenSSL client on the same CentOS box:

curl --tlsv1.2 https://tlstest.foobar.net:8443/
wget https://tlstest.foobar.net:8443/
openssl s_client -connect tlstest.foobar.net:8443 -tls1_2

They all work, too.

So apparently this only affects Git, and only on EL, and only if TLSv1.1 or TLSv1.2 is enforced on the server side. My assumption is that pretty much all the big Git providers let TLSv1.0 pass because the current Windows release (msysgit) also fails with 1.1 or 1.2 and they don't want to leave Windows users in the dark. But this will get increasingly important as the msysgit2 release is nearing with TLSv1.2 support. Once *any* big Git site (e.g. GitHub) starts to enforce 1.1 or 1.2, EL users will be the ones left in the dark.

If anyone's willing to test this, I can provide you with a server which you can test against. Please let me know if you need any more details, thanks.

Comment 2 Petr Stodulka 2015-04-30 14:28:27 UTC
Thanks for detailed report Viktor, I will look at it closer during next two weeks - probably after next week.

Comment 3 Viktor B 2015-04-30 20:52:23 UTC
Thank you Petr, take your time :)

In the meanwhile I have quite some updates to report. I talked to a RHEL user and he verified that this issue is also present both on RHEL6 and RHEL7.

I also talked to kaie of NSS who gave me a tremendous amount of help (thank you a lot!). The idea was to try connecting with tstclnt and vfyserv which are part of NSS. For this, first you gotta create a security database folder:

$ mkdir nssdb
$ certutil -N -d nssdb
$ modutil -add nssckbi-module -dbdir nssdb -libfile /usr/lib64/nss/libnssckbi.so
(On Windows, modutil -add nssckbi-module -dbdir nssdb -libfile c:\path\to\nssckbi.dll)

Set up the PATH:

$ export PATH=$PATH:/usr/lib64/nss/unsupported-tools

Then you can test the connection with tstclnt:

$ tstclnt -p 8443 -d nssdb -h tlstest.foobar.net
tstclnt: error enabling SSLv2 : SSL_ERROR_SSL2_DISABLED: Peer only supports SSL version 2, which is locally disabled.

There's something wrong with detecting protocol versions, so let's explicitly specify it:

$ tstclnt -p 8443 -d nssdb -V tls1.2:tls1.2 -h tlstest.foobar.net
subject DN: CN=*.foobar.net,OU=EssentialSSL Wildcard,OU=Domain Control Validated
issuer  DN: CN=COMODO RSA Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB
0 cache hits; 1 cache misses, 0 cache not reusable
0 stateless resumes
Received 0 Cert Status items (OCSP stapled data)

So it's ok. Now let's try with vfyserv from NSS 3.17.3 on Windows or 3.17.4 on Ubuntu:

$ vfyserv -p 8443 -d nssdb tlstest.foobar.net
Connecting to host tlstest.foobar.net (addr 104.40.152.x) on port 8443
Error in function PR_Write: -5961
 - TCP connection reset by peer

Oops, the same error Git is giving me. Unfortunately you can't specify protocol for vfyserv. And this seems to be the key! 3.18 changes the maximum enabled TLS protocol version *enabled by default* to 1.2. kaie also checked and the above command works on Fedora 21, which has NSS 3.18. So I've built NSS 3.18 on my Windows box and tried again:

$ vfyserv -p 8443 -d nssdb tlstest.foobar.net
Connecting to host tlstest.foobar.net (addr 104.40.152.x) on port 8443
Handshake Complete: SERVER CONFIGURED CORRECTLY
   bulk cipher AES-256, 256 secret key bits, 256 key bits, status: 1
   subject DN:
 CN=*.foobar.net,OU=EssentialSSL Wildcard,OU=Domain Control Validated
   issuer  DN:
 CN=COMODO RSA Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB
   0 cache hits; 0 cache misses, 0 cache not reusable
***** Connection 1 read 511 bytes total.

Bingo. So what I *guess* the situation is, is that the Git client doesn't "ask" NSS via the API to enable TLS 1.2 and that's why it fails.

Now the questions remain:

- Why does Git work on Ubuntu with NSS 3.17? My best guess is that Ubuntu has this patched in Git, libcurl, NSS or anywhere in between to explicitly enable TLSv1.2. 
- Why does Git work on Windows? From what I understand, msysgit doesn't even use NSS at all and relies on OpenSSL instead, but FIXME.
- Why does Git work on OS X? Probably also one of the above.

So apparently the trivial fix is to upgrade NSS to 3.18 but I'm pretty sure it's not going to happen within release. The less trivial fix is to find out how exactly TLSv1.2 is explicitly enabled on Ubuntu.

Comment 5 Viktor B 2015-05-01 13:06:08 UTC
Created attachment 1020853 [details]
Allow the use of TLS 1.2 in NSS by default

Actually, I didn't think of the most trivial solution: diff between NSS 3.17 and 3.18 and find the TLS enablement bump. The patch is totally obvious. EL7 has NSS 3.16.2.3, so I tested it with that release. Built it both without and with the patch. The results:

Before:

$ vfyserv -p 8443 -d nssdb tlstest.foobar.net
Connecting to host tlstest.foobar.net (addr 104.40.152.x) on port 8443
Error in function PR_Write: -5961
 - TCP connection reset by peer

After:

$ vfyserv -p 8443 -d nssdb tlstest.foobar.net
Connecting to host tlstest.foobar.net (addr 104.40.152.x) on port 8443
Handshake Complete: SERVER CONFIGURED CORRECTLY
   bulk cipher AES-256, 256 secret key bits, 256 key bits, status: 1
   subject DN:
 CN=*.foobar.net,OU=EssentialSSL Wildcard,OU=Domain Control Validated
   issuer  DN:
 CN=COMODO RSA Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB
   0 cache hits; 0 cache misses, 0 cache not reusable
***** Connection 1 read 511 bytes total.

I haven't actually rebuilt Git using such a patched NSS release yet but it'll most likely work, too.

Comment 6 Viktor B 2015-05-03 15:00:44 UTC
Okay, I've tested it all through. First I patched NSS 3.16.2.3, then built it. Then installed it (yeah, messy, it's just a test VM and I was lazy to fiddle around with SRPMs):

cd ../dist                                                       &&
install -v -m755 Linux*/lib/*.so              /usr/lib64         &&
install -v -m644 Linux*/lib/{*.chk,libcrmf.a} /usr/lib64         &&
install -v -m755 -d                           /usr/include/nss   &&
cp -v -RL {public,private}/nss/*              /usr/include/nss   &&
chmod -v 644                                  /usr/include/nss/* &&
install -v -m755 Linux*/bin/{certutil,nss-config,pk12util} /usr/bin

Then built cURL 7.29.0 and installed it with the default /usr/local prefix:

./configure
make
sudo make install

Then built Git 1.8.3.1 with:

./configure --with-curl=/usr/local
make prefix=/usr/local/git all
sudo make prefix=/usr/local/git install

Then checked the results:

$ git clone https://tlstest.foobar.net:8443/r/test.git
Cloning into 'test'...
fatal: unable to access 'https://tlstest.foobar.net:8443/r/test.git/': TCP connection reset by peer

$ /usr/local/git/bin/git clone https://tlstest.foobar.net:8443/r/test.git
Cloning into 'test'...
remote: Counting objects: 3, done
remote: Finding sources: 100% (3/3)
remote: Getting sizes: 100% (2/2)
remote: Compressing objects: 100% (37/37)
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.

Yay!

Comment 7 Petr Stodulka 2015-05-06 13:08:15 UTC
Hi Viktor, thanks for digging in this. I wrote with Kai and nss-3.18 is planned for rhel-7.2 with set TLSv1.2 as default.

Comment 8 Viktor B 2015-05-06 14:45:48 UTC
Thank you Petr and Kai, looking forward to 7.2!

Comment 13 Viktor B 2015-05-20 16:38:39 UTC
I see NSS has been updated to 3.18 already in 7.1 along with several packages depending on it (e.g. Tomcat).

So at this point all the fix takes is a rebuild of cURL and Git. Is there any chance for this happening in 7.1? Thanks.

Comment 14 Kai Engert (:kaie) (inactive account) 2015-05-20 16:45:16 UTC
The current plan is to keep the old default in 7.1, and make the change of default in 7.2

In the source RPM there is a file named nss-revert-tls-version-defaults.patch

If you required this fix urgently, you could recompile NSS locally without applying that patch.

Comment 15 Viktor B 2015-05-20 16:48:50 UTC
Thanks for the info. Either way, that's rather strange coz when I compiled cURL and Git from source after the NSS 3.18 update, things started to work. Maybe I messed up my configs so I'll re-check sometime later.

Comment 16 Kai Engert (:kaie) (inactive account) 2015-05-20 16:56:59 UTC
If you compiled curl and git yourself using plain upstream sources, maybe it wasn't built against nss, but rather against openssl? You could check using the "ld" command on the libcurl.so file you that you built yourself. If it's built against nss, you'd see "libnss3" in the output.

Comment 17 Kamil Dudka 2015-05-20 17:23:59 UTC
There is no need to rebuild git.  Once bug #1170339 and bug #1171318 are fixed, the problem behind this bug will go away.

Comment 19 Kamil Dudka 2015-12-07 15:39:37 UTC
This bug has been fixed since curl-7.29.0-24.el7 and nss-3.19.1-14.el7.

*** This bug has been marked as a duplicate of bug 1171318 ***

Comment 20 Viktor B 2015-12-15 08:31:55 UTC
Confirmed, this has been resolved in 7.2, thanks.