Bug 1129719

Summary: Issues with ssl client verification using chain ca pem
Product: [Retired] Pulp Reporter: dgao
Component: API/integrationAssignee: Randy Barlow <rbarlow>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: unspecified Docs Contact:
Priority: urgent    
Version: 2.4 BetaCC: bbuckingham, bmbouter, jmatthew, pthomas, rbarlow, skarmark
Target Milestone: ---Keywords: Triaged
Target Release: 2.4.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1138320 (view as bug list) Environment:
Last Closed: 2014-09-23 17:54:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 950743, 1138320    
Attachments:
Description Flags
Create necessary certs to recreate scenario.
none
Test ssl verification using chain ca cert. none

Description dgao 2014-08-13 14:22:54 UTC
Created attachment 926463 [details]
Create necessary certs to recreate scenario.

I'm running into an issue with latest pulp REST binding unable to verify server certificate if the CA cert is a chain cert. It looks like the new pulp is using code from m2crypto library that does not support this feature. Attached are 2 small scripts that will recreate this scenario. 

Note: test_m2crypto.py have pieces of code yanked from pulp bindings/server.py 
Note2: The scripts assume pulp is installed locally.

Here are the steps:

1) ./create_rhui_ssl_certs.sh - This would output a bunch of certs, the important ones are copied to ./certs dir. 
2) Edit line 8 of test_m2crypto.py to point to $HOME/certs/server-ca-chain.pem 
3) Edit /etc/httpd/conf.d/ssl.conf with following key-value pair:
   3.1) SSLCertificateFile $HOME/certs/test-cert.pem
   3.2) SSLCertificateKeyFile $HOME/certs/test-key.pem
4) Restart httpd 
5) python test_m2crypto.py
6) openssl verify -verbose -CAfile $HOME/certs/server-ca-chain.pem $HOME/certs/test-cert.pem


Output should look like:

[root@rhua ~]# python test_m2crypto.py 
certificate verify failed
[root@rhua ~]# openssl verify -verbose -CAfile /root/certs/server-ca-chain.pem /root/certs/test-cert.pem 
/root/certs/test-cert.pem: OK


The version of pulp I'm using is:

[root@rhua ~]# rpm -qa | grep "pulp"
python-isodate-0.5.0-1.pulp.el6.noarch
python-pulp-rpm-common-2.4.0-0.30.beta.el6.noarch
createrepo-0.9.9-21.2.pulp.el6.noarch
pulp-admin-client-2.4.0-0.30.beta.el6.noarch
python-kombu-3.0.15-12.pulp.el6.noarch
pulp-puppet-plugins-2.4.0-0.30.beta.el6.noarch
pulp-selinux-2.4.0-0.30.beta.el6.noarch
pulp-rpm-admin-extensions-2.4.0-0.30.beta.el6.noarch
m2crypto-0.21.1.pulp-8.el6.x86_64
python-pulp-common-2.4.0-0.30.beta.el6.noarch
python-pulp-puppet-common-2.4.0-0.30.beta.el6.noarch
python-pulp-bindings-2.4.0-0.30.beta.el6.noarch
python-pulp-client-lib-2.4.0-0.30.beta.el6.noarch
mod_wsgi-3.4-1.pulp.el6.x86_64
pulp-server-2.4.0-0.30.beta.el6.noarch
pulp-rpm-plugins-2.4.0-0.30.beta.el6.noarch
pulp-puppet-admin-extensions-2.4.0-0.30.beta.el6.noarch
pulp-v2-cds-server-1.0.1-1.git.3.9a1a04f.el6.noarch

Comment 1 dgao 2014-08-13 14:23:43 UTC
Created attachment 926464 [details]
Test ssl verification using chain ca cert.

Comment 2 Randy Barlow 2014-08-25 19:44:56 UTC
Brian has determined that this does not affect Satellite 6 GA as they do not use chain certificates. Given that M2Crypto does not support this feature, it will be a significant effort to resolve this issue.

I will mark this as untriaged so we can decide if we really want to keep this in 2.4.2.

Comment 3 John Matthews 2014-08-25 20:10:20 UTC
When revisiting this BZ, please know that lack of CA Chain support will negatively impact RHUI customers.

RHUI customers often use CA Chains when managing their SSL certificates.

Comment 4 Brian Bouterse 2014-08-27 19:33:13 UTC
This is also a common case for Satellite 6 installations, but not with the default installer. Given use cases that require this by other projects, I'm triaging this as 2.4.2, Urgent.

Comment 5 Randy Barlow 2014-09-03 20:38:15 UTC
It turned out that the 1 in the call to set_verify wasn't a boolean, but an integer specifying how deep we should be willing to follow certificate chains. I believe that setting it to 1 meant that the peer certificate had to be signed by a root certificate authority.

To fix this, I changed that line to this[0]:

ssl_context.set_verify(SSL.verify_peer, depth=100)

I would recommend using Apache's SSLCertificateChainFile directive[1] to give the certificate chain to the client instead of using Pulp's ca_path setting to do this. Pulp's ca_path should only need to be a path to trusted root certificates.

I will write steps to reproduce this bug in another comment.

[0] https://github.com/pulp/pulp/pull/1142
[1] https://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslcertificatechainfile

Comment 6 Randy Barlow 2014-09-03 20:59:09 UTC
To reproduce this bug:

1) Create a root certificate authority (self signed v1 certificate will do).
2) Configure a host running pulp-admin to trust the root CA certificate.
3) Create an intermediate certificate authority.
4) Sign the intermediate certificate with the root certificate from step 1 (this must generate a v3 certificate with the appropriate certificate authority extensions[0].
5) Create a server certificate for Apache to use.
6) Sign the server certificate with the intermediate certificate.
7) Configure Apache's SSLCertificateChainFile directive (in /etc/httpd/conf.d/ssl.conf) to point at the intermediate CA certificate you created in step 3. This will cause Apache to hand that CA out along with its own certificate signed by that CA.
8) Try to use pulp-admin rpm repo list (or any other command that uses the server).

You should see step 8 fail before upgrading to the version that fixes this bug.

Once it is fixed, step 8 should succeed.

[0] For example, I generated one that had these extensions, as printed by openssl x509… -text:

        X509v3 extensions:
            X509v3 Subject Key Identifier:
                C7:AF:E2:AC:53:E6:5F:60:D6:52:3B:61:83:02:9F:4F:5C:11:52:DB
            X509v3 Authority Key Identifier:
                keyid:5C:88:1C:B7:DF:DC:A8:C1:19:D1:C2:26:E0:DD:4C:A7:91:EC:D7:F3

            X509v3 Basic Constraints:
                CA:TRUE
            X509v3 Key Usage:
                Certificate Sign, CRL Sign
            Netscape Cert Type:
                SSL CA, S/MIME CA
            X509v3 Subject Alternative Name:
                email:rbarlow
            X509v3 Issuer Alternative Name:
                <EMPTY>

Comment 7 Preethi Thomas 2014-09-09 19:31:33 UTC
verified
[root@pulp-24-server ~]# rpm -qa pulp-server
pulp-server-2.4.1-0.7.beta.el6.noarch
[root@pulp-24-server ~]# 

followed David's steps from the description 

[root@pulp-24-server ~]# python test_m2crypto.py certificate verify failed
[root@pulp-24-server ~]# openssl verify -verbose -CAfile /root/certs/server-ca-chain.pem /root/certs/test-cert.pem 
/root/certs/test-cert.pem: OK
[root@pulp-24-server ~]# 



Also followed steps from 

https://bugzilla.redhat.com/show_bug.cgi?id=1129719#c6

Comment 8 Randy Barlow 2014-09-23 17:54:25 UTC
This is fixed in Pulp-2.4.1-1.