Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 800093

Summary: CRL preventing repository access - Client certificate did not match the global repo auth CA certificate
Product: Red Hat Satellite Reporter: James Laska <jlaska>
Component: Content ManagementAssignee: Mike McCune <mmccune>
Status: CLOSED CURRENTRELEASE QA Contact: Og Maciel <omaciel>
Severity: high Docs Contact:
Priority: high    
Version: 6.0.0CC: bkearney, cpelland, ftaylor, jmatthew, jturner, mmccune, omaciel, scollier
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-08-22 18:30:22 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description James Laska 2012-03-05 17:56:25 UTC
Description of problem:

Unable to deploy CloudForms images using katello hosted content.  It appears that attempts to access the pulp repos using a valid crt and key are blocked by the CRL.  Removing the CRL file (/etc/pki/pulp/content/40e36648.r0) works around the problem.

Version-Release number of selected component (if applicable):
 * candlepin-0.5.23-1.el6.src.rpm
 * katello-0.1.301-2.el6.src.rpm
 * katello-candlepin-cert-key-pair-1.0-1.src.rpm
 * katello-certs-tools-1.0.3-1.el6.src.rpm
 * katello-cli-0.1.100-2.el6.src.rpm
 * katello-configure-0.1.101-1.el6.src.rpm
 * katello-qpid-broker-key-pair-1.0-1.src.rpm
 * katello-qpid-client-key-pair-1.0-1.src.rpm
 * katello-selinux-0.1.8-1.el6.src.rpm
 * pulp-1.0.0-4.el6.src.rpm

How reproducible:
 * 2 of 2 attempts end up with

Steps to Reproduce:
1. # curl --insecure -L --cert /tmp/uebercert.crt --key /tmp/uebercert.key https://flatline-katello.usersys.redhat.com/pulp/repos/redhat/Stage/content/beta/rhel/server/6/6Server/x86_64/cf-tools/1.0/os/repodata/repomd.xml
  
Actual results:

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /pulp/repos/redhat/Stage/content/beta/rhel/server/6/6Server/x86_64/cf-tools/1.0/os/repodata/repomd.xml
on this server.</p>

Expected results:

<?xml version="1.0" encoding="UTF-8"?>
<repomd xmlns="http://linux.duke.edu/metadata/repo" xmlns:rpm="http://linux.duke.edu/metadata/rpm">
  <revision>1330696594</revision>
  <data type="other_db">
    <location href="repodata/59b8470f65ad8b95a911b95c2d3ebdf4655826ea-other.sqlite.bz2"/>
    <checksum type="sha">59b8470f65ad8b95a911b95c2d3ebdf4655826ea</checksum>
    <timestamp>1330696595.47</timestamp>
    <size>18742</size>
    <open-size>132096</open-size>
    <open-checksum type="sha">b13f8d54bf12296aa3fcc9f14332bc6a29825b55</open-checksum>
    <database_version>10</database_version>
  </data>
...

Additional info:

== /var/log/httpd/ssl_kt_error_log ==
[Mon Mar 05 10:30:31 2012] [error] [client 10.11.231.160] Cert verification failed against 1 ca cert(s) and 1 CRL(s)
[Mon Mar 05 10:30:31 2012] [error] [client 10.11.231.160] Current Time: <Mon Mar  5 10:30:31 2012>
[Mon Mar 05 10:30:31 2012] [error] [client 10.11.231.160] Certificate to verify: 
[Mon Mar 05 10:30:31 2012] [error] [client 10.11.231.160] \tsubject=</CN=8a8b66ac35d3d4120135d4b82e930093>, issuer=</C=US/ST=North Carolina/L=Raleigh/O=Red Hat/OU=Cloud BU/CN=flatline-katello.usersys.redhat.com>, subject.as_hash=<1564283010>, issuer.as_hash=<1088644680>, fingerprint=<A25F74A08C68E3D1EC32CEB930181196>, serial=<3368552365664118609>, version=<2>, check_ca=<0>, notBefore=<Mar  2 18:42:14 2012 GMT>, notAfter=<Dec  3 02:18:42 2021 GMT>
[Mon Mar 05 10:30:31 2012] [error] [client 10.11.231.160] Using a CA Chain with 1 cert(s)
[Mon Mar 05 10:30:31 2012] [error] [client 10.11.231.160] \tCA: subject=</C=US/ST=North Carolina/L=Raleigh/O=Red Hat/OU=Cloud BU/CN=flatline-katello.usersys.redhat.com>, issuer=</C=US/ST=North Carolina/L=Raleigh/O=Red Hat/OU=Cloud BU/CN=flatline-katello.usersys.redhat.com>, subject.as_hash=<1088644680>, issuer.as_hash=<1088644680>, fingerprint=<B32208DB5AD4FA547119D0EA51992B56>, serial=<17830954723505377029>, version=<2>, check_ca=<1>, notBefore=<Mar  2 14:32:16 2012 GMT>, notAfter=<Jan 18 14:32:16 2038 GMT>
[Mon Mar 05 10:30:31 2012] [error] [client 10.11.231.160] Using a CRL Stack with 1 CRL(s)
[Mon Mar 05 10:30:31 2012] [error] [client 10.11.231.160] \tCRL: issuer=</C=US/ST=North Carolina/L=Raleigh/O=Red Hat/OU=Cloud BU/CN=flatline-katello.usersys.redhat.com>, issuer.as_hash=<1088644680> lastUpdate=<Mar  4 17:00:00 2012 GMT>, nextUpdate=<Mar  5 17:00:00 2012 GMT>Client certificate did not match the global repo auth CA certificate
[Mon Mar 05 10:30:31 2012] [error] [client 10.11.231.160] mod_wsgi (pid=14081): Client denied by server configuration: '/var/www/pub/repos/redhat/Stage/content/beta/rhel/server/6/6Server/x86_64/cf-tools/1.0/os/repodata/repomd.xml'.
[Mon Mar 05 10:30:31 2012] [info] [client 10.11.231.160] Connection closed to child 3 with standard shutdown (server flatline-katello.usersys.redhat.com:443)

== Suggested Beta3 workaround ==

# CRL_HASH=$(openssl x509 -subject_hash -in /etc/candlepin/certs/candlepin-ca.crt  | head -n1)
# mv /etc/pki/pulp/content/${CRL_HASH}.r0  /etc/pki/pulp/content/RENAMED_${CRL_HASH}.r0

Comment 3 John Matthews 2012-03-06 20:55:28 UTC
The cause of the CRL not working was the lack of "CRL Sign" under "X509v3 Key Usage".
Example:
X509v3 Key Usage:
  Digital Signature, Key Encipherment, Certificate Sign, CRL Sign

Change is here:
$ git diff
diff --git a/certs-tools/certs/sslToolConfig.py b/certs-tools/certs/sslToolConfig.py
index a8ae8e1..0c31f51 100644
--- a/certs-tools/certs/sslToolConfig.py
+++ b/certs-tools/certs/sslToolConfig.py
@@ -368,9 +368,9 @@ x509_extensions         = req_ca_x509_extensions
 
 [ req_ca_x509_extensions ]
 basicConstraints = CA:true
-keyUsage = digitalSignature, keyEncipherment, keyCertSign
+keyUsage = digitalSignature, keyEncipherment, keyCertSign, cRLSign
 extendedKeyUsage = serverAuth, clientAuth
-nsCertType = server
+nsCertType = server, sslCA
 # PKIX recommendations harmless if included in all certificates.
 nsComment               = "Katello SSL Tool Generated Certificate"
 subjectKeyIdentifier    = hash


With above change:
      X509v3 extensions:
            X509v3 Basic Constraints:
                CA:TRUE
            X509v3 Key Usage:
                Digital Signature, Key Encipherment, Certificate Sign, CRL Sign
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            Netscape Cert Type:
                SSL Server, SSL CA
            Netscape Comment:
                Katello SSL Tool Generated Certificate
            X509v3 Subject Key Identifier:
                97:304:FD:618:C0:23:F2:7B:9A:ED:B16:7F:B2:55:97:BA:40
            X509v3 Authority Key Identifier:
                keyid:97:304:FD:618:C0:23:F2:7B:9A:ED:B16:7F:B2:55:97:BA:40
                DirName:/C=US/ST=North Carolina/L=Raleigh/O=Red Hat/OU=Cloud BU/CN=beta.lan
                serial:81:16:C6:95:B6:39:67:52

Without change:
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:TRUE
            X509v3 Key Usage:
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            Netscape Cert Type:
                SSL Server
            Netscape Comment:
                Katello SSL Tool Generated Certificate
            X509v3 Subject Key Identifier:
                E5:43:F7:80:B0:6A:69:EA:0F:73:E08:771:09:01:70:AF:83:FE
            X509v3 Authority Key Identifier:
                keyid:E5:43:F7:80:B0:6A:69:EA:0F:73:E08:771:09:01:70:AF:83:FE
                DirName:/C=US/ST=North Carolina/L=Raleigh/O=Red Hat/OU=Cloud BU/CN=beta.lan
                serial:B1:4A:85:8B:45:ED:3D:65


For background, this is the command I ran after patching the sslToolConfig.py to examine what the new ca would look like.

katello-ssl-tool --gen-ca -p "$(cat /etc/katello/candlepin_ca_password-file)" --set-country 'US' --set-state 'North Carolina' --set-city 'Raleigh' --set-org 'Red Hat' --set-org-unit 'Cloud BU' --set-common-name `hostname` --set-email '' --ca-key 'candlepin-cert.key' --ca-cert 'candlepin-cert.crt' --ca-cert-rpm  'katello-candlepin-cert-key-pair' -vvv --force

Comment 6 James Laska 2012-03-07 13:43:39 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Due to a bug in how certificates are constructed during System Engine configuration, the certificate restriction list (CRL) blocks all katello-hosted repository access.  This issue will be resolved in a future katello update.  In the meantime, to work around this problem, you can disable CRL support entirely using the following commands:

$ CRL_HASH=$(openssl x509 -subject_hash -in
/etc/candlepin/certs/candlepin-ca.crt  | head -n1)
$ mv /etc/pki/pulp/content/${CRL_HASH}.r0 
/etc/pki/pulp/content/DISABLED_${CRL_HASH}.r0

Comment 7 Mike McCune 2012-03-08 22:27:18 UTC
Fixed in katello-certs-tools-1.0.4-1 and above

Comment 9 Og Maciel 2012-03-09 19:53:19 UTC
Validated:
* candlepin-0.5.24-1.el6.noarch
* candlepin-tomcat6-0.5.24-1.el6.noarch
* katello-0.1.303-1.el6.noarch
* katello-all-0.1.303-1.el6.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.0.4-1.el6.noarch
* katello-cli-0.1.102-1.el6.noarch
* katello-cli-common-0.1.102-1.el6.noarch
* katello-common-0.1.303-1.el6.noarch
* katello-configure-0.1.104-1.el6.noarch
* katello-glue-candlepin-0.1.303-1.el6.noarch
* katello-glue-foreman-0.1.303-1.el6.noarch
* katello-glue-pulp-0.1.303-1.el6.noarch
* katello-qpid-broker-key-pair-1.0-1.noarch
* katello-qpid-client-key-pair-1.0-1.noarch
* katello-selinux-0.1.8-1.el6.noarch
* pulp-1.0.0-4.el6.noarch
* pulp-common-1.0.0-4.el6.noarch
* pulp-selinux-server-1.0.0-4.el6.noarch

Comment 10 James Laska 2012-04-20 13:15:44 UTC
Removing technical note and requires_release_note? flag.  This issue has been fixed and requires *no* any release notes.

Comment 11 James Laska 2012-04-20 13:15:45 UTC
Deleted Technical Notes Contents.

Old Contents:
Due to a bug in how certificates are constructed during System Engine configuration, the certificate restriction list (CRL) blocks all katello-hosted repository access.  This issue will be resolved in a future katello update.  In the meantime, to work around this problem, you can disable CRL support entirely using the following commands:

$ CRL_HASH=$(openssl x509 -subject_hash -in
/etc/candlepin/certs/candlepin-ca.crt  | head -n1)
$ mv /etc/pki/pulp/content/${CRL_HASH}.r0 
/etc/pki/pulp/content/DISABLED_${CRL_HASH}.r0