Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 843494

Summary: SSL issues with custom client-side FQDNs and cert update
Product: Red Hat Update Infrastructure for Cloud Providers Reporter: mkovacik
Component: RHUAAssignee: James Slagle <jslagle>
Status: CLOSED NOTABUG QA Contact: mkovacik
Severity: unspecified Docs Contact:
Priority: high    
Version: 2.1CC: tsanders, whayutin
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-07-27 19:51:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
content cert and ca cert screen log
none
Cds-side certs
none
pulp-server-ca.crt vs repository ca issuer none

Description mkovacik 2012-07-26 13:08:18 UTC
Description of problem:
Having changed the configuration for the CDSes to be able to serve contents via custom client-side FQDNs I've encountered some SSL exceptions with the client. The cause is a key mismatch (observed on CDS httpd ssl error log):
  [Thu Jul 26 06:14:45 2012] [error] [client 209.132.186.34] user /CN=Red Hat Update Infrastructure: authentication failure for "/pulp/repos///content/dist/rhel/rhui/server/6/6Server/x86_64/rhui/2.0/os/repodata/repomd.xml": Password Mismatch


Version-Release number of selected component (if applicable):
RHEL-6.3-RHUI-2.1-20120705.0-Server-x86_64-DVD1.iso

How reproducible:
1 of 1

Steps to Reproduce:
1. deploy rhui with 2 CDSes in EC2 using private DNS records for clients to access contents 
2. change your mind and re-configure the deployment to use public DNS records as custom CDS client-side FQDNs (generate new keys for all nodes, generate and apply new configuration rpms on respective nodes)
3. try accessign the contents via public DNS (requires a client outside of EC2)
  
Actual results:
### having disabled all but one repo; same error for any of them

[root@dhcp-31-102 ~]# yum repolist
Loaded plugins: product-id, rhui-lb, subscription-manager
Updating Red Hat repositories.
https://ec2-204-236-245-246.compute-1.amazonaws.com/pulp/repos///content/dist/rhel/rhui/server/6/6Server/x86_64/rhui/2.0/os/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 401"
Trying other mirror.
https://ec2-23-22-69-164.compute-1.amazonaws.com/pulp/repos///content/dist/rhel/rhui/server/6/6Server/x86_64/rhui/2.0/os/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 401"
Trying other mirror.
repo id                                                                                                                      repo name                                                                                                                               status
rhui-rhel-x86_64-6-rhui-2-rpms                                                                                               Red Hat Update Infrastructure 2.0 (RPMs)                                                                                                0
repolist: 0


Additional info:
### PULP info about the repository

[root@ip-10-80-226-7 ~]# pulp-admin -p admin -u admin repo info --id rhel-x86_64-6-rhui-2-rpms-6Server-x86_64

Id                      rhel-x86_64-6-rhui-2-rpms-6Server-x86_64
Name                    Red Hat Update Infrastructure 2.0 (RPMs) (6Server-x86_64)
Repo URL                ec2-204-236-245-246.compute-1.amazonaws.com
Feed URL                https://cdn.redhat.com//content/dist/rhel/rhui/server/6/6Server/x86_64/rhui/2.0/os
Feed Type               remote                   
Content Type            yum                      
Feed Certs              CA:Yes   Cert:Yes
Consumer Certs          CA:Yes   Cert:Yes
Architecture            noarch                   
Sync Schedule           2012-07-19T16:13:10-04:00/PT6H
Packages                37                       
Files                   0                        
Distributions           None                     
Publish                 True                     
Clones                  []                       
Groups                  [u'redhat', u'rhel-x86_64-6-rhui-2-rpms']
Filters                 []                       
Notes                   {}                       
Preserve Metadata       True                     
Checksum Type           sha256                   

[root@dhcp-31-102 ~]# cat /etc/yum.repos.d/rh-cloud.repo
[rhui-custom-10001]
name=Custom Repositories - 10001
mirrorlist=https://ec2-23-22-69-164.compute-1.amazonaws.com/pulp/mirror/gkrellm/$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
sslverify=1
sslcacert=/etc/pki/entitlement/ca.crt
sslclientcert=/etc/pki/entitlement/product/content.crt
sslclientkey=/etc/pki/entitlement/key.pem

[rhui-rhel-6-rhui-server-rpms]
name=Red Hat Enterprise Linux 6 Server (RPMs) from RHUI
mirrorlist=https://ec2-23-22-69-164.compute-1.amazonaws.com/pulp/mirror//content/dist/rhel/rhui/server/6/$releasever/$basearch/os
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
sslverify=1
sslcacert=/etc/pki/entitlement/ca.crt
sslclientcert=/etc/pki/entitlement/product/content.crt
sslclientkey=/etc/pki/entitlement/key.pem

[rhui-rhel-x86_64-6-rhui-2-rpms]
name=Red Hat Update Infrastructure 2.0 (RPMs)
mirrorlist=https://ec2-23-22-69-164.compute-1.amazonaws.com/pulp/mirror//content/dist/rhel/rhui/server/6/$releasever/$basearch/rhui/2.0/os
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
sslverify=1
sslcacert=/etc/pki/entitlement/ca.crt
sslclientcert=/etc/pki/entitlement/product/content.crt
sslclientkey=/etc/pki/entitlement/key.pem

[rhui-custom-10000]
name=Custom Repositories - 10000
mirrorlist=https://ec2-23-22-69-164.compute-1.amazonaws.com/pulp/mirror/bhello/$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
sslverify=1
sslcacert=/etc/pki/entitlement/ca.crt
sslclientcert=/etc/pki/entitlement/product/content.crt
sslclientkey=/etc/pki/entitlement/key.pem

[root@dhcp-31-102 ~]# openssl x509 -in /etc/pki/entitlement/ca.crt -noout -subject
subject= /C=US/ST=NC/L=Raleigh/CN=ip-10-80-226-7.ec2.internal CA
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=NC, L=Raleigh, CN=ip-10-80-226-7.ec2.internal CA
        Validity
            Not Before: Jul 26 08:55:26 2012 GMT
            Not After : Jul 26 08:55:26 2013 GMT
        Subject: CN=Red Hat Update Infrastructure
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d7:aa:5a:f6:95:87:1c:5d:ea:18:ad:cf:50:a2:
                    9a:fd:d0:dd:b3:83:95:06:d9:03:3c:ab:4f:de:20:
                    86:3c:95:fa:6e:5b:f7:bc:05:e6:27:ec:95:7b:cb:
                    d5:e7:df:63:e2:e3:97:be:b3:20:16:bf:ce:eb:ae:
                    ca:72:8f:8c:4e:e7:95:04:b3:64:81:19:86:2d:20:
                    34:8c:ad:6f:a6:85:6b:c3:d1:2f:cb:0f:ee:df:bb:
                    19:6f:b4:91:4b:27:0e:ce:d7:2a:8e:aa:b7:aa:59:
                    53:f7:46:a1:79:6a:5b:07:66:97:f4:2b:20:ff:d8:
                    ad:7f:83:3b:02:7e:11:75:64:79:6a:e4:3b:23:db:
                    e7:d4:a9:1e:0c:2c:99:58:cb:53:04:32:e6:d9:a8:
                    4f:e0:8f:f6:ef:1a:a1:3c:c6:f7:cd:e6:72:01:7b:
                    0e:ac:48:d9:fc:7c:6e:90:ad:46:e0:ac:1d:fa:26:
                    58:17:01:e9:db:25:e1:c6:75:03:f2:49:e4:a3:58:
                    da:07:89:ac:aa:81:cb:f9:62:8b:ef:97:61:01:a5:
                    e8:43:b4:72:9e:fc:36:3d:23:f1:1a:61:23:29:f4:
                    da:a8:05:03:50:29:46:c9:76:cb:52:9e:4d:2f:df:
                    7f:b6:12:8b:f7:fd:35:a2:b9:46:ba:5f:02:20:77:
                    b9:fd
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            1.3.6.1.4.1.2312.9.2.10000.1.1: 
                ..Custom Repositories - 10000
            1.3.6.1.4.1.2312.9.2.10000.1.2: 
                ..custom-10000
            1.3.6.1.4.1.2312.9.2.10000.1.6: 
                ..bhello/$basearch
            1.3.6.1.4.1.2312.9.2.10001.1.1: 
                ..Custom Repositories - 10001
            1.3.6.1.4.1.2312.9.2.10001.1.2: 
                ..custom-10001
            1.3.6.1.4.1.2312.9.2.10001.1.6: 
                ..gkrellm/$basearch
            1.3.6.1.4.1.2312.9.2.1166.1.1: 
                .2Red Hat Enterprise Linux 6 Server (RPMs) from RHUI
            1.3.6.1.4.1.2312.9.2.1166.1.2: 
                ..rhel-6-rhui-server-rpms
            1.3.6.1.4.1.2312.9.2.1166.1.6: 
                .9/content/dist/rhel/rhui/server/6/$releasever/$basearch/os
            1.3.6.1.4.1.2312.9.2.1074.1.1: 
                .(Red Hat Update Infrastructure 2.0 (RPMs)
            1.3.6.1.4.1.2312.9.2.1074.1.2: 
                ..rhel-x86_64-6-rhui-2-rpms
            1.3.6.1.4.1.2312.9.2.1074.1.6: 
                .B/content/dist/rhel/rhui/server/6/$releasever/$basearch/rhui/2.0/os
    Signature Algorithm: sha1WithRSAEncryption
        7a:b5:15:d8:e9:e6:5f:38:54:6e:4f:a5:43:66:8b:9c:03:24:
        34:36:c4:90:84:31:a5:aa:ee:4b:e3:53:d5:f5:4b:74:96:56:
        52:5d:ad:9a:d0:aa:82:6c:78:b4:92:e4:3d:85:7d:83:f9:98:
        43:ab:1f:c9:a8:45:f9:39:1b:e4:32:13:31:e9:2e:5c:d8:65:
        0d:af:e3:ea:58:25:24:89:de:7b:c9:d5:79:4a:c5:19:8f:83:
        93:29:23:0e:7a:f8:6f:bd:ed:65:74:d2:39:23:3c:c3:0c:40:
        ae:84:19:d9:81:2b:1a:65:90:59:35:1e:6f:98:60:fd:67:dd:
        57:b3:6f:2b:bd:19:f0:85:5c:2d:fe:c5:7a:81:ed:7d:f7:ad:
        f2:40:8b:83:8a:00:f2:ca:4d:da:84:8f:60:25:40:3c:5a:1b:
        93:3a:c9:47:4c:9c:0b:dd:43:5c:c3:bb:38:99:7b:25:8d:74:
        2f:c4:d7:05:b0:8a:08:03:9e:f4:1f:2c:db:9e:f3:d4:ac:3c:
        22:86:31:6f:8f:b0:e7:a8:ee:f0:78:5b:b9:49:29:96:8f:03:
        eb:16:3c:23:67:5b:64:f1:45:19:2f:a8:de:a9:9d:52:cc:77:
        02:fc:b6:ad:a6:bd:68:46:4c:ff:28:cd:b4:9e:f9:b8:a3:6a:
        df:53:e1:a0
[root@dhcp-31-102 ~]#

Comment 2 James Slagle 2012-07-26 16:39:25 UTC
Something strange is going on here.  You show the contents of /etc/pki/entitlement/ca.crt, which is the CA cert, but that has entitlement paths in it like that of an entitlement cert.

What does /etc/pki/entitlement/product/content.crt look like?

Comment 3 mkovacik 2012-07-26 17:03:33 UTC
Created attachment 600557 [details]
content cert and ca cert screen log

Attached the requested in a screen log

Comment 4 James Slagle 2012-07-27 15:46:58 UTC
I think this is a certificate misconfiguration issue.

Can you attach the following files from the CDS:
/etc/pki/pulp/content/rhui-2.0-6Server-x86_64/consumer-rhui-2.0-6Server-x86_64.cert
/etc/pki/pulp/content/rhui-2.0-6Server-x86_64/consumer-rhui-2.0-6Server-x86_64.ca

Comment 5 James Slagle 2012-07-27 15:50:11 UTC
Also, did you generate a new entitlement cert and then use that in rhui-manager?  Meaning, did you delete the old one from /etc/pki/rhui/entitlement-ca.crt and upload the new one on the next start with rhui-manager?

Comment 6 mkovacik 2012-07-27 17:33:00 UTC
(In reply to comment #5)
> Also, did you generate a new entitlement cert and then use that in
> rhui-manager?  Meaning, did you delete the old one from
> /etc/pki/rhui/entitlement-ca.crt and upload the new one on the next start
> with rhui-manager?

I think so, yes; I do follow steps in this paragraph: https://engineering.redhat.com/trac/IntegratedMgmtQE/wiki/RHUI_2.0_setup#ec:rh20:replacing-certificates which includes removing the file you mention.

Comment 7 mkovacik 2012-07-27 17:51:21 UTC
Created attachment 600841 [details]
Cds-side certs

Attaching what requested in comment 4. Seems the issuer CN is localhost unlike of what is present in client config rpm (CN=ip-10-80-226-7.ec2.internal CA)...

Comment 8 James Slagle 2012-07-27 18:50:53 UTC
The CA that signed the client configuration rpm does not match the CDS expects.

Did you regenerate client config rpm's after updating the entitlement CA?
Did you do a CDS sync after updating the entitlement CA?

Comment 9 mkovacik 2012-07-27 18:54:21 UTC
Well, I can't tell; let me try generating fresh config and sysncing CDSes... Hopefully that's the issue...

Comment 10 mkovacik 2012-07-27 19:15:51 UTC
Created attachment 600860 [details]
pulp-server-ca.crt vs repository ca issuer

The sync didn't help; I've noticed that /etc/pki/pulp/pulp-server-ca.crt CN is different than the repo cert issuer CN (localhost) on the CDS1; but these files aren't owned by any package, so they have to be deployed by some other mechanizm (sync probably; see the attachment)

Comment 11 James Slagle 2012-07-27 19:51:37 UTC
Ended up logging onto Milan's machines and just walking through the procedure to update the entitlement CA again.  Not sure why it didn't work the first time.  Given we've seen it work many times, going to close this as NOTABUG.