Description of problem: The OpenShift CLI doesn't seem to trust Let's Encrypt certificates. Our cluster (https://master.na39.openshift.opentlc.com) has Let's Encrypt certificates on both Master and Router. This works fine in browser but on the command line there still is a warning that the certificate is from an unknown authority. When everyone else (including Google, Microsoft, Firefox, etc) has started trusting Let's Encrypt certificates maybe it's time for us to do the same. Version-Release number of selected component (if applicable): 3.10.14 (but really every OpenShift release). How reproducible: Every time Steps to Reproduce: 1. oc login -u wkulhane-redhat.com https://master.na39.openshift.opentlc.com Actual results: ➭ oc login -u wkulhane-redhat.com https://master.na39.openshift.opentlc.com The server uses a certificate signed by an unknown authority. You can bypass the certificate check, but any data you send to the server could be intercepted by others. Use insecure connections? (y/n): Expected results: No warning. Additional info:
The CLI is erroring because it is seeing a self signed certificate: $ curl -vvvvvv https://master.na39.openshift.opentlc.com * Rebuilt URL to: https://master.na39.openshift.opentlc.com/ * Trying 54.190.206.122... * TCP_NODELAY set * Connected to master.na39.openshift.opentlc.com (54.190.206.122) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (OUT), TLS alert, Server hello (2): * SSL certificate problem: unable to get local issuer certificate * stopped the pause stream! * Closing connection 0 curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. $ openssl s_client -showcerts -connect master.na39.openshift.opentlc.com:443 </dev/null CONNECTED(00000003) depth=1 CN = openshift-signer@1528704451 verify error:num=19:self signed certificate in certificate chain --- Certificate chain 0 s:/CN=172.30.0.1 i:/CN=openshift-signer@1528704451 -----BEGIN CERTIFICATE----- MIIEYzCCA0ugAwIBAgIBBDANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu c2hpZnQtc2lnbmVyQDE1Mjg3MDQ0NTEwHhcNMTgwNjExMDgwNzMyWhcNMjAwNjEw MDgwNzMzWjAVMRMwEQYDVQQDEwoxNzIuMzAuMC4xMIIBIjANBgkqhkiG9w0BAQEF AAOCAQ8AMIIBCgKCAQEA6Sg2+QrqR5YZoDVVmjRYXUo8pwOoCPgJj6xQ6Cjc6Qqv Om5ObrwQ5yKOh5qooBEviUHerwCie8noHL+yGkLKaQIQ2WchCPR35T38IQMt9mB3 3Uk5azQFzkajtx3x0FeMLM7gMenT5lrGBk1X51jatg52YXz6BlVyRIF+HEAP/5mW I2h5qqX9pESmG3+7ufiME7Pzwydur0PG6IdWybfaaW2C54KTb/1vp8dOuUOSkMeo +tjyZ9hb1TNh+NmILHGxqMfjGEu4oJdKgBYLqh+PP9Lda3ooyAJCeaKOeXSXWENU xHiZuH+T+iDGjBSKZa6dFv6D09WdPmagAd9Ue6cf/wIDAQABo4IBqzCCAacwDgYD VR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAw ggFwBgNVHREEggFnMIIBY4IyZWMyLTU0LTE5MC0yMDYtMTIyLnVzLXdlc3QtMi5j b21wdXRlLmFtYXpvbmF3cy5jb22CCmt1YmVybmV0ZXOCEmt1YmVybmV0ZXMuZGVm YXVsdIIWa3ViZXJuZXRlcy5kZWZhdWx0LnN2Y4Ika3ViZXJuZXRlcy5kZWZhdWx0 LnN2Yy5jbHVzdGVyLmxvY2FsgiFtYXN0ZXIubmEzOS5vcGVuc2hpZnQub3BlbnRs Yy5jb22CFW1hc3RlcjEubmEzOS5pbnRlcm5hbIIJb3BlbnNoaWZ0ghFvcGVuc2hp ZnQuZGVmYXVsdIIVb3BlbnNoaWZ0LmRlZmF1bHQuc3ZjgiNvcGVuc2hpZnQuZGVm YXVsdC5zdmMuY2x1c3Rlci5sb2NhbIIKMTcyLjMwLjAuMYINMTkyLjE5OS4wLjI1 MYIONTQuMTkwLjIwNi4xMjKHBKweAAGHBMDHAPuHBDa+znowDQYJKoZIhvcNAQEL BQADggEBALJxZJSLbkyOlOuK3FxVIhZxfYMwStRr6hAguSPHj6ifltGSH+E2qgED nZKxkoi2pydESTKRH9Pxq4nIbOvMxF7vjbrIEqrX6lOnfCChIcsPchZTi/JN/LvE DMCID+9CiVcmQHv3cp7G7rfDUzZL1bGt6GbBFOCP7EBBFpuytlFgCg8brPprNi4e 45OeapR/rRQ8spBo7/DG6xpEIZNeBZQiWL6+DNLOSX0lpn++aXX3/fl46nEa5On3 KhjKAJNbRU+Zn/Xwy3UKWrbjibEyic0ToIlAEuuoIiL5VjXcm0Fr7rjWbUoaGfKc bTJeEoXW8c1K2DoYu+9M/+/45r5rNXs= -----END CERTIFICATE----- 1 s:/CN=openshift-signer@1528704451 i:/CN=openshift-signer@1528704451 -----BEGIN CERTIFICATE----- MIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu c2hpZnQtc2lnbmVyQDE1Mjg3MDQ0NTEwHhcNMTgwNjExMDgwNzMxWhcNMjMwNjEw MDgwNzMyWjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1Mjg3MDQ0NTEw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDrZ1wzn2TYRjcBt9MSCR/s y/QIQDgZFpHG5pHM78hc1NtMdaKDhVN2LlNHP6KOBrPP3GNpRNuJFcy75HdkBqvd Ad/UIju1Y9uFNYOxai2L3rdhTXobSXzvSdIJ/NR08jO4bDENzSdqKqmJ1M/JNxK/ O9f9vWMheI66BkT5HmSASo8NQucvoDZGqWLocLASbzkOuf3+E2nlfXfu0lBlxYwe wrfVJRkGAkw7zL7wJWt37JggWI5G2sZA4U+winNi1DNte+rM4CtFNe4dLyuSoqEX Xu6rSS+YFG/RPzF71EH7xYFs/ZLZ1TSxxE1QmuPE+CPO4rN2vCHVRcQ5CFg65wE3 AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG SIb3DQEBCwUAA4IBAQCeSesmBeSPTUrTwVaMU9kgsSh8gjO3bsudtRzVhKsxiV/L 1g2DpZO/VmsgnGHOOFhSsRzRnSCMS/EXOA0OIbnQ4DDjYXdJE9Xd6sM1nI5HVYAD QcNEEN/J0EnYS+dvSoVdVnTCTNeTC+w9SSnFLZw9JVi7sJqsHdBNMSF0YJualLIV FYm1N9QlL7Y4yI/RrHmBxdbnPnLcPBaO41ykEDmx33g2i+j0jjdOq+aPgA4nSWmu MZqT9zF+moklM97CmFfLoj+VhnzCXm3k+npV3fUSQ3rqxbi1DdBiLlW5zlqAtw2Z RARvLSPF2dkwEuJagEk/JVMv6NnxEdP15btYF8kq -----END CERTIFICATE----- --- Server certificate subject=/CN=172.30.0.1 issuer=/CN=openshift-signer@1528704451 --- Acceptable client certificate CA names /CN=openshift-signer@1528704451 /CN=openshift-signer@1528704538 Client Certificate Types: RSA sign, ECDSA sign Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1 Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA1:ECDSA+SHA1 Peer signing digest: SHA384 Server Temp Key: X25519, 253 bits --- SSL handshake has read 2557 bytes and written 321 bytes Verification error: self signed certificate in certificate chain --- New, TLSv1.2, Cipher is ECDHE-RSA-CHACHA20-POLY1305 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-CHACHA20-POLY1305 Session-ID: C04F44352657D3718AE4CB5A321B71C37A74BDAC1E4BF180D61D19E3244D0824 Session-ID-ctx: Master-Key: D28EACAA981EC5113FE152FE77BBE1FC9E12E3B1E566C4C6547C4C02D1A9351B84F6B8A3C8D693303F9B11DF799C4228 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket: 0000 - 67 21 08 52 ed a6 22 74-8b c1 b9 ac a3 98 2f 97 g!.R.."t....../. 0010 - dc e0 20 5e 19 96 cf c1-58 04 53 f3 76 79 67 b8 .. ^....X.S.vyg. 0020 - 45 62 fd 95 fe 5f be c9-a7 fc 15 26 6c da fe ac Eb..._.....&l... 0030 - 64 62 92 2a 6d 02 1d cb-8f 96 cb 6a 93 bf 93 bd db.*m......j.... 0040 - aa 99 c0 cb 64 07 f7 8f-8e 84 0f ea d2 85 28 a9 ....d.........(. 0050 - 25 e2 7b 26 22 8d 88 67-f9 f0 57 e3 23 1b 65 a9 %.{&"..g..W.#.e. 0060 - bb bf b5 8c 44 29 4c 3a-ef 6e a4 37 b6 3a 9c 41 ....D)L:.n.7.:.A 0070 - c6 1c aa 8c 03 25 54 71- .....%Tq Start Time: 1534275502 Timeout : 7200 (sec) Verify return code: 19 (self signed certificate in certificate chain) Extended master secret: no --- DONE I assume you have something configured to use a different certificate with browsers as I can correctly see your Let's Encrypt cert with the browser. Note that we just use your local cert store or CA data in your config file to verify connections. Thus if the CLI was actually seeing the Let's Encrypt cert, it would not error.
This is quite odd. We are setting the master (and router) certificates during installation. Host file: openshift_master_named_certificates=[{"certfile": "/root/.acme.sh/master.na39.openshift.opentlc.com/master.na39.openshift.opentlc.com.cer", "keyfile": "/root/.acme.sh/master.na39.openshift.opentlc.com/master.na39.openshift.opentlc.com.key", "cafile": "/root/lets-encrypt-x3-cross-signed.pem"}] openshift_hosted_router_certificate={"certfile": "/root/.acme.sh/master.na39.openshift.opentlc.com/master.na39.openshift.opentlc.com.cer", "keyfile": "/root/.acme.sh/master.na39.openshift.opentlc.com/master.na39.openshift.opentlc.com.key", "cafile": "/root/lets-encrypt-x3-cross-signed.pem"} How would we set a different certificate for the API than for the web console? I went through the docs briefly but didn't see anything. Is this a bug in the installer maybe then?
Only thing I can think of is to try setting a name on your named certificate config, I wonder if it's getting confused by the wildcard in the SAN. See the names field here https://docs.openshift.com/container-platform/3.9/install_config/certificate_customization.html#configuring-custom-certificates Also, we generally try to use different hostname for the named certificate than what the internal cluster components use. so master.na39.openshift.opentlc.com should only be specified for masterPublicURL etc, whereas everything else should be a different hostname so that the API server can present the correct CA.
Wolfgang, please reassign to the correct component or close if not relevant anymore
Well to be honest I'm not sure what to do with this. Doesn't seem there is another way to set host names and certs
Really sounds like a configuration or intaller issue, reassigning
This seems to be related to that issue here https://community.letsencrypt.org/t/curl-refuses-to-accept-my-cert-saying-the-certificate-issuer-is-not-recognized/40917/3 "The certificate chain is incomplete. (Chrome and Firefox only accept it because they’ve cached the Let’s Encrypt intermediate from visits to other websites.)" You need to provide the path to the fullchain certificate here openshift_master_named_certificates=[{"certfile":....}] You can get it with acme.sh using the --install-cert argument, e.g.: acme.sh --install-cert -d example.org [...] --fullchain-file /etc/acme.sh/domains/fullchain.pem
I think you would need to either use the full chain like Nikolas suggests, or use the ISRG signed X3 CA in the cafile parameter instead of the Identrust cross-signed CA. The cross-signed CA is an incomplete chain if used by itself.
Wolfgang, Can you confirm that comment 7 and comment 8 address the issue for you?
Closing for now as this appears to be a certificate trust chain problem that can be addressed via configuration changes.
That did infact fix it. The acme.sh script already put a fullcert.cer into the $HOME/.acme.sh/{domain} directory - alongside the "basic" server certificate. Using the fullcert.cer as the certificate for master and router now removes the insecure connection error on 'oc login'. Thanks for the hint!!
Just as a side note: According to https://github.com/Neilpang/acme.sh#3-install-the-cert-to-apachenginx-etc you should not use $HOME/.acme.sh/{domain} "DO NOT use the certs files in ~/.acme.sh/ folder, they are for internal use only, the folder structure may change in the future." You should install them. Please consider this to prevent future issues.
Except.... that won't work when using the certs for an OCP Cluster from the installer playbooks, will it?
I use it that way. If you're running acme.sh on your installer node (the one with the openshift-ansible playbooks), you could do it kind of that way: First, issue the certs (you've already done that). As you're using AWS dns export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY export AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN acme.sh --issue --dns dns_aws -d master.na39.openshift.opentlc.com -d *.apps.na39.openshift.opentlc.com This put's the certs into ~/.acme.sh/, but this is a acme.sh implementation details. Then install the certs somewhere where you can reference them from your ansible inventory file mkdir -p /etc/acme.sh/domain acme.sh --install-cert -d master.na39.openshift.opentlc.com -d *.apps.na39.openshift.opentlc.com \ --cert-file /etc/acme.sh/domain/cert.pem \ --key-file /etc/acme.sh/domain/key.pem \ --fullchain-file /etc/acme.sh/domain/fullchain.pem \ --ca-file /etc/acme.sh/domain/ca.cer \ --reloadcmd "cd /path/to/playbooks/ && ansible-playbook redeploy-certificates.yml" The cron job of acme.sh remebers the place where you've installed the certificates and updates them accordingly. You just need to make sure that the user running acme.sh can execute the command in --reloadcmd (maybe it's the simplest when you use the same as for the ansible installer) Then ajust the inventory configurations openshift_master_named_certificates=[{"certfile": "/etc/acme.sh/domain/fullchain.pem", "keyfile": "/etc/acme.sh/domain/key.pem", "names": ["master.na39.openshift.opentlc.com"], "cafile": "/etc/acme.sh/domain/ca.cer"}] openshift_hosted_router_certificate={"certfile": "/etc/acme.sh/domain/fullchain.pem", "keyfile": "/etc/acme.sh/domain/key.pem", "cafile": "/etc/acme.sh/domain/ca.cer"} openshift_hosted_registry_routecertificates={"certfile": "/etc/acme.sh/domain/fullchain.pem", "keyfile": "/etc/acme.sh/domain/key.pem", "cafile": "/etc/acme.sh/domain/ca.cer"} If you're running the acme.sh script on a different node you need a way to start the redeploy-certificates playbook differently.
Ah. Cool. Did not know about that (obviously). Will update my playbooks.