Description of problem: The logging deploy failed at task 'openshift_logging : Run JKS generation script' Version-Release number of selected component (if applicable): openshift-ansible-3.9.22 How reproducible: always Steps to Reproduce: 1.Deploy logging 2. 3. Actual results: TASK [openshift_logging : pulling down signing items from host] **************** changed: [openshift-181.lab.eng.nay.redhat.com] => (item=ca.crt) changed: [openshift-181.lab.eng.nay.redhat.com] => (item=ca.key) changed: [openshift-181.lab.eng.nay.redhat.com] => (item=ca.serial.txt) changed: [openshift-181.lab.eng.nay.redhat.com] => (item=ca.crl.srl) changed: [openshift-181.lab.eng.nay.redhat.com] => (item=ca.db) TASK [openshift_logging : template] ******************************************** changed: [openshift-181.lab.eng.nay.redhat.com -> localhost] TASK [openshift_logging : Run JKS generation script] *************************** fatal: [openshift-181.lab.eng.nay.redhat.com -> localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "+ '[' 4 -lt 1 ']'\n+ dir=/tmp/openshift-logging-ansible-jODsUS\n+ SCRATCH_DIR=/tmp/openshift-logging-ansible-jODsUS\n+ PROJECT=logging\n+ MORE_ES_NAMES=es.0416-8p6.qe.rhcloud.com\n+ escomma=,\n+ MORE_ES_OPS_NAMES=es-ops.0416-8p6.qe.rhcloud.com\n+ esopscomma=,\n+ [[ ! -f /tmp/openshift-logging-ansible-jODsUS/system.admin.jks ]]\n+ generate_JKS_client_cert system.admin\n+ NODE_NAME=system.admin\n+ ks_pass=kspass\n+ ts_pass=tspass\n+ dir=/tmp/openshift-logging-ansible-jODsUS\n+ echo Generating keystore and certificate for node system.admin\n+ keytool -genkey -alias system.admin -keystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -keyalg RSA -keysize 2048 -validity 712 -keypass kspass -storepass kspass -dname 'CN=system.admin, OU=OpenShift, O=Logging'\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -deststoretype pkcs12\".\n+ echo Generating certificate signing request for node system.admin\n+ keytool -certreq -alias system.admin -keystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -file /tmp/openshift-logging-ansible-jODsUS/system.admin.jks.csr -keyalg rsa -keypass kspass -storepass kspass -dname 'CN=system.admin, OU=OpenShift, O=Logging'\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -deststoretype pkcs12\".\n+ echo Sign certificate request with CA\n+ openssl ca -in /tmp/openshift-logging-ansible-jODsUS/sys tem.admin.jks.csr -notext -out /tmp/openshift-logging-ansible-jODsUS/system.admin.jks.crt -config /tmp/openshift-logging-ansible-jODsUS/signing.conf -extensions v3_req -batch -extensions server_ext\nUsing configuration from /tmp/openshift-logging-ansible-jODsUS/signing.conf\nCheck that the request matches the signature\nSignature ok\nCertificate Details:\n Serial Number: 6 (0x6)\n Validity\n Not Before: Apr 16 06:59:26 2018 GMT\n Not After : Apr 15 06:59:26 2020 GMT\n Subject:\n organizationName = Logging\n organizationalUnitName = OpenShift\n commonName = system.admin\n X509v3 extensions:\n X509v3 Key Usage: critical\n Digital Signature, Key Encipherment\n X509v3 Basic Constraints: \n CA:FALSE\n X509v3 Extended Key Usage: \n TLS Web Server Authentication, TLS Web Client Authentication\n X509v3 Subject Key Identifier: \n 5D:12:E8:DD:E8:24:7B:A6:02:81:B3:E9:5C:68:73:4F:5E:22:16:D2\n X509v3 Authority Key Identifier: \n 0.\nCertificate is to be certified until Apr 15 06:59:26 2020 GMT (730 days)\n\nWrite out database with 1 new entries\nData Base Updated\n+ echo 'Import back to keystore (including CA chain)'\n+ keytool -import -file /tmp/openshift-logging-ansible-jODsUS/ca.crt -keystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -storepass kspass -noprompt -alias sig-ca\nCertificate was added to keystore\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -deststoretype pkcs12\".\n+ keytool -import -file /tmp/openshift-logging-ansible-jODsUS/system.admin.jks.crt -keystore /tmp /openshift-logging-ansible-jODsUS/system.admin.jks -storepass kspass -noprompt -alias system.admin\nCertificate reply was installed in keystore\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -deststoretype pkcs12\".\n+ echo All done for system.admin\n+ [[ ! -f /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks ]]\n++ join , logging-es logging-es-ops\n++ local IFS=,\n++ shift\n++ echo logging-es,logging-es-ops\n+ generate_JKS_chain true elasticsearch logging-es,logging-es-ops\n+ dir=/tmp/openshift-logging-ansible-jODsUS\n+ ADD_OID=true\n+ NODE_NAME=elasticsearch\n+ CERT_NAMES=logging-es,logging-es-ops\n+ ks_pass=kspass\n+ ts_pass=tspass\n+ rm -rf elasticsearch\n+ extension_names=\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es-ops\n+ '[' true = true ']'\n+ extension_names=,dns:logging-es,dns:logging-es-ops,oid:1.2.3.4.5.5\n+ echo Generating keystore and certificate for node elasticsearch\n+ keytool -genkey -alias elasticsearch -keystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -keypass kspass -storepass kspass -keyalg RSA -keysize 2048 -validity 712 -dname 'CN=elasticsearch, OU=OpenShift, O=Logging' -ext san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es-ops,oid:1.2.3.4.5.5\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -deststoretype pkcs12\".\n+ echo Generating certificate signing request for node elasticsearch\n+ keytool -cer treq -alias elasticsearch -keystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -storepass kspass -file /tmp/openshift-logging-ansible-jODsUS/elasticsearch.csr -keyalg rsa -dname 'CN=elasticsearch, OU=OpenShift, O=Logging' -ext san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es-ops,oid:1.2.3.4.5.5\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -deststoretype pkcs12\".\n+ echo Sign certificate request with CA\n+ openssl ca -in /tmp/openshift-logging-ansible-jODsUS/elasticsearch.csr -notext -out /tmp/openshift-logging-ansible-jODsUS/elasticsearch.crt -config /tmp/openshift-logging-ansible-jODsUS/signing.conf -extensions v3_req -batch -extensions server_ext\nUsing configuration from /tmp/openshift-logging-ansible-jODsUS/signing.conf\nCheck that the request matches the signature\nSignature ok\nCertificate Details:\n Serial Number: 7 (0x7)\n Validity\n Not Before: Apr 16 06:59:27 2018 GMT\n Not After : Apr 15 06:59:27 2020 GMT\n Subject:\n organizationName = Logging\n organizationalUnitName = OpenShift\n commonName = elasticsearch\n X509v3 extensions:\n X509v3 Key Usage: critical\n Digital Signature, Key Encipherment\n X509v3 Basic Constraints: \n CA:FALSE\n X509v3 Extended Key Usage: \n TLS Web Server Authentication, TLS Web Client Authentication\n X509v3 Subject Key Identifier: \n CD:22:F3:75:83:E6:D5:E3:9A:2C:A8:75:6C:AD:DA:0B:BB:1A:16:1F\n X509v3 Authority Key Identifier: \n 0.\n X509v3 Subject Alternative Name: \n DNS:localhost, IP Address:127.0.0.1, DNS:logging-es, DNS:logging-es-ops, Registered ID:1.2.3.4.5.5\nCertificate is to be certified until Apr 15 06:59:27 2020 GMT (730 days)\n\nWrite out database with 1 new entries\nData Base Updated\n+ echo 'Import back to keystore (including CA chain)'\n+ keytool -import -file /tmp/openshift-logging-ansible-jODsUS/ca.crt -keystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -storepass kspass -noprompt -alias sig-ca\nCertificate was added to keystore\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -deststoretype pkcs12\".\n+ keytool -import -file /tmp/openshift-logging-ansible-jODsUS/elasticsearch.crt -keystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -storepass kspass -noprompt -alias elasticsearch\nCertificate reply was installed in keystore\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -deststoretype pkcs12\".\n+ echo All done for elasticsearch\n+ [[ ! -f /tmp/openshift-logging-ansible-jODsUS/logging-es.jks ]]\n++ join , logging-es logging-es.logging.svc.cluster.local logging-es-cluster logging-es-cluster.logging.svc.cluster.local logging-es-ops logging-es-ops.logging.svc.cluster.local logging-es-ops-cluster logging-es-ops-cluster.logging.svc.cluster.local\n++ local IFS=,\n++ shift\n++ echo logging-es,logging-es.logging.svc.cluster.local,logging-es-cluster,logging-es-cluster.logging.svc.cluster.local,logging-es-ops,logging-es-ops.logging.svc.cluster.local,logging-es-ops-cluster,logging-es-ops-cluster.logging.svc.clu ster.local\n+ generate_JKS_chain false logging-es logging-es,logging-es.logging.svc.cluster.local,logging-es-cluster,logging-es-cluster.logging.svc.cluster.local,logging-es-ops,logging-es-ops.logging.svc.cluster.local,logging-es-ops-cluster,logging-es-ops-cluster.logging.svc.cluster.local,es.0416-8p6.qe.rhcloud.com,es-ops.0416-8p6.qe.rhcloud.com\n+ dir=/tmp/openshift-logging-ansible-jODsUS\n+ ADD_OID=false\n+ NODE_NAME=logging-es\n+ CERT_NAMES=logging-es,logging-es.logging.svc.cluster.local,logging-es-cluster,logging-es-cluster.logging.svc.cluster.local,logging-es-ops,logging-es-ops.logging.svc.cluster.local,logging-es-ops-cluster,logging-es-ops-cluster.logging.svc.cluster.local,es.0416-8p6.qe.rhcloud.com,es-ops.0416-8p6.qe.rhcloud.com\n+ ks_pass=kspass\n+ ts_pass=tspass\n+ rm -rf logging-es\n+ extension_names=\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.lo gging.svc.cluster.local,dns:logging-es-ops-cluster\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local,dns:es.0416-8p6.qe.rhcloud.com\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local,dns:es.0416-8p6.qe.rhcloud.com,dns:es-ops.0416-8p6.qe.rhcloud.com\n+ '[' false = true ']'\n+ echo Generating keystore and certificate for node logging-es\n+ keytool -genkey -alias logging-es -keystore /tmp/openshift-logging-ansible-jODsUS/logging-es.jks -keypass kspass -storepass kspass -keyalg RSA -keysize 2048 -validity 712 -dname 'CN=logging-es, OU=OpenShift, O=Logging' -ext san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local,dns:es.0416-8p6.qe.rhcloud.com,dns:es-ops.0416-8p6.qe.rhcloud.com\n", "stdout": "Generating keystore and certificate for node system.admin\nGenerating certificate signing request for node system.admin\nSign certificate request with CA\nImport back to keystore (including CA chain)\nAll done for system.admin\nGenerating keystore and certificate for node elasticsearch\nGenerating certificate signing request for node elasticsearch\nSign certificate request with CA\nImport back to keystore (including CA chain)\nAll done for elasticsearch\nGenerating keystore and certificate for node logging-es\nkeytool error: java.lang.RuntimeException: java.io.IOException: DNSName components must begin with a letter\n", "stdout_lines": ["Generating keystore and certificate for node system.admin", "Generating certificate signing request for node system.admin", "Sign certificate request with CA", "Import back to keystore (including CA chain)", "All done for system.admin", "Generating keystore and certificate for node elasticsearch", "Generating certificate signing request for node elasticsearch", "Sign certificate request with CA", "Import back to keystore (including CA chain)", "All done for elasticsearch", "Generating keystore and certificate for node logging-es", "keytool error: java.lang.RuntimeException: java.io.IOException: DNSName components must begin with a letter"]} PLAY RECAP ********************************************************************* localhost : ok=11 changed=0 unreachable=0 failed=0 openshift-181.lab.eng.nay.redhat.com : ok=104 changed=18 unreachable=0 failed=1 openshift-182.lab.eng.nay.redhat.com : ok=0 changed=0 unreachable=0 failed=0 openshift-210.lab.eng.nay.redhat.com : ok=0 changed=0 unreachable=0 failed=0 openshift-217.lab.eng.nay.redhat.com : ok=0 changed=0 unreachable=0 failed=0 openshift-226.lab.eng.nay.redhat.com : ok=0 changed=0 unreachable=0 failed=0 INSTALLER STATUS *************************************************************** Initialization : Complete (0:00:22) Logging Install : In Progress (0:00:38) This phase can be restarted by running: playbooks/openshift-logging/config.yml Expected results: Logging can be deployed Additional info:
keytool error: java.lang.RuntimeException: java.io.IOException: DNSName components must begin with a letter The problem is the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1554878 I unconditionally added the external ES hostnames - not aware that it was possible to have a valid hostname/domain name that would _not_ be a valid DNS value in subject alt name :-(
san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local,dns:es.0416-8p6.qe.rhcloud.com,dns:es-ops.0416-8p6.qe.rhcloud.com It is complaining about the hostnames es.0416-8p6.qe.rhcloud.com and es-ops.0416-8p6.qe.rhcloud.com
Commits pushed to master at https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible/commit/2f6708286d16be57ed17ac3ba6608abae0c9b457 Bug 1567767 - openshift_logging : Run JKS generation script failed https://bugzilla.redhat.com/show_bug.cgi?id=1567767 Hostnames or hostname components beginning with a digit are not allowed to be a DNS item in a certificate subjectAltName. If such hostnames are presented as Elasticsearch hostnames, ops and non-ops, then issue a warning, and exclude them from the Elasticsearch server certificate subjectAltName. https://github.com/openshift/openshift-ansible/commit/313dbb3d25c8573874e50f6ce92f64f1d7772cc2 Merge pull request #7996 from richm/bug-1567767 Bug 1567767 - openshift_logging : Run JKS generation script failed
committed to release-3.9: https://github.com/openshift/openshift-ansible/commit/8ae62129daa55b46aff9454869d53865899340e9
Issue is fixed with openshift-ansible-3.9.30, logging could be deployed successfully without error. # openshift version openshift v3.9.30 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.16
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1796