Bug 1569350

Summary:	[3.10] openshift_logging : Run JKS generation script failed
Product:	OpenShift Container Platform	Reporter:	Anping Li <anli>
Component:	Logging	Assignee:	Rich Megginson <rmeggins>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.10.0	CC:	aos-bugs, jokerman, mmccomas, rmeggins
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	3.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: When creating the Elasticsearch server certificate, the external Elasticsearch hostnames are being unconditionally added to the subjectAltName. Consequence: Install fails because only hostname components beginning with a letter are allowed in the subjectAltName, so hostnames like es.0xdeadbeef.com disallowed and cause an error. Fix: Issue a warning if the Elasticsearch hostname contains a component which does not begin with a letter, and do not add it to the subjectAltName. Result: Logging install completes successfully.	Story Points:	---
Clone Of:	1567767	Environment:
Last Closed:	2018-07-30 19:13:21 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1567767
Bug Blocks:	1568660

Description Anping Li 2018-04-19 06:34:18 UTC

+++ This bug was initially created as a clone of Bug #1567767 +++

Description of problem:
The logging deploy failed at task 'openshift_logging : Run JKS generation script'

Version-Release number of selected component (if applicable):
openshift-ansible-3.9.22

How reproducible:
always

Steps to Reproduce:
1.Deploy logging
2.
3.

Actual results:

TASK [openshift_logging : pulling down signing items from host] ****************
changed: [openshift-181.lab.eng.nay.redhat.com] => (item=ca.crt)
changed: [openshift-181.lab.eng.nay.redhat.com] => (item=ca.key)
changed: [openshift-181.lab.eng.nay.redhat.com] => (item=ca.serial.txt)
changed: [openshift-181.lab.eng.nay.redhat.com] => (item=ca.crl.srl)
changed: [openshift-181.lab.eng.nay.redhat.com] => (item=ca.db)

TASK [openshift_logging : template] ********************************************
changed: [openshift-181.lab.eng.nay.redhat.com -> localhost]

TASK [openshift_logging : Run JKS generation script] ***************************
fatal: [openshift-181.lab.eng.nay.redhat.com -> localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "+ '[' 4 -lt 1 ']'\n+ dir=/tmp/openshift-logging-ansible-jODsUS\n+ SCRATCH_DIR=/tmp/openshift-logging-ansible-jODsUS\n+ PROJECT=logging\n+ MORE_ES_NAMES=es.0416-8p6.qe.rhcloud.com\n+ escomma=,\n+ MORE_ES_OPS_NAMES=es-ops.0416-8p6.qe.rhcloud.com\n+ esopscomma=,\n+ [[ ! -f /tmp/openshift-logging-ansible-jODsUS/system.admin.jks ]]\n+ generate_JKS_client_cert system.admin\n+ NODE_NAME=system.admin\n+ ks_pass=kspass\n+ ts_pass=tspass\n+ dir=/tmp/openshift-logging-ansible-jODsUS\n+ echo Generating keystore and certificate for node system.admin\n+ keytool -genkey -alias system.admin -keystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -keyalg RSA -keysize 2048 -validity 712 -keypass kspass -storepass kspass -dname 'CN=system.admin, OU=OpenShift, O=Logging'\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -deststoretype pkcs12\".\n+ echo Generating certificate signing request for node system.admin\n+ keytool -certreq -alias system.admin -keystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -file /tmp/openshift-logging-ansible-jODsUS/system.admin.jks.csr -keyalg rsa -keypass kspass -storepass kspass -dname 'CN=system.admin, OU=OpenShift, O=Logging'\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -deststoretype pkcs12\".\n+ echo Sign certificate request with CA\n+ openssl ca -in /tmp/openshift-logging-ansible-jODsUS/sys
tem.admin.jks.csr -notext -out /tmp/openshift-logging-ansible-jODsUS/system.admin.jks.crt -config /tmp/openshift-logging-ansible-jODsUS/signing.conf -extensions v3_req -batch -extensions server_ext\nUsing configuration from /tmp/openshift-logging-ansible-jODsUS/signing.conf\nCheck that the request matches the signature\nSignature ok\nCertificate Details:\n        Serial Number: 6 (0x6)\n        Validity\n            Not Before: Apr 16 06:59:26 2018 GMT\n            Not After : Apr 15 06:59:26 2020 GMT\n        Subject:\n            organizationName          = Logging\n            organizationalUnitName    = OpenShift\n            commonName                = system.admin\n        X509v3 extensions:\n            X509v3 Key Usage: critical\n                Digital Signature, Key Encipherment\n            X509v3 Basic Constraints: \n                CA:FALSE\n            X509v3 Extended Key Usage: \n                TLS Web Server Authentication, TLS Web Client Authentication\n            X509v3 Subject Key Identifier: \n                5D:12:E8:DD:E8:24:7B:A6:02:81:B3:E9:5C:68:73:4F:5E:22:16:D2\n            X509v3 Authority Key Identifier: \n                0.\nCertificate is to be certified until Apr 15 06:59:26 2020 GMT (730 days)\n\nWrite out database with 1 new entries\nData Base Updated\n+ echo 'Import back to keystore (including CA chain)'\n+ keytool -import -file /tmp/openshift-logging-ansible-jODsUS/ca.crt -keystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -storepass kspass -noprompt -alias sig-ca\nCertificate was added to keystore\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -deststoretype pkcs12\".\n+ keytool -import -file /tmp/openshift-logging-ansible-jODsUS/system.admin.jks.crt -keystore /tmp
/openshift-logging-ansible-jODsUS/system.admin.jks -storepass kspass -noprompt -alias system.admin\nCertificate reply was installed in keystore\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/system.admin.jks -deststoretype pkcs12\".\n+ echo All done for system.admin\n+ [[ ! -f /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks ]]\n++ join , logging-es logging-es-ops\n++ local IFS=,\n++ shift\n++ echo logging-es,logging-es-ops\n+ generate_JKS_chain true elasticsearch logging-es,logging-es-ops\n+ dir=/tmp/openshift-logging-ansible-jODsUS\n+ ADD_OID=true\n+ NODE_NAME=elasticsearch\n+ CERT_NAMES=logging-es,logging-es-ops\n+ ks_pass=kspass\n+ ts_pass=tspass\n+ rm -rf elasticsearch\n+ extension_names=\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es-ops\n+ '[' true = true ']'\n+ extension_names=,dns:logging-es,dns:logging-es-ops,oid:1.2.3.4.5.5\n+ echo Generating keystore and certificate for node elasticsearch\n+ keytool -genkey -alias elasticsearch -keystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -keypass kspass -storepass kspass -keyalg RSA -keysize 2048 -validity 712 -dname 'CN=elasticsearch, OU=OpenShift, O=Logging' -ext san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es-ops,oid:1.2.3.4.5.5\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -deststoretype pkcs12\".\n+ echo Generating certificate signing request for node elasticsearch\n+ keytool -cer
treq -alias elasticsearch -keystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -storepass kspass -file /tmp/openshift-logging-ansible-jODsUS/elasticsearch.csr -keyalg rsa -dname 'CN=elasticsearch, OU=OpenShift, O=Logging' -ext san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es-ops,oid:1.2.3.4.5.5\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -deststoretype pkcs12\".\n+ echo Sign certificate request with CA\n+ openssl ca -in /tmp/openshift-logging-ansible-jODsUS/elasticsearch.csr -notext -out /tmp/openshift-logging-ansible-jODsUS/elasticsearch.crt -config /tmp/openshift-logging-ansible-jODsUS/signing.conf -extensions v3_req -batch -extensions server_ext\nUsing configuration from /tmp/openshift-logging-ansible-jODsUS/signing.conf\nCheck that the request matches the signature\nSignature ok\nCertificate Details:\n        Serial Number: 7 (0x7)\n        Validity\n            Not Before: Apr 16 06:59:27 2018 GMT\n            Not After : Apr 15 06:59:27 2020 GMT\n        Subject:\n            organizationName          = Logging\n            organizationalUnitName    = OpenShift\n            commonName                = elasticsearch\n        X509v3 extensions:\n            X509v3 Key Usage: critical\n                Digital Signature, Key Encipherment\n            X509v3 Basic Constraints: \n                CA:FALSE\n            X509v3 Extended Key Usage: \n                TLS Web Server Authentication, TLS Web Client Authentication\n            X509v3 Subject Key Identifier: \n                CD:22:F3:75:83:E6:D5:E3:9A:2C:A8:75:6C:AD:DA:0B:BB:1A:16:1F\n            X509v3 Authority Key Identifier: \n                0.\n            X509v3 Subject Alternative Name: \n                DNS:localhost, IP
 Address:127.0.0.1, DNS:logging-es, DNS:logging-es-ops, Registered ID:1.2.3.4.5.5\nCertificate is to be certified until Apr 15 06:59:27 2020 GMT (730 days)\n\nWrite out database with 1 new entries\nData Base Updated\n+ echo 'Import back to keystore (including CA chain)'\n+ keytool -import -file /tmp/openshift-logging-ansible-jODsUS/ca.crt -keystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -storepass kspass -noprompt -alias sig-ca\nCertificate was added to keystore\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -deststoretype pkcs12\".\n+ keytool -import -file /tmp/openshift-logging-ansible-jODsUS/elasticsearch.crt -keystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -storepass kspass -noprompt -alias elasticsearch\nCertificate reply was installed in keystore\n\nWarning:\nThe JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using \"keytool -importkeystore -srckeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -destkeystore /tmp/openshift-logging-ansible-jODsUS/elasticsearch.jks -deststoretype pkcs12\".\n+ echo All done for elasticsearch\n+ [[ ! -f /tmp/openshift-logging-ansible-jODsUS/logging-es.jks ]]\n++ join , logging-es logging-es.logging.svc.cluster.local logging-es-cluster logging-es-cluster.logging.svc.cluster.local logging-es-ops logging-es-ops.logging.svc.cluster.local logging-es-ops-cluster logging-es-ops-cluster.logging.svc.cluster.local\n++ local IFS=,\n++ shift\n++ echo logging-es,logging-es.logging.svc.cluster.local,logging-es-cluster,logging-es-cluster.logging.svc.cluster.local,logging-es-ops,logging-es-ops.logging.svc.cluster.local,logging-es-ops-cluster,logging-es-ops-cluster.logging.svc.clu
ster.local\n+ generate_JKS_chain false logging-es logging-es,logging-es.logging.svc.cluster.local,logging-es-cluster,logging-es-cluster.logging.svc.cluster.local,logging-es-ops,logging-es-ops.logging.svc.cluster.local,logging-es-ops-cluster,logging-es-ops-cluster.logging.svc.cluster.local,es.0416-8p6.qe.rhcloud.com,es-ops.0416-8p6.qe.rhcloud.com\n+ dir=/tmp/openshift-logging-ansible-jODsUS\n+ ADD_OID=false\n+ NODE_NAME=logging-es\n+ CERT_NAMES=logging-es,logging-es.logging.svc.cluster.local,logging-es-cluster,logging-es-cluster.logging.svc.cluster.local,logging-es-ops,logging-es-ops.logging.svc.cluster.local,logging-es-ops-cluster,logging-es-ops-cluster.logging.svc.cluster.local,es.0416-8p6.qe.rhcloud.com,es-ops.0416-8p6.qe.rhcloud.com\n+ ks_pass=kspass\n+ ts_pass=tspass\n+ rm -rf logging-es\n+ extension_names=\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.lo
gging.svc.cluster.local,dns:logging-es-ops-cluster\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local,dns:es.0416-8p6.qe.rhcloud.com\n+ for name in '${CERT_NAMES//,/ }'\n+ extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local,dns:es.0416-8p6.qe.rhcloud.com,dns:es-ops.0416-8p6.qe.rhcloud.com\n+ '[' false = true ']'\n+ echo Generating keystore and certificate for node logging-es\n+ keytool -genkey -alias logging-es -keystore /tmp/openshift-logging-ansible-jODsUS/logging-es.jks -keypass kspass -storepass kspass -keyalg RSA -keysize 2048 -validity 712 -dname 'CN=logging-es, OU=OpenShift, O=Logging' -ext san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local,dns:es.0416-8p6.qe.rhcloud.com,dns:es-ops.0416-8p6.qe.rhcloud.com\n", "stdout": "Generating keystore and certificate for node system.admin\nGenerating certificate signing request for node system.admin\nSign certificate request with CA\nImport back to
 keystore (including CA chain)\nAll done for system.admin\nGenerating keystore and certificate for node elasticsearch\nGenerating certificate signing request for node elasticsearch\nSign certificate request with CA\nImport back to keystore (including CA chain)\nAll done for elasticsearch\nGenerating keystore and certificate for node logging-es\nkeytool error: java.lang.RuntimeException: java.io.IOException: DNSName components must begin with a letter\n", "stdout_lines": ["Generating keystore and certificate for node system.admin", "Generating certificate signing request for node system.admin", "Sign certificate request with CA", "Import back to keystore (including CA chain)", "All done for system.admin", "Generating keystore and certificate for node elasticsearch", "Generating certificate signing request for node elasticsearch", "Sign certificate request with CA", "Import back to keystore (including CA chain)", "All done for elasticsearch", "Generating keystore and certificate for node logging-es", "keytool error: java.lang.RuntimeException: java.io.IOException: DNSName components must begin with a letter"]}

PLAY RECAP *********************************************************************
localhost                  : ok=11   changed=0    unreachable=0    failed=0   
openshift-181.lab.eng.nay.redhat.com : ok=104  changed=18   unreachable=0    failed=1   
openshift-182.lab.eng.nay.redhat.com : ok=0    changed=0    unreachable=0    failed=0   
openshift-210.lab.eng.nay.redhat.com : ok=0    changed=0    unreachable=0    failed=0   
openshift-217.lab.eng.nay.redhat.com : ok=0    changed=0    unreachable=0    failed=0   
openshift-226.lab.eng.nay.redhat.com : ok=0    changed=0    unreachable=0    failed=0   


INSTALLER STATUS ***************************************************************
Initialization             : Complete (0:00:22)
Logging Install            : In Progress (0:00:38)
    This phase can be restarted by running: playbooks/openshift-logging/config.yml

Expected results:
Logging can be deployed

Additional info:

--- Additional comment from Rich Megginson on 2018-04-16 13:02:13 EDT ---

keytool error: java.lang.RuntimeException: java.io.IOException: DNSName components must begin with a letter

The problem is the fix for https://bugzilla.redhat.com/show_bug.cgi?id=1554878

I unconditionally added the external ES hostnames - not aware that it was possible to have a valid hostname/domain name that would _not_ be a valid DNS value in subject alt name :-(

--- Additional comment from Rich Megginson on 2018-04-16 13:26:47 EDT ---

san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local,dns:es.0416-8p6.qe.rhcloud.com,dns:es-ops.0416-8p6.qe.rhcloud.com

It is complaining about the hostnames es.0416-8p6.qe.rhcloud.com and es-ops.0416-8p6.qe.rhcloud.com

--- Additional comment from openshift-github-bot on 2018-04-17 22:49:52 EDT ---

Commits pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/2f6708286d16be57ed17ac3ba6608abae0c9b457
Bug 1567767 - openshift_logging : Run JKS generation script failed

https://bugzilla.redhat.com/show_bug.cgi?id=1567767
Hostnames or hostname components beginning with a digit are not
allowed to be a DNS item in a certificate subjectAltName.  If
such hostnames are presented as Elasticsearch hostnames, ops and
non-ops, then issue a warning, and exclude them from the Elasticsearch
server certificate subjectAltName.

https://github.com/openshift/openshift-ansible/commit/313dbb3d25c8573874e50f6ce92f64f1d7772cc2
Merge pull request #7996 from richm/bug-1567767

Bug 1567767 - openshift_logging : Run JKS generation script failed

--- Additional comment from Rich Megginson on 2018-04-18 10:01:01 EDT ---

committed to release-3.9: https://github.com/openshift/openshift-ansible/commit/8ae62129daa55b46aff9454869d53865899340e9

Comment 1 Anping Li 2018-04-19 06:35:30 UTC

Same issue with openshift3/ose-ansible/images/v3.10.0-0.22.0.0. Waiting the new oepnshift-ansible packages to include the fix.

Comment 2 Jeff Cantrill 2018-04-19 12:48:05 UTC

Fixed in https://github.com/openshift/openshift-ansible/pull/7996

Comment 3 Junqi Zhao 2018-04-20 05:45:35 UTC

There is not error in logging deployment process

# rpm -qa | grep openshift-ansible
openshift-ansible-3.10.0-0.24.0.git.0.fd7f37c.el7.noarch
openshift-ansible-docs-3.10.0-0.24.0.git.0.fd7f37c.el7.noarch
openshift-ansible-roles-3.10.0-0.24.0.git.0.fd7f37c.el7.noarch
openshift-ansible-playbooks-3.10.0-0.24.0.git.0.fd7f37c.el7.noarch

Comment 5 errata-xmlrpc 2018-07-30 19:13:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816