Bug 1747305 - Couldn't regenerate the cert if the /etc/origin/logging is deleted
Summary: Couldn't regenerate the cert if the /etc/origin/logging is deleted
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.11.z
Assignee: Noriko Hosoi
QA Contact: Anping Li
Depends On:
TreeView+ depends on / blocked
Reported: 2019-08-30 05:08 UTC by Anping Li
Modified: 2019-10-18 01:34 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2019-10-18 01:34:36 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Github openshift openshift-ansible pull 11909 None closed [release-3.11] Bug 1747305: Couldn't regenerate the cert if the /etc/origin/logging… 2020-03-29 11:24:13 UTC
Red Hat Product Errata RHBA-2019:3139 None None None 2019-10-18 01:34:58 UTC

Description Anping Li 2019-08-30 05:08:35 UTC
Description of problem:
The redeploy-certificates.yml failed if the /etc/origin/logging is deleted on masters

Version-Release number of selected component (if applicable):

How reproducible:

1. Deploy logging
2. rm -rf /etc/origin/logging
3. redeploy certificates using playbook
ansible-playbook playbooks/openshift-logging/redeploy-certificates.yml

Actual results:
TASK [openshift_logging_kibana : Generate oauth secret] ************************
Friday 30 August 2019  02:52:56 +0000 (0:00:01.162)       0:01:09.801 ********* 
changed: [ec2-54-224-41-112.compute-1.amazonaws.com]

TASK [include_role : openshift_logging_kibana] *********************************
Friday 30 August 2019  02:52:57 +0000 (0:00:01.144)       0:01:10.946 ********* 

TASK [openshift_logging_kibana : Retrieving the cert to use when generating secrets for the logging components] ***
Friday 30 August 2019  02:52:57 +0000 (0:00:00.273)       0:01:11.220 ********* 
ok: [ec2-54-224-41-112.compute-1.amazonaws.com] => (item={u'name': u'ca_file', u'file': u'ca.crt'})
ok: [ec2-54-224-41-112.compute-1.amazonaws.com] => (item={u'name': u'kibana_internal_key', u'file': u'kibana-internal.key'})
ok: [ec2-54-224-41-112.compute-1.amazonaws.com] => (item={u'name': u'kibana_internal_cert', u'file': u'kibana-internal.crt'})
ok: [ec2-54-224-41-112.compute-1.amazonaws.com] => (item={u'name': u'server_tls', u'file': u'server-tls.json'})
ok: [ec2-54-224-41-112.compute-1.amazonaws.com] => (item={u'name': u'session_secret', u'file': u'session_secret'})
ok: [ec2-54-224-41-112.compute-1.amazonaws.com] => (item={u'name': u'oauth_secret', u'file': u'oauth_secret'})

TASK [include_role : {{logging_role}}] *****************************************
Friday 30 August 2019  02:52:58 +0000 (0:00:01.548)       0:01:12.769 ********* 

TASK [openshift_logging_elasticsearch : Set ES secret] *************************
Friday 30 August 2019  02:52:59 +0000 (0:00:00.397)       0:01:13.166 ********* 
fatal: [ec2-54-224-41-112.compute-1.amazonaws.com]: FAILED! => {"changed": false, "msg": {"cmd": "/usr/bin/oc -ojson secrets new logging-elasticsearch key=/etc/origin/logging/logging-es.jks truststore=/etc/origin/logging/truststore.jks searchguard.key=/etc/origin/logging/elasticsearch.jks searchguard.truststore=/etc/origin/logging/truststore.jks admin-key=/etc/origin/logging/system.admin.key admin-cert=/etc/origin/logging/system.admin.crt admin-ca=/etc/origin/logging/ca.crt admin.jks=/etc/origin/logging/system.admin.jks passwd.yml=/etc/origin/logging/passwd.yml -n openshift-logging", "results": {}, "returncode": 1, "stderr": "Command \"new\" is deprecated, use oc create secret\nerror: error reading /etc/origin/logging/passwd.yml: no such file or directory\n", "stdout": ""}}

PLAY RECAP *********************************************************************
ec2-3-85-242-224.compute-1.amazonaws.com : ok=0    changed=0    unreachable=0    failed=0   
ec2-3-89-32-175.compute-1.amazonaws.com : ok=0    changed=0    unreachable=0    failed=0   
ec2-34-229-54-18.compute-1.amazonaws.com : ok=0    changed=0    unreachable=0    failed=0   
ec2-35-172-116-251.compute-1.amazonaws.com : ok=0    changed=0    unreachable=0    failed=0   
ec2-54-224-41-112.compute-1.amazonaws.com : ok=122  changed=29   unreachable=0    failed=1   
localhost                  : ok=11   changed=0    unreachable=0    failed=0   

INSTALLER STATUS ***************************************************************
Initialization         : Complete (0:00:14)
Logging Cert Redeploy  : In Progress (0:00:59)

Actual results:

The certificates can be regenerated.

Comment 1 Anping Li 2019-08-30 05:10:14 UTC
Workaround:  preserve the file /etc/origin/logging/passwd.yml when you delete files under /etc/origin/logging

Comment 3 Anping Li 2019-10-12 05:53:12 UTC
TASK [openshift_logging_kibana : Generating Kibana route template] *************
Saturday 12 October 2019  05:40:20 +0000 (0:00:00.094)       0:01:42.038 ****** 
fatal: [ci-vm-10-0-148-139.hosted.upshift.rdu2.redhat.com]: FAILED! => {"msg": "The field 'vars' has an invalid value, which includes an undefined variable. The error was: 'openshift_logging_kibana_ops_hostname' is undefined\n\nThe error appears to have been in '/usr/share/ansible/openshift-ansible/roles/openshift_logging_kibana/tasks/generate_route.yaml': line 27, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Generating Kibana route template\n  ^ here\n"}

PLAY RECAP *********************************************************************
ci-vm-10-0-148-139.hosted.upshift.rdu2.redhat.com : ok=147  changed=38   unreachable=0    failed=1   
ci-vm-10-0-149-248.hosted.upshift.rdu2.redhat.com : ok=0    changed=0    unreachable=0    failed=0   
ci-vm-10-0-150-201.hosted.upshift.rdu2.redhat.com : ok=0    changed=0    unreachable=0    failed=0   
ci-vm-10-0-150-223.hosted.upshift.rdu2.redhat.com : ok=0    changed=0    unreachable=0    failed=0   
ci-vm-10-0-151-110.hosted.upshift.rdu2.redhat.com : ok=0    changed=0    unreachable=0    failed=0   
ci-vm-10-0-151-111.hosted.upshift.rdu2.redhat.com : ok=0    changed=0    unreachable=0    failed=0   
localhost                  : ok=11   changed=0    unreachable=0    failed=0

Comment 4 Anping Li 2019-10-12 07:18:38 UTC
verify using non-ops cluster.  Trace the bug in comment 3 in BZ1747307

Comment 6 errata-xmlrpc 2019-10-18 01:34:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.