Bug 1428849
| Summary: | [IntService_public_324] Kibana route is inaccessible after logging was upgraded to 3.5.0 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Xia Zhao <xiazhao> |
| Component: | Installer | Assignee: | ewolinet |
| Status: | CLOSED ERRATA | QA Contact: | Xia Zhao <xiazhao> |
| Severity: | low | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.5.0 | CC: | aos-bugs, ewolinet, jokerman, juzhao, mmccomas, xiazhao |
| Target Milestone: | --- | ||
| Target Release: | 3.5.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: |
undefined
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-14 21:01:20 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Attachments: | |||
|
Description
Xia Zhao
2017-03-03 13:51:36 UTC
Can you provide the Kibana and ES pod logs? @ewolinet,
It is maybe related to kibana-proxy OauthClient.Secret, use 'oadm diagnostics AggregatedLogging', shows the following error:
ERROR: [AGL0130 from diagnostic AggregatedLogging@openshift/origin/pkg/diagnostics/cluster/aggregated_logging/kibana.go:52]
The kibana-proxy OauthClient.Secret does not match the decoded oauth secret in 'logging-kibana-proxy'
ES, Kibana logs and other info please see the attached file
Created attachment 1261101 [details]
ES, Kibana logs and other info
Created attachment 1261105 [details]
ES, Kibana logs and other info, use this one
Created attachment 1261106 [details]
ES, Kibana logs and other info, use this one
Thanks for the output from the oc diagnostics, that really helped! Based on the Ansible logs its looking like we updated the oauth-client but we didn't generate a new logging-kibana-proxy secret template to update the value there. Junqi, If you rerun the logging playbook, do you see this issue of not being able to connect go away? When I rerun the playbook off of master I see the secret being recreated and patched (unlike in https://bugzilla.redhat.com/attachment.cgi?id=1261106) Upgraded from 3.3.1 to 3.5.0, es pod can not start up, Unable to read /etc/elasticsearch/secret/searchguard.truststore, defect already in bugzilia, https://bugzilla.redhat.com/show_bug.cgi?id=1428711. use 'oadm diagnostics AggregatedLogging', still shows the following error: ERROR: [AGL0130 from diagnostic AggregatedLogging@openshift/origin/pkg/diagnostics/cluster/aggregated_logging/kibana.go:52] The kibana-proxy OauthClient.Secret does not match the decoded oauth secret in 'logging-kibana-proxy' ES, kibana pod log and other info, please see the attached file. Created attachment 1261452 [details]
ES, Kibana logs and other info, upgrade from 3.3.1 to 3.5.0
I was able to recreate this, for some reason we are not correctly detecting that the kibana-proxy secret is different and should have a patch generated. Issue reproduced with the current playbook from repo https://github.com/openshift/openshift-ansible, -b master head commit id is 4c6d052e913c8a033c0198f4199b0f37f697bbe7 @Xia, The cherry picked changes for that have not yet been merged into master. The PR above was merged into the release-1.5 branch though. Can you verify using that in the mean-time? This is the PR for the cherry picked changes into master: https://github.com/openshift/openshift-ansible/pull/3629 (In reply to ewolinet from comment #16) > @Xia, > > The cherry picked changes for that have not yet been merged into master. The > PR above was merged into the release-1.5 branch though. Can you verify using > that in the mean-time? > > This is the PR for the cherry picked changes into master: > https://github.com/openshift/openshift-ansible/pull/3629 @Eric, My appologize that I somehow missed your message (and please feel free to set it to "need info" status tagging me, I usually look at that list, thanks!) , retested with the playbooks on repo https://github.com/openshift/openshift-ansible -b release-1.5 head commit: e33897ca1d71a7c5817c9cff5ce6cdb632d63ddd and meet this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1435176 I'll retest to provide the logs due to https://bugzilla.redhat.com/show_bug.cgi?id=1435176#c1 Verified this defect when upgrade logging form 3.4.1 to 3.5.0, elasticsearch attached hostpath PV, Kibana route is accessible after upgrade openshift-ansible version: openshift-ansible-3.5.45-1.git.0.eb0859b.el7.noarch openshift-ansible-playbooks-3.5.45-1.git.0.eb0859b.el7.noarch Image ID: openshift3/logging-deployer 3.4.1 1adc612d46b0 2 days ago 889.5 MB openshift3/logging-elasticsearch 3.4.1 246537fe4546 4 days ago 399.2 MB openshift3/logging-auth-proxy 3.4.1 d85303b2c262 2 weeks ago 219.8 MB openshift3/logging-kibana 3.4.1 03900b0b9416 2 weeks ago 339.1 MB openshift3/logging-fluentd 3.4.1 e4b97776c79b 2 weeks ago 233 MB openshift3/logging-curator 3.4.1 091de35492d6 2 weeks ago 244.3 M openshift3/logging-elasticsearch 3.5.0 5ff198b5c68d 4 days ago 399.4 MB openshift3/logging-kibana 3.5.0 a6159c640977 2 weeks ago 342.4 MB openshift3/logging-fluentd 3.5.0 32a4ac0a3e18 2 weeks ago 232.5 MB openshift3/logging-curator 3.5.0 8cfcb23f26b6 3 weeks ago 211.1 MB openshift3/logging-auth-proxy 3.5.0 139f7943475e 9 weeks ago 220 MB Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3438 |