Hide Forgot
Created attachment 1259553 [details] screenshot when visit kibana route from web browser Description of problem: Upgrade logging stacks from 3.3.1 to 3.5.0 by using ansible scripts, after ansible execution process completed successfully, the kibana route turned to be inaccesible. Get this error on browser: "Application is not available The application is currently not serving requests at this endpoint. It may not have been started or is still starting." Version-Release number of selected component (if applicable): openshift-ansible-3.5.20-1.git.0.5a5fcd5.el7.noarch How reproducible: Always Steps to Reproduce: 1. Install logging 3.3.1 stacks on a OCP 3.5.0 master, attach elasticsearch with the HostPath PV 2. Visit kibana route before upgrade 3. Upgrade logging stacks to 3.5.0 by using ansible scripts (inventory file attached) 4. Visit kibana route post upgrade Actual results: 2. Kibana route accessible with log entries 4. Kibana route inaccessible, get error in browser Expected results: ES should be running fine after upgrade Additional info: Ansible log and upgrade inventory file was same as https://bugzilla.redhat.com/show_bug.cgi?id=1428249, they are same scenario.
Can you provide the Kibana and ES pod logs?
@ewolinet, It is maybe related to kibana-proxy OauthClient.Secret, use 'oadm diagnostics AggregatedLogging', shows the following error: ERROR: [AGL0130 from diagnostic AggregatedLogging@openshift/origin/pkg/diagnostics/cluster/aggregated_logging/kibana.go:52] The kibana-proxy OauthClient.Secret does not match the decoded oauth secret in 'logging-kibana-proxy' ES, Kibana logs and other info please see the attached file
Created attachment 1261101 [details] ES, Kibana logs and other info
Created attachment 1261105 [details] ES, Kibana logs and other info, use this one
Created attachment 1261106 [details] ES, Kibana logs and other info, use this one
Thanks for the output from the oc diagnostics, that really helped! Based on the Ansible logs its looking like we updated the oauth-client but we didn't generate a new logging-kibana-proxy secret template to update the value there.
Junqi, If you rerun the logging playbook, do you see this issue of not being able to connect go away? When I rerun the playbook off of master I see the secret being recreated and patched (unlike in https://bugzilla.redhat.com/attachment.cgi?id=1261106)
Upgraded from 3.3.1 to 3.5.0, es pod can not start up, Unable to read /etc/elasticsearch/secret/searchguard.truststore, defect already in bugzilia, https://bugzilla.redhat.com/show_bug.cgi?id=1428711. use 'oadm diagnostics AggregatedLogging', still shows the following error: ERROR: [AGL0130 from diagnostic AggregatedLogging@openshift/origin/pkg/diagnostics/cluster/aggregated_logging/kibana.go:52] The kibana-proxy OauthClient.Secret does not match the decoded oauth secret in 'logging-kibana-proxy' ES, kibana pod log and other info, please see the attached file.
Created attachment 1261452 [details] ES, Kibana logs and other info, upgrade from 3.3.1 to 3.5.0
I was able to recreate this, for some reason we are not correctly detecting that the kibana-proxy secret is different and should have a patch generated.
https://github.com/openshift/openshift-ansible/pull/3627
Issue reproduced with the current playbook from repo https://github.com/openshift/openshift-ansible, -b master head commit id is 4c6d052e913c8a033c0198f4199b0f37f697bbe7
@Xia, The cherry picked changes for that have not yet been merged into master. The PR above was merged into the release-1.5 branch though. Can you verify using that in the mean-time? This is the PR for the cherry picked changes into master: https://github.com/openshift/openshift-ansible/pull/3629
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1435144
(In reply to ewolinet from comment #16) > @Xia, > > The cherry picked changes for that have not yet been merged into master. The > PR above was merged into the release-1.5 branch though. Can you verify using > that in the mean-time? > > This is the PR for the cherry picked changes into master: > https://github.com/openshift/openshift-ansible/pull/3629 @Eric, My appologize that I somehow missed your message (and please feel free to set it to "need info" status tagging me, I usually look at that list, thanks!) , retested with the playbooks on repo https://github.com/openshift/openshift-ansible -b release-1.5 head commit: e33897ca1d71a7c5817c9cff5ce6cdb632d63ddd and meet this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1435176 I'll retest to provide the logs due to https://bugzilla.redhat.com/show_bug.cgi?id=1435176#c1
Verified this defect when upgrade logging form 3.4.1 to 3.5.0, elasticsearch attached hostpath PV, Kibana route is accessible after upgrade openshift-ansible version: openshift-ansible-3.5.45-1.git.0.eb0859b.el7.noarch openshift-ansible-playbooks-3.5.45-1.git.0.eb0859b.el7.noarch Image ID: openshift3/logging-deployer 3.4.1 1adc612d46b0 2 days ago 889.5 MB openshift3/logging-elasticsearch 3.4.1 246537fe4546 4 days ago 399.2 MB openshift3/logging-auth-proxy 3.4.1 d85303b2c262 2 weeks ago 219.8 MB openshift3/logging-kibana 3.4.1 03900b0b9416 2 weeks ago 339.1 MB openshift3/logging-fluentd 3.4.1 e4b97776c79b 2 weeks ago 233 MB openshift3/logging-curator 3.4.1 091de35492d6 2 weeks ago 244.3 M openshift3/logging-elasticsearch 3.5.0 5ff198b5c68d 4 days ago 399.4 MB openshift3/logging-kibana 3.5.0 a6159c640977 2 weeks ago 342.4 MB openshift3/logging-fluentd 3.5.0 32a4ac0a3e18 2 weeks ago 232.5 MB openshift3/logging-curator 3.5.0 8cfcb23f26b6 3 weeks ago 211.1 MB openshift3/logging-auth-proxy 3.5.0 139f7943475e 9 weeks ago 220 MB
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3438