Bug 1428849 - [IntService_public_324] Kibana route is inaccessible after logging was upgraded to 3.5.0
Summary: [IntService_public_324] Kibana route is inaccessible after logging was upgrad...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.5.0
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 3.5.z
Assignee: ewolinet
QA Contact: Xia Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-03-03 13:51 UTC by Xia Zhao
Modified: 2017-12-14 21:01 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-12-14 21:01:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
screenshot when visit kibana route from web browser (105.86 KB, image/png)
2017-03-03 13:51 UTC, Xia Zhao
no flags Details
ES, Kibana logs and other info (29.11 KB, text/plain)
2017-03-08 07:28 UTC, Junqi Zhao
no flags Details
ES, Kibana logs and other info, use this one (29.11 KB, text/plain)
2017-03-08 07:33 UTC, Junqi Zhao
no flags Details
ES, Kibana logs and other info, use this one (31.60 KB, text/plain)
2017-03-08 07:38 UTC, Junqi Zhao
no flags Details
ES, Kibana logs and other info, upgrade from 3.3.1 to 3.5.0 (148.92 KB, text/plain)
2017-03-09 07:27 UTC, Junqi Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:3438 0 normal SHIPPED_LIVE OpenShift Container Platform 3.6 and 3.5 bug fix and enhancement update 2017-12-15 01:58:11 UTC

Description Xia Zhao 2017-03-03 13:51:36 UTC
Created attachment 1259553 [details]
screenshot when visit kibana route from web browser

Description of problem:
Upgrade logging stacks from 3.3.1 to 3.5.0 by using ansible scripts, after ansible execution process completed successfully, the kibana route turned to be inaccesible. Get this error on browser:

"Application is not available

The application is currently not serving requests at this endpoint. It may not have been started or is still starting."

Version-Release number of selected component (if applicable):
openshift-ansible-3.5.20-1.git.0.5a5fcd5.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. Install logging 3.3.1 stacks on a OCP 3.5.0 master, attach elasticsearch with the HostPath PV
2. Visit kibana route before upgrade
3. Upgrade logging stacks to 3.5.0 by using ansible scripts (inventory file attached)
4. Visit kibana route post upgrade

Actual results:
2. Kibana route accessible with log entries
4. Kibana route inaccessible, get error in browser

Expected results:
ES should be running fine after upgrade

Additional info:
Ansible log and upgrade inventory file was same as https://bugzilla.redhat.com/show_bug.cgi?id=1428249, they are same scenario.

Comment 1 ewolinet 2017-03-07 14:56:48 UTC
Can you provide the Kibana and ES pod logs?

Comment 2 Junqi Zhao 2017-03-08 07:27:45 UTC
@ewolinet,

It is maybe related to kibana-proxy OauthClient.Secret, use 'oadm diagnostics AggregatedLogging', shows the following error:

ERROR: [AGL0130 from diagnostic AggregatedLogging@openshift/origin/pkg/diagnostics/cluster/aggregated_logging/kibana.go:52]
       The kibana-proxy OauthClient.Secret does not match the decoded oauth secret in 'logging-kibana-proxy'


ES, Kibana logs and other info please see the attached file

Comment 3 Junqi Zhao 2017-03-08 07:28:35 UTC
Created attachment 1261101 [details]
ES, Kibana logs and other info

Comment 4 Junqi Zhao 2017-03-08 07:33:43 UTC
Created attachment 1261105 [details]
ES, Kibana logs and other info, use this one

Comment 5 Junqi Zhao 2017-03-08 07:38:27 UTC
Created attachment 1261106 [details]
ES, Kibana logs and other info, use this one

Comment 6 ewolinet 2017-03-08 17:43:19 UTC
Thanks for the output from the oc diagnostics, that really helped! 

Based on the Ansible logs its looking like we updated the oauth-client but we didn't generate a new logging-kibana-proxy secret template to update the value there.

Comment 7 ewolinet 2017-03-08 21:18:56 UTC
Junqi,

If you rerun the logging playbook, do you see this issue of not being able to connect go away? When I rerun the playbook off of master I see the secret being recreated and patched (unlike in https://bugzilla.redhat.com/attachment.cgi?id=1261106)

Comment 10 Junqi Zhao 2017-03-09 07:26:44 UTC
Upgraded from 3.3.1 to 3.5.0,
es pod can not start up, Unable to read /etc/elasticsearch/secret/searchguard.truststore, defect already in bugzilia, https://bugzilla.redhat.com/show_bug.cgi?id=1428711.

use 'oadm diagnostics AggregatedLogging', still shows the following error:
ERROR: [AGL0130 from diagnostic AggregatedLogging@openshift/origin/pkg/diagnostics/cluster/aggregated_logging/kibana.go:52]
       The kibana-proxy OauthClient.Secret does not match the decoded oauth secret in 'logging-kibana-proxy'


ES, kibana pod log and other info, please see the attached file.

Comment 11 Junqi Zhao 2017-03-09 07:27:32 UTC
Created attachment 1261452 [details]
ES, Kibana logs and other info, upgrade from 3.3.1 to 3.5.0

Comment 13 ewolinet 2017-03-10 20:24:09 UTC
I was able to recreate this, for some reason we are not correctly detecting that the kibana-proxy secret is different and should have a patch generated.

Comment 15 Xia Zhao 2017-03-15 09:20:05 UTC
Issue reproduced with the current playbook from repo https://github.com/openshift/openshift-ansible, -b master

head commit id is 4c6d052e913c8a033c0198f4199b0f37f697bbe7

Comment 16 ewolinet 2017-03-15 15:12:09 UTC
@Xia,

The cherry picked changes for that have not yet been merged into master. The PR above was merged into the release-1.5 branch though. Can you verify using that in the mean-time?

This is the PR for the cherry picked changes into master: https://github.com/openshift/openshift-ansible/pull/3629

Comment 17 Xia Zhao 2017-03-23 10:00:27 UTC
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1435144

Comment 18 Xia Zhao 2017-03-23 10:35:08 UTC
(In reply to ewolinet from comment #16)
> @Xia,
> 
> The cherry picked changes for that have not yet been merged into master. The
> PR above was merged into the release-1.5 branch though. Can you verify using
> that in the mean-time?
> 
> This is the PR for the cherry picked changes into master:
> https://github.com/openshift/openshift-ansible/pull/3629

@Eric,

My appologize that I somehow missed your message (and please feel free to set it to "need info" status tagging me, I usually look at that list, thanks!) , retested with the playbooks on repo

https://github.com/openshift/openshift-ansible -b release-1.5
head commit: e33897ca1d71a7c5817c9cff5ce6cdb632d63ddd

and meet this issue: https://bugzilla.redhat.com/show_bug.cgi?id=1435176

I'll retest to provide the logs due to https://bugzilla.redhat.com/show_bug.cgi?id=1435176#c1

Comment 21 Junqi Zhao 2017-03-27 03:21:30 UTC
Verified this defect when upgrade logging form 3.4.1 to 3.5.0, elasticsearch attached hostpath PV, Kibana route is accessible after upgrade

openshift-ansible version:
openshift-ansible-3.5.45-1.git.0.eb0859b.el7.noarch
openshift-ansible-playbooks-3.5.45-1.git.0.eb0859b.el7.noarch

Image ID:
openshift3/logging-deployer        3.4.1               1adc612d46b0        2 days ago          889.5 MB
openshift3/logging-elasticsearch   3.4.1               246537fe4546        4 days ago          399.2 MB
openshift3/logging-auth-proxy      3.4.1               d85303b2c262        2 weeks ago         219.8 MB
openshift3/logging-kibana          3.4.1               03900b0b9416        2 weeks ago         339.1 MB
openshift3/logging-fluentd         3.4.1               e4b97776c79b        2 weeks ago         233 MB
openshift3/logging-curator         3.4.1               091de35492d6        2 weeks ago         244.3 M



openshift3/logging-elasticsearch   3.5.0               5ff198b5c68d        4 days ago          399.4 MB
openshift3/logging-kibana          3.5.0               a6159c640977        2 weeks ago         342.4 MB
openshift3/logging-fluentd         3.5.0               32a4ac0a3e18        2 weeks ago         232.5 MB
openshift3/logging-curator         3.5.0               8cfcb23f26b6        3 weeks ago         211.1 MB
openshift3/logging-auth-proxy      3.5.0               139f7943475e        9 weeks ago         220 MB

Comment 24 errata-xmlrpc 2017-12-14 21:01:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3438


Note You need to log in before you can comment on or make changes to this bug.